Positioning method, device, electronic device and storage medium for head mounted display device

Abstract

The present application discloses a method, device, electronic device and storage medium for positioning a head-mounted display device. The positioning method includes: receiving an image transmitted by a camera device of the head-mounted display device; judging whether the current frame image belongs to a first scene according to a first preset judgment condition; if the current frame image belongs to the first scene, calculating the relative position between the head-mounted display device and the sound source; and updating the posture data of the head-mounted display device using the current frame image and the relative position. The positioning method of the head-mounted display device of the present application greatly improves the positioning accuracy of the head-mounted display device and can well meet the needs of practical applications.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to a positioning method, device, electronic device and storage medium for a head-mounted display device. Background Art

[0002] With the development of Virtual Reality (VR) and Augmented Reality (AR), the content of VR and AR is becoming more and more abundant, and the experience realization of many of these contents requires the handle tracking function and Inside-Out 6DOF (6 degrees of freedom) tracking function of HMD (head-mounted display) devices. The timeliness and accuracy of 6DOF positioning of head-mounted devices are crucial to the realization of VR and AR content, but the positioning accuracy of head-mounted display devices in the existing technology is lacking and needs to be improved urgently. Summary of the invention

[0003] The purpose of the present application is to provide a positioning method, device, electronic device and storage medium for a head-mounted display device. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0004] According to one aspect of an embodiment of the present application, a positioning method for a head mounted display device is provided, comprising:

[0005] Receiving an image transmitted by a camera device of the head mounted display device;

[0006] Determining whether the current frame image belongs to the first scene according to a first preset determination condition;

[0007] If the current frame image belongs to the first scene, calculating the relative position between the head mounted display device and the sound source;

[0008] The posture data of the head mounted display device is updated using the current frame image and the relative position.

[0009] In some embodiments of the present application, judging whether the current frame image belongs to the first scene according to the first preset condition includes:

[0010] Obtaining the number of pixels in the current frame image that reach a preset brightness threshold;

[0011] Calculate the ratio of the number of pixels reaching the preset brightness threshold in all pixels of the current frame image;

[0012] If both the number and the proportion of the number reach their respective corresponding preset thresholds, it is determined that the current frame image belongs to the first scene;

[0013] Otherwise, it is determined that the current frame image does not belong to the first scene.

[0014] In some embodiments of the present application, the head-mounted display device has a first sound-to-electric conversion device and a second sound-to-electric conversion device; the calculating the relative position between the head-mounted display device and the sound source includes:

[0015] Acquire the distance between the first sound-to-electricity conversion device and the second sound-to-electricity conversion device, and the time difference between the first sound-to-electricity conversion device and the second sound-to-electricity conversion device receiving the sound;

[0016] Calculating the product of the time difference and the speed of sound, and calculating the quotient of the product and the distance;

[0017] The arc cosine value of the quotient is calculated to obtain a positional relationship angle; the positional relationship angle is used to characterize the relative position between the head mounted display device and the sound source.

[0018] In some embodiments of the present application, the updating of the posture data of the head mounted display device using the current frame image and the relative position includes:

[0019] Based on the current frame image, an average value of the angle used to characterize the relative position and the rotation angle of the head mounted display device is calculated, and the rotation angle in the posture data is updated with the average value.

[0020] In some embodiments of the present application, the method further includes:

[0021] If the current frame image does not belong to the first scene, determining whether the current frame image belongs to the second scene according to a second preset determination condition;

[0022] If the current frame image belongs to the second scene, obtaining the speed and displacement of the head mounted display device according to the acceleration of the head mounted display device;

[0023] The position and posture data of the head mounted display device is updated using the current frame image and the speed and displacement.

[0024] In some embodiments of the present application, judging whether the current frame image belongs to the second scene according to the second preset condition includes:

[0025] Acquire all feature pixels in the current frame image;

[0026] If the number of the characteristic pixel points is less than a preset number threshold, it is determined that the current frame image belongs to the second scene; otherwise, it is determined that the current frame image does not belong to the second scene.

[0027] In some embodiments of the present application, acquiring the velocity and displacement of the head-mounted display device according to the acceleration of the head-mounted display device includes:

[0028] Performing time-integrated calculation on the acceleration of the head mounted display device to obtain a velocity of the head mounted display device;

[0029] The speed of the head mounted display device is integrated over time to obtain the displacement of the head mounted display device.

[0030] In some embodiments of the present application, the updating of the posture data of the head mounted display device by using the current frame image and the speed and displacement includes:

[0031] Based on the current frame image, the old speed and the old displacement in the posture data are respectively updated using the speed and the displacement.

[0032] According to another aspect of an embodiment of the present application, a positioning device for a head mounted display device is provided, comprising:

[0033] A receiving module, used for receiving images transmitted by a camera device of the head mounted display device;

[0034] A judgment module, used for judging whether the current frame image belongs to the first scene according to a first preset judgment condition;

[0035] an acquisition module, configured to calculate a relative position between the head mounted display device and a sound source if the current frame image belongs to a first scene;

[0036] An updating module is used to update the posture data of the head mounted display device using the current frame image and the relative position.

[0037] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a positioning method for a head-mounted display device as described in any one of the above.

[0038] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and the program is executed by a processor to implement the positioning method of the head-mounted display device described in any one of the above items.

[0039] The technical solution provided by one aspect of the embodiments of the present application may have the following beneficial effects:

[0040] The positioning method of a head-mounted display device provided in an embodiment of the present application receives an image transmitted by a camera device of the head-mounted display device, determines whether a current frame image belongs to a first scene according to a first preset judgment condition, and if the current frame image belongs to the first scene, calculates the relative position between the head-mounted display device and a sound source, and updates the posture data of the head-mounted display device using the current frame image and the relative position, thereby greatly improving the positioning accuracy of the head-mounted display device and can well meet the needs of practical applications.

[0041] Other features and advantages of the present application will be described in the subsequent description, and some of them will become obvious from the description, or some of them can be inferred or determined unambiguously from the description, or can be understood by implementing the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flow chart of a positioning method for a head mounted display device according to an embodiment of the present application is shown;

[0044] Figure 2 A schematic diagram showing a first scenario of an embodiment of the present application is shown;

[0045] Figure 3 A flowchart showing an implementation method of determining whether the current frame image belongs to the first scene according to the first preset condition;

[0046] Figure 4 A schematic diagram showing a method of determining whether a current frame image satisfies a condition of a first scene in one embodiment;

[0047] Figure 5 A flowchart of calculating the relative position between the head mounted display device and the sound source in one embodiment of the present application is shown;

[0048] Figure 6 A schematic diagram showing a method of obtaining a relative position between a head mounted display device and a sound source in one embodiment of the present application is shown;

[0049] Figure 7A schematic diagram showing a second scenario of an embodiment of the present application is shown;

[0050] Figure 8 A flowchart showing an implementation method of determining whether the current frame image belongs to the second scene according to a second preset condition;

[0051] Fig. 9 A schematic diagram showing an implementation method for determining whether a current frame image meets the conditions of a second scene;

[0052] Fig.10 A flow chart showing how to obtain the velocity and displacement of the head mounted display device according to the acceleration of the head mounted display device in one embodiment of the present application is shown;

[0053] Fig.11 A structural block diagram of a positioning device for a head mounted display device according to an embodiment of the present application is shown;

[0054] Fig.12 A structural block diagram of an electronic device according to an embodiment of the present application is shown;

[0055] Fig.13 A schematic diagram of a computer-readable storage medium according to an embodiment of the present application is shown.

[0056] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0059] VR devices include head-mounted display devices, which include display screens, camera devices, sound-to-electricity conversion devices, processors, gyroscopes and other components.

[0060] like Figure 1As shown, an embodiment of the present application provides a positioning method for a head mounted display device, comprising:

[0061] S10: Receive an image transmitted by a camera device of the head mounted display device.

[0062] The camera device of the head mounted display device transmits the real-time captured images to the processor. The camera device can be, for example, a camera or other device.

[0063] S20: Determine whether the current frame image belongs to the first scene according to a first preset determination condition.

[0064] like Figure 2 As shown, the first scene may be, for example, a scene in which a user uses a VR device in a living room environment (i.e., a living room scene). For example, when a user wears a head-mounted display device to play games in the living room at home, there is a TV or a projector in the living room. At this time, the user and the head-mounted display device are relatively close to the TV or projector panel, resulting in that most of the HMD Camera screen is filled with the screen of the TV or projector panel.

[0065] For example, the first preset condition can be set to that the current frame image meets the Light (图片存在亮度区域>r) &LightArea (区域>n) When this condition is met, the current frame image belongs to the first scene. (图片存在亮度区域>r) Indicates that the number of pixels in the current frame whose brightness is greater than the preset brightness threshold is greater than r; LightArea (区域>n) It represents that the number of pixels in the current frame image whose brightness is greater than the preset brightness threshold accounts for more than n. That is, the first preset condition may be that the number of pixels in the current frame image whose brightness is greater than the preset brightness threshold is greater than r and the number of pixels in the current frame image whose brightness is greater than the preset brightness threshold accounts for more than n. The values ​​of r and n can be set according to actual needs. For example, r can be 100, n can be 40%, or other values ​​can be set according to actual needs.

[0066] In some embodiments, f(x, y) = i(x, y)r(x, y), where the closer the f value is to 0, the lower the brightness, and the closer the f value is to 255, the higher the brightness, where x and y represent the horizontal and vertical coordinates of the pixel, respectively, i represents the incident component of the pixel, and r represents the reflected component of the pixel, f(i, j) = a*g(i, j) + b, where a is a preset image gain, b is a preset image bias, g(i, j) represents the original image pixel, and f(i, j) represents the existing image pixel. By calculating the value of f(i, j) of each pixel in the image, the brightness value of each pixel in the image can be obtained.

[0067] like Figure 3As shown, in some implementations, judging whether the current frame image belongs to the first scene according to the first preset condition includes steps S201 to S203:

[0068] S201. Obtain the number of pixels in the current frame image that reach a preset brightness threshold.

[0069] The brightness values ​​of all pixels in the current frame image are obtained, and compared with the preset brightness thresholds respectively, and the number of pixels reaching the preset brightness threshold is counted.

[0070] S202: Calculate the ratio of the number of pixels reaching the preset brightness threshold in all pixels of the current frame image.

[0071] For example, assuming that there are a pixels in total in the current frame image, and the number of pixels reaching the preset brightness threshold is b, then the aforementioned number accounts for b / a*100%.

[0072] S203: if the number and the proportion of the number both reach their respective corresponding preset thresholds, determining that the current frame image belongs to the first scene;

[0073] Otherwise, it is determined that the current frame image does not belong to the first scene.

[0074] The number of pixels reaching the preset brightness threshold corresponds to a preset number threshold, and the number ratio corresponds to a preset ratio threshold. When the number of pixels reaching the preset brightness threshold reaches the preset number threshold and the number ratio reaches the preset ratio threshold, it is determined that the current frame image belongs to the first scene.

[0075] For example, assuming that the preset number threshold is 100 and the preset proportion threshold is 40%, when the number of pixels reaching the preset brightness threshold is greater than 100 and the number of pixels reaching the preset brightness threshold accounts for more than 40% of all pixels in the current frame image, it is determined that the current frame image belongs to the first scene.

[0076] In a certain implementation manner, determining whether the current frame image meets the first preset condition specifically includes the following steps:

[0077] like Figure 4 As shown, draw two circles of the same size on the image. The two circles are tangent to each other. The diameter d of the circle is equal to half of the height h of the image. The centers of the two circles are respectively at h / 4 and 3h / 4 away from the left edge of the image. Assuming that the image resolution = h640*w480, Figure 4 The area of ​​the two tangent circles in the image occupies about 50% of the screen pixel area. The binary method is used to determine the brightness value of the points on the circumference of the two tangent circles. After each large circle is checked, it can be quickly determined whether the current frame image meets the conditions of the first scene.

[0078] S30: If the current frame image belongs to the first scene, calculate the relative position between the head mounted display device and the sound source. The sound source may be located at a certain distance outside the head mounted display device.

[0079] In some embodiments, the head mounted display device comprises a first acoustic-to-electric conversion device and a second acoustic-to-electric conversion device; the first acoustic-to-electric conversion device and the second acoustic-to-electric conversion device are symmetrically located on the left and right sides of the head mounted display device; the first acoustic-to-electric conversion device and the second acoustic-to-electric conversion device may be microphones (MIC); Figure 5 As shown, the calculating the relative position between the head mounted display device and the sound source includes:

[0080] S301, obtaining the distance between the first acoustic-to-electric conversion device and the second acoustic-to-electric conversion device, and the time difference between the first acoustic-to-electric conversion device and the second acoustic-to-electric conversion device receiving the sound;

[0081] S302, calculating the product of the time difference and the speed of the sound, and calculating the quotient of the product and the distance;

[0082] Specifically, the quotient calculated in step S302 = time difference * speed of sound / distance;

[0083] S303: Calculate the arccosine value of the quotient to obtain a positional relationship angle; the positional relationship angle is used to characterize the relative position between the head mounted display device and the sound source.

[0084] The relative position may be an angle between a first straight line and a second straight line, wherein the first straight line is a line connecting the first acoustic-to-electric conversion device and the second acoustic-to-electric conversion device, and the second straight line is a line connecting the first acoustic-to-electric conversion device and the sound source. The first acoustic-to-electric conversion device may be, for example, a microphone on the left side of the head mounted display device.

[0085] In some embodiments, the detailed steps of measuring the relative position between the head mounted display device and the sound source by using MIC are as follows:

[0086] like Figure 6 As shown, the first acoustic-to-electric conversion device and the second acoustic-to-electric conversion device are microphones a and b respectively; according to the geometric relationship, the positional relationship angle between the position of the sound source Z and the HMD device can be calculated. There are two microphones, microphone a and microphone b, on the HMD device, where c is the propagation speed of the sound signal in the air (346 m / s at a temperature of 25°), and m is the distance between the two microphones a and b. is the positional relationship angle between the HMD and the sound source, is the angle between the straight line ab and the straight line aZ. The calculation formula is as follows (τ is the time difference between microphone a and microphone b receiving the sound):

[0087]

[0088] S40: Update the posture data of the head mounted display device using the current frame image and the relative position.

[0089] For example, step S40 may include:

[0090] Based on the current frame image, an average value of the angle used to characterize the relative position and the rotation angle of the head mounted display device is calculated, and the rotation angle in the posture data is updated with the average value.

[0091] The old position and posture data of the head-mounted display device may be the position and posture data of the head-mounted display device acquired in advance before performing the steps of the positioning method of this embodiment.

[0092] In some implementations, in the first scenario, MIC ranging is mainly used to correct and supplement 6DOF depth information and perform additive fusion on the calculated 6DOF data, thereby achieving pose data update.

[0093] In some embodiments, the relative position of the sound source calculated by the MIC information on the head-mounted display device can be fused into the posture rotation quaternion matrix to further assist the accuracy and smoothness of the posture data output. Fusion method: Calculate the average value of the angle and the rotation angle of the head-mounted display device through the calculated angle between the HMD and the sound source, and use the average value as the angle in the posture, thereby reducing the error and ensuring the smoothness of the posture calculation and posture update. The rotation angle of the head-mounted display device can be measured by the gyroscope of the head-mounted display device.

[0094] In certain embodiments, the method further comprises:

[0095] S50: If the current frame image does not belong to the first scene, determine whether the current frame image belongs to the second scene according to a second preset determination condition.

[0096] like Figure 7 As shown, the second scene is, for example, a user using a VR device in an office or conference room (i.e., a conference room scene). The user wearing the head-mounted display device often faces a solid-color wall, and the user wearing the head-mounted display device approaches or moves away from the solid-color wall.

[0097] like Figure 8 As shown, in some implementations, judging whether the current frame image belongs to the second scene according to the second preset condition includes:

[0098] S501: Acquire all characteristic pixel points in the current frame image.

[0099] The characteristic pixel is the pixel whose absolute value of the difference between the brightness value and the average brightness value in the current frame image is greater than the preset brightness difference threshold. The average brightness value is the average brightness of all pixels in the current frame image. Assuming the preset brightness difference threshold is w, the average brightness of all pixels in the current frame image is p, and the brightness value of one pixel is q, if |qp|>w, then the pixel with the brightness value of q is the characteristic pixel.

[0100] S502: If the number of the characteristic pixel points is less than a preset number threshold, determine that the current frame image belongs to the second scene; otherwise, determine that the current frame image does not belong to the second scene.

[0101] The preset quantity threshold can be set according to actual needs.

[0102] In a certain implementation manner, determining whether the current frame image satisfies the second preset condition mainly involves calculating characteristic pixel points according to a predetermined triangular area, and includes the following steps:

[0103] like Fig. 9 As shown, first determine 5 triangular areas according to the pixel coordinates of the image. The 5 triangles are all equilateral triangles. The center point of the triangle at the center position is the center point of the image, and its coordinates are (h / 2, w / 2), where w is the width of the image. The coordinates of the other 4 triangles are determined, and the brightness value inside each triangular area is judged to obtain the judgment result for entering the next step.

[0104] S60: If the current frame image belongs to the second scene, obtaining a speed and a displacement of the head-mounted display device according to the acceleration of the head-mounted display device.

[0105] like Fig.10 As shown, in some embodiments, obtaining the velocity and displacement of the head mounted display device according to the acceleration of the head mounted display device includes:

[0106] S401, performing time-integrated calculation on the acceleration of the head mounted display device to obtain a velocity of the head mounted display device;

[0107] S402: Perform time-integrated calculation on the velocity of the head-mounted display device to obtain a displacement of the head-mounted display device.

[0108] In some embodiments, obtaining the velocity and displacement of the head mounted display device according to the acceleration of the head mounted display device includes:

[0109] The velocity and displacement are measured by IMU (Inertial Measurement Unit). The velocity information is obtained by integrating the acceleration information with the time parameter. At the same time, the displacement is obtained by integrating the velocity and time. The calculation formula is as follows:

[0110]

[0111]

[0112] D is the area obtained by integral calculation, which is equivalent to the displacement distance, V is the calculated velocity, A represents the acceleration, δt represents the sampling period, and n represents the sampling sequence number.

[0113] S70: Update the posture data of the head mounted display device using the current frame image and the speed and displacement.

[0114] For example, step S70 may include: based on the current frame image, using the speed and displacement to respectively update the old speed and the old displacement in the posture data.

[0115] In some embodiments, in the second scenario, the IMU data is directly used to update the posture information, wherein the 6DOF posture calculation can be calculated using a common spatial positioning algorithm output.

[0116] In some embodiments, the execution order of the two steps of "determining whether the current frame image belongs to the first scene according to the first preset condition" and "determining whether the current frame image belongs to the second scene according to the second preset condition" can be selected according to actual needs.

[0117] In some embodiments, the method of this embodiment further includes:

[0118] S80: If the current frame image belongs neither to the first scene nor to the second scene, an error prompt message is displayed.

[0119] For example, after judgment, it is determined that the current frame image belongs neither to the first scene nor to the second scene, then an error prompt message is displayed on the display screen to remind the user that a scene error has occurred during the application of the head-mounted display device.

[0120] The positioning method of the head-mounted display device provided in the embodiment of the present application receives the image transmitted by the camera device of the head-mounted display device, determines whether the current frame image belongs to the first scene according to the first preset judgment condition, and if the current frame image belongs to the first scene, calculates the relative position between the head-mounted display device and the sound source, and uses the current frame image and the relative position to update the posture data of the head-mounted display device, which greatly improves the positioning accuracy of the head-mounted display device and can well meet the needs of practical applications. The method of the embodiment of the present application can effectively improve the tracking and positioning accuracy and efficiency of the head-mounted display device, reduce system resource consumption, improve resource utilization, and effectively improve user experience.

[0121] like Fig.11 As shown, another embodiment of the present application provides a positioning device for a head mounted display device, comprising:

[0122] A receiving module, used for receiving images transmitted by a camera device of the head mounted display device;

[0123] A judgment module, used for judging whether the current frame image belongs to the first scene according to a first preset judgment condition;

[0124] an acquisition module, configured to calculate a relative position between the head mounted display device and a sound source if the current frame image belongs to a first scene;

[0125] An updating module is used to update the posture data of the head mounted display device using the current frame image and the relative position.

[0126] In some embodiments, the judgment module is further configured to judge whether the current frame image belongs to the second scene according to a second preset judgment condition if the current frame image does not belong to the first scene;

[0127] The acquisition module is further configured to acquire the speed and displacement of the head mounted display device according to the acceleration of the head mounted display device if the current frame image belongs to the second scene;

[0128] The updating module is further used to update the posture data of the head mounted display device using the current frame image and the speed and displacement.

[0129] In some implementations, the determining module determines whether the current frame image belongs to the first scene according to the first preset condition, including:

[0130] Obtaining the number of pixels in the current frame image that reach a preset brightness threshold;

[0131] Calculate the ratio of the number of pixels reaching the preset brightness threshold in all pixels of the current frame image;

[0132] If both the number and the proportion of the number reach their respective corresponding preset thresholds, it is determined that the current frame image belongs to the first scene;

[0133] Otherwise, it is determined that the current frame image does not belong to the first scene.

[0134] In some embodiments, the head mounted display device has a first sound-to-electric conversion device and a second sound-to-electric conversion device; the acquisition module calculates the relative position between the head mounted display device and the sound source, including:

[0135] Acquire the distance between the first sound-to-electricity conversion device and the second sound-to-electricity conversion device, and the time difference between the first sound-to-electricity conversion device and the second sound-to-electricity conversion device receiving the sound;

[0136] Calculating the product of the time difference and the speed of sound, and calculating the quotient of the product and the distance;

[0137] The arc cosine value of the quotient is calculated to obtain a positional relationship angle; the positional relationship angle is used to characterize the relative position between the head mounted display device and the sound source.

[0138] In some embodiments, the updating module updates the posture data of the head mounted display device using the current frame image and the relative position, including:

[0139] Based on the current frame image, an average value of the angle used to characterize the relative position and the rotation angle of the head mounted display device is calculated, and the rotation angle in the posture data is updated with the average value.

[0140] In some implementations, the determining module determines whether the current frame image belongs to the second scene according to the second preset condition, including:

[0141] Acquire all feature pixels in the current frame image;

[0142] If the number of the characteristic pixel points is less than a preset number threshold, it is determined that the current frame image belongs to the second scene; otherwise, it is determined that the current frame image does not belong to the second scene.

[0143] In some embodiments, the acquiring module acquires the velocity and displacement of the head mounted display device according to the acceleration of the head mounted display device, including:

[0144] Performing time-integrated calculation on the acceleration of the head mounted display device to obtain a velocity of the head mounted display device;

[0145] The speed of the head mounted display device is integrated over time to obtain the displacement of the head mounted display device.

[0146] In some embodiments, the updating module performs the updating of the posture data of the head mounted display device using the current frame image and the speed and displacement, including:

[0147] Based on the current frame image, the old speed and the old displacement in the posture data are respectively updated using the speed and the displacement.

[0148] The positioning device of the head-mounted display device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0149] Another embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the positioning method of the head-mounted display device of any of the above-mentioned embodiments.

[0150] like Fig.12 As shown, the electronic device 10 may include: a processor 100, a memory 101, a bus 102 and a communication interface 103, wherein the processor 100, the communication interface 103 and the memory 101 are connected via the bus 102; the memory 101 stores a computer program that can be run on the processor 100, and the processor 100 executes the method provided in any of the aforementioned embodiments of the present application when running the computer program.

[0151] The memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0152] The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 101 is used to store programs, and the processor 100 executes the programs after receiving execution instructions. The method disclosed in any implementation of the above-mentioned embodiment of the present application may be applied to the processor 100, or implemented by the processor 100.

[0153] The processor 100 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 100 or an instruction in the form of software. The above processor 100 may be a general-purpose processor, which may include a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the above method in combination with its hardware.

[0154] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0155] Another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the positioning method of a head-mounted display device of any of the above-mentioned embodiments.

[0156] Please refer to Fig.13 The computer-readable storage medium shown is a CD 20 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided in any of the aforementioned embodiments is executed.

[0157] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0158] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0159] It should be noted that:

[0160] The term "module" is not intended to be limited to a specific physical form. Depending on the specific application, a module can be implemented as hardware, firmware, software, and / or a combination thereof. In addition, different modules can share common components or even be implemented by the same components. There may or may not be clear boundaries between different modules.

[0161] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general devices can also be used together with examples based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the application is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the application described herein, and the description made to specific languages ​​above is for the purpose of disclosing the best mode of implementation of the application.

[0162] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0163] The above-mentioned embodiments only express the implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for positioning a head mounted display device, characterized in that: include: Receiving an image transmitted by a camera device of the head mounted display device; Determine whether the current frame image belongs to the first scene according to a first preset determination condition; If the current frame image belongs to the first scene, calculating the relative position between the head mounted display device and the sound source; Updating the posture data of the head mounted display device using the current frame image and the relative position; The determining, according to the first preset condition, whether the current frame image belongs to the first scene includes: Obtaining the number of pixels in the current frame image that reach a preset brightness threshold; Calculate the ratio of the number of pixels reaching the preset brightness threshold in all pixels of the current frame image; If both the number and the proportion of the number reach their respective corresponding preset thresholds, it is determined that the current frame image belongs to the first scene; Otherwise, determining that the current frame image does not belong to the first scene; The updating of the posture data of the head mounted display device by using the current frame image and the relative position includes: Based on the current frame image, an average value of the angle used to characterize the relative position and the rotation angle of the head mounted display device is calculated, and the rotation angle in the posture data is updated with the average value.

2. The method according to claim 1, characterized in that The head mounted display device comprises a first sound-to-electric conversion device and a second sound-to-electric conversion device; the step of calculating the relative position between the head mounted display device and the sound source comprises: Acquire the distance between the first sound-to-electricity conversion device and the second sound-to-electricity conversion device, and the time difference between the first sound-to-electricity conversion device and the second sound-to-electricity conversion device receiving the sound; Calculate the product of the time difference and the speed of sound, and calculate the quotient of the product and the distance; The arc cosine value of the quotient is calculated to obtain a positional relationship angle; the positional relationship angle is used to characterize the relative position between the head mounted display device and the sound source.

3. The method according to claim 1, characterized in that The method further comprises: If the current frame image does not belong to the first scene, determining whether the current frame image belongs to the second scene according to a second preset determination condition; If the current frame image belongs to the second scene, obtaining the speed and displacement of the head mounted display device according to the acceleration of the head mounted display device; The position and posture data of the head mounted display device is updated using the current frame image and the speed and displacement.

4. The method according to claim 3, characterized in that: The determining, according to the second preset determination condition, whether the current frame image belongs to the second scene includes: Acquire all characteristic pixel points in the current frame image; the characteristic pixel points are pixel points whose absolute value of the difference between the brightness value and the average brightness value in the current frame image is greater than a preset brightness difference threshold; If the number of the characteristic pixel points is less than a preset number threshold, it is determined that the current frame image belongs to the second scene; otherwise, it is determined that the current frame image does not belong to the second scene.

5. The method according to claim 3, characterized in that: The obtaining the velocity and displacement of the head mounted display device according to the acceleration of the head mounted display device comprises: Performing time-integrated calculation on the acceleration of the head mounted display device to obtain a velocity of the head mounted display device; The speed of the head mounted display device is integrated over time to obtain the displacement of the head mounted display device.

6. The method according to claim 3, characterized in that The updating of the posture data of the head mounted display device by using the current frame image and the speed and displacement includes: Based on the current frame image, the old speed and the old displacement in the posture data are respectively updated using the speed and the displacement.

7. A positioning device for a head mounted display device, characterized in that: include: A receiving module, used for receiving images transmitted by a camera device of the head mounted display device; A judgment module, used for judging whether the current frame image belongs to the first scene according to a first preset judgment condition; an acquisition module, configured to calculate a relative position between the head mounted display device and a sound source if the current frame image belongs to a first scene; An updating module, configured to update the posture data of the head mounted display device using the current frame image and the relative position; The judging module is further used for: Obtaining the number of pixels in the current frame image that reach a preset brightness threshold; Calculate the ratio of the number of pixels reaching the preset brightness threshold in all pixels of the current frame image; If both the number and the proportion of the number reach their respective corresponding preset thresholds, it is determined that the current frame image belongs to the first scene; Otherwise, determining that the current frame image does not belong to the first scene; The updating module is further used to: calculate, based on the current frame image, an average value of the angle used to characterize the relative position and the rotation angle of the head mounted display device, and update the rotation angle in the posture data with the average value.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 6.

Images