Method, device, equipment and storage medium for determining posture of head-mounted device

Abstract

The present invention discloses a method, device, equipment and storage medium for determining the posture of a head-mounted device. The method includes: obtaining a current frame image and a previous frame image of the current frame image; performing a feature point extraction operation on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image; wherein the feature point extraction operation includes: extracting the overall feature point set of the image to be processed; extracting the feature point set of a moving object in the image to be processed; removing the feature point set of the moving object from the overall feature point set and outputting the feature point set; and determining whether the posture of the head-mounted device has changed according to the first feature point set and the second feature point set.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a method for determining a posture of a head-mounted device, a device for determining a posture of a head-mounted device, a head-mounted device, and a computer storage medium. Background Art

[0002] At present, VR (Virtual Reality) devices or AR (Augmented Reality) devices have been increasingly widely used in daily life. During the operation of AR or VR devices, it is usually necessary to determine whether their posture has changed. On this basis, the posture tracking of VR / AR devices is further realized.

[0003] When a VR / AR device determines whether its own posture has changed, the specific determination process is: a camera for collecting images of the external environment is set on the VR / AR device; at the current moment, the current frame image is collected by the camera; and feature points in the current frame image and the previous frame image of the current frame image are used to estimate whether the posture of the head-mounted device at the current moment has changed.

[0004] However, the existing methods for determining whether the posture of a VR / AR device has changed often have errors. For example, using the existing methods for determining whether the posture of a VR / AR device has changed, it is determined that the posture of the VR / AR device has changed, but in fact the posture of the VR / AR device has not changed. Summary of the invention

[0005] An object of the present invention is to provide a new technical solution for determining the position of a head mounted device.

[0006] According to a first aspect of the present invention, a method for determining a posture of a head mounted device is provided, comprising:

[0007] Acquire a current frame image and a frame image previous to the current frame image;

[0008] Performing a feature point extraction operation on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image;

[0009] The feature point extraction operation includes: extracting the overall feature point set of the image to be processed; extracting the feature point set of the moving object in the image to be processed; removing the feature point set of the moving object from the overall feature point set, and outputting the feature point set;

[0010] Determine whether the posture of the head mounted device changes according to the first feature point set and the second feature point set.

[0011] Optionally, extracting a moving object feature point set in the image to be processed includes:

[0012] Identifying the type of the moving object in the image to be processed;

[0013] The moving object feature point set is extracted from the image to be processed using a feature point extraction model that matches the type.

[0014] Optionally, the method further includes the step of obtaining a feature point extraction model matching the type, including:

[0015] Acquire a training sample set, wherein the training sample set includes multiple groups of training samples, and one group of the training samples includes a sample image and a feature point set of the type of moving object in the sample image;

[0016] The sample image is used as an input of a feature point extraction model, and a feature point set of the type of moving object in the sample image is used as supervision to train the feature point extraction model to obtain a feature point extraction model matching the type.

[0017] Optionally, before performing a feature point extraction operation on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image, the method further includes:

[0018] Detecting whether the moving object exists in at least one of the current frame image and the previous frame image;

[0019] If yes, the step of performing feature point extraction operations on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image is performed.

[0020] Optionally, the detecting whether the movable object exists in at least one of the current frame image and the previous frame image includes:

[0021] Input each of the current frame image and the previous frame image into different types of moving object recognition models to obtain recognition results;

[0022] According to the recognition result, it is determined whether there is a moving object of one of the different types in the corresponding frame image.

[0023] Optionally, the feature point extraction model is a histogram of oriented gradients algorithm.

[0024] Optionally, the head-mounted device is a virtual reality head-mounted display device, an augmented reality head-mounted display device, or a mixed reality head-mounted display device.

[0025] According to a second aspect of the present invention, a device for determining a posture of a head mounted device is provided, the device comprising:

[0026] An acquisition module, used for acquiring a current frame image and a frame image previous to the current frame image;

[0027] An execution module is used to perform a feature extraction operation on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image; wherein the feature extraction operation includes: extracting an overall feature point set of the image to be processed; extracting a feature point set of a moving object in the image to be processed; and removing the feature point set of the moving object from the overall feature point set to obtain an output feature point set;

[0028] A determination module is used to determine whether the posture of the head mounted device changes according to the first feature point set and the second feature point set.

[0029] According to a third aspect of the present invention, there is provided a head mounted device, the head mounted device comprising the apparatus according to the second aspect; or,

[0030] It comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the method for determining the posture of a head mounted device as described in any one of the first aspects.

[0031] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for determining a posture of a head mounted device according to any one of the first aspects is implemented.

[0032] In this embodiment, by acquiring the current frame image and the previous frame image of the current frame image; performing feature point extraction operations on the current frame image and the previous frame image, a first feature point set of the current frame image that does not contain feature points of moving objects, and a second feature point set of the previous frame image that does not contain feature points of moving objects are obtained. In this way, since the feature points of moving objects have been filtered out in both the first feature point set and the second feature point set, the first feature point set and the second feature point set both contain feature points of objects with fixed positions in the external environment, that is, the first feature point set and the second feature point set can accurately reflect whether the posture of the head-mounted device has changed. In this way, the posture of the head-mounted device determined by using the first feature point set and the second feature point set is accurate, which avoids the occurrence of errors in the results determined by using the traditional method of determining whether the posture of the head-mounted device has changed.

[0033] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0035] Figure 1 is a block diagram of a hardware configuration of a head mounted device for implementing a method for determining a posture of a head mounted device provided in accordance with an embodiment of the present invention;

[0036] Figure 2 is a flow chart of a method for determining a posture of a head mounted device provided by an embodiment of the present invention;

[0037] Figure 3 is a structural schematic diagram of a device for determining a posture of a head-mounted device provided by an embodiment of the present invention;

[0038] Figure 4 It is a structural schematic diagram of a head mounted device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0040] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0041] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0042] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0043] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0044] At present, when VR / AR devices use traditional methods to determine whether their posture has changed, the specific determination process is: set a camera on the VR / AR device to collect images of the external environment; at the current moment, use the camera to collect the current frame image; use the feature points in the current frame image and the previous frame image of the current frame image to determine whether the current frame image and the previous frame image are the same. If they are the same, it is determined that the posture of the head-mounted device at the current moment has not changed, otherwise it has changed.

[0045] However, when the user wearing the VR / AR device remains still and someone moves in the field of view of the camera, the two adjacent frames of images captured by the camera are different. In this way, if the traditional method is continued to be used to determine whether the posture of the VR / AR device has changed, it will be determined that the posture of the VR / AR device has changed. This is obviously inconsistent with the actual situation. In order to solve this problem, a new posture determination method for a head-mounted device provided by the present invention is proposed.

[0046] <Hardware Configuration Example>

[0047] Figure 1 It is a block diagram of the hardware configuration of a head mounted device for implementing a method for determining a posture of a head mounted device provided according to an embodiment of the present invention.

[0048] The head mounted device 1000 may be a virtual reality (VR) head mounted display device, or an augmented reality (AR) head mounted display device, or a mixed reality (MR) head mounted display device. The head mounted device 1000 may be embodied in the form of head mounted glasses, a head mounted helmet, and the like.

[0049] The head mounted device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 may be a central processing unit CPU, a microprocessor MCU, and the like. The memory 1200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, and the like. The interface device 1300 may include, for example, a USB interface, a headphone interface, and the like. The communication device 1400 may be capable of wired or wireless communication, for example. The display device 1500 may be, for example, a liquid crystal display screen, a touch display screen, and the like. The input device 1600 may include, for example, a touch screen, a keyboard, and the like. The user may input / output voice information through the speaker 1700 and the microphone 1800.

[0050] Despite Figure 1 In the figure, multiple devices are shown for the head mounted device 1000, however, the present invention may only involve some of the devices, for example, the head mounted device 1000 only involves the memory 1200 and the processor 1100.

[0051] In the embodiment of the present invention, the memory 1200 of the head mounted device 1000 is used to store instructions, and the instructions are used to control the processor 1100 to execute the method for determining the posture of the head mounted device provided in the embodiment of the present invention.

[0052] In the above description, a technician can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0053] <Method Example>

[0054] An embodiment of the present invention provides a method for determining the position and posture of a head mounted device, the method being applied to Figure 1 The head mounted device 1000 shown in FIG. The method is performed by a posture determination device of the head mounted device. The device may be as follows: Figure 1 The head mounted device 1000 shown can also be Figure 1 The hardware modules and / or software modules in the head mounted device shown. It should be noted that in the following embodiments, the head mounted device will be used as the execution subject of the method for determining the position and posture of the head mounted device to illustrate the method provided by the present application.

[0055] like Figure 2 As shown, an embodiment of the present invention provides a method for determining a posture of a head mounted device, including the following steps S2100-S2300:

[0056] S2100: Acquire a current frame image and a frame image previous to the current frame image.

[0057] In this embodiment, a camera fixed relatively to the head-mounted device is provided on the head-mounted device. The camera is used to collect images of the external environment. The image collected by the camera at the current collection moment is recorded as the current frame image, and the image collected at the previous collection moment of the current moment is recorded as the previous frame image of the current frame image.

[0058] It should be noted that the camera provided on the head-mounted device is usually a camera in a 6DOF system in the head-mounted device.

[0059] S2200 , performing a feature point extraction operation on a current frame image and a previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image.

[0060] The feature point extraction operation includes the following steps S2210-S2230:

[0061] S2210, extracting an overall feature point set of the image to be processed.

[0062] In this embodiment, when the feature point extraction operation is performed on the current frame image, the current frame image is used as the image to be processed. Correspondingly, when the feature point extraction operation is performed on the previous frame image, the previous frame image is used as the image to be processed. In addition, the overall feature point set refers to the set of all feature points in the image to be processed.

[0063] In this embodiment, the specific implementation of the above S2210 may be: extracting feature points from the image to be processed using a feature extraction algorithm; and taking a set consisting of all the extracted feature points as the overall feature point set in the above S2210.

[0064] In one example, the feature extraction algorithm may be an ORB (Oriented Fast and Rotated Brief) feature extraction algorithm. Of course, the feature extraction algorithm may also be other types of algorithms, such as a SIFT (Scale-invariant feature transform) algorithm. The present invention does not limit this.

[0065] S2220, extracting a feature point set of a moving object in the image to be processed.

[0066] In this embodiment, there are moving objects in the image to be processed. The moving objects refer to objects that may move in the external environment. It should be noted that in this embodiment, the number of moving objects in the image to be processed is not limited.

[0067] Exemplarily, the moving object is a person and / or an animal, etc. It should be noted that for different external environments, the corresponding moving objects may be different.

[0068] For example, when the external environment is a living room, the sofa, tables and chairs in the living room are immovable objects, that is, objects with fixed positions. The people in the living room are moving objects. On this basis, the moving objects can be people.

[0069] In one embodiment, the above S2220 may be specifically implemented as follows: S2221 and S2222:

[0070] S2221. Identify the type of moving objects in the image to be processed.

[0071] In one embodiment, since the moving objects in the external environment are usually fixed, the specific implementation of the above S2221 can be: the user manually identifies the moving objects in the external environment, and manually inputs the type of the moving objects into the head-mounted device, which is stored by the head-mounted device. On this basis, when the head-mounted device executes the above S2221, it directly reads the type of the moving object from its own storage. In this embodiment, the head-mounted device does not need to identify the type of the moving object in the image to be processed, which can reduce the computing overhead of the head-mounted device.

[0072] In another embodiment, the above S2221 may be specifically implemented as follows: the head mounted device automatically identifies the type of moving objects in the image to be processed. Specifically:

[0073] The head mounted device inputs the image to be processed into different types of moving object recognition models respectively to obtain recognition results; wherein the recognition result may be whether the moving object in the image to be processed is a moving object that can be recognized by the corresponding type of moving object recognition model, and the moving object recognition model may be a target object recognition algorithm.

[0074] According to the recognition result, determine which type of mobile object in the frame image to be processed can be recognized by the mobile object recognition model; and use the determined type as the type recognized in the above S2221. It should be noted that the type of mobile object that can be recognized by each mobile object recognition model needs to be marked in advance.

[0075] In this embodiment, the head mounted device automatically identifies the type of moving objects in the image to be processed, thereby avoiding human intervention, thereby improving the user experience of the user using the head mounted device.

[0076] S2222. Extract a moving object feature point set from the image to be processed using a feature point extraction model that matches the type.

[0077] In this embodiment, the feature point extraction model matching the type is a model that can extract feature points of a moving object of the corresponding type from the image to be processed.

[0078] In one example, when the moving object is a person, the feature point extraction model matching the type in the above S2222 is a model that can extract feature points corresponding to the person from the image to be processed.

[0079] On the basis of the above S2222, the method for determining the posture of the head mounted device provided in this embodiment further includes the step of obtaining a feature point extraction model matching the type. This step includes the following S2222-1 and S2222-2:

[0080] S2222-1. Obtain a training sample set.

[0081] The training sample set includes multiple groups of training samples, and one group of training samples includes a sample image and a feature point set of a moving object of the type described in the sample image.

[0082] In this embodiment, the type refers to the type corresponding to the moving object. And the feature points in the feature point set of the moving object of the type in the sample are marked in advance by the user.

[0083] S2222-2. Use the sample image as input of the feature point extraction model, and use the feature point set of the type of moving object in the sample image as supervision to train the feature point extraction model to obtain a feature point extraction model that matches the type.

[0084] In this embodiment, the feature point extraction model is trained using sample images until the feature point extraction model converges. The converged feature point extraction model is used as the final model. The final model can extract feature points of the type of moving object in the input image.

[0085] In one embodiment, the feature point extraction model may be a Histogram of Oriented Gradients algorithm, which is referred to as the HOG (Histogram of Oriented Gradient) algorithm.

[0086] S2230, eliminating the feature point set of the moving object from the overall feature point set, and outputting the feature point set.

[0087] In this embodiment, when the image to be processed is the current frame image, the feature point set output after the feature point extraction operation is performed on the image to be processed is recorded as the first feature point set. Correspondingly, when the image to be processed is the previous frame image, the feature point set output after the feature point extraction operation is performed on the image to be processed is recorded as the second feature point set.

[0088] Based on the above S2210-S2230, it can be known that the first feature point set in the current frame image does not include the feature points of the moving object, that is, the first feature point set in the current frame image only includes the feature points of the objects with fixed positions. Correspondingly, the first feature point set in the previous frame image does not include the feature points of the moving object, that is, the second feature point set in the previous frame image only includes the feature points of the objects with fixed positions.

[0089] S2300: Determine whether the posture of the head mounted device changes based on the first feature point set and the second feature point set.

[0090] In this embodiment, the specific implementation of the above S2300 may be: according to the first feature point set and the second feature point set, using the PNP (Perspective-n-Point) algorithm, determining whether the posture of the head mounted device has changed.

[0091] In this embodiment, by acquiring the current frame image and the previous frame image of the current frame image; performing feature point extraction operations on the current frame image and the previous frame image, a first feature point set of the current frame image that does not contain feature points of moving objects, and a second feature point set of the previous frame image that does not contain feature points of moving objects are obtained. In this way, since the feature points of moving objects have been filtered out in both the first feature point set and the second feature point set, the first feature point set and the second feature point set both contain feature points of objects with fixed positions in the external environment, that is, the first feature point set and the second feature point set can accurately reflect whether the posture of the head-mounted device has changed. In this way, the posture of the head-mounted device determined by using the first feature point set and the second feature point set is accurate, which avoids the occurrence of errors in the results determined by using the traditional method of determining whether the posture of the head-mounted device has changed.

[0092] Based on any of the above embodiments, the method for determining the posture of a head mounted device provided in this embodiment further includes the following S2400 and S2500 before the above S2200:

[0093] S2400, detecting whether there is a moving object in at least one of the current frame image and the previous frame image.

[0094] In one embodiment, the above S2400 may be specifically implemented as follows: S2410 and S2420:

[0095] S2410: Input each frame of the current frame image and the previous frame image into different types of moving object recognition models to obtain recognition results.

[0096] In one embodiment, when the current frame image is input into different types of moving object recognition models respectively, the recognition result is whether the moving object in the current frame image is a moving object that can be recognized by the corresponding type of moving object recognition model.

[0097] Correspondingly, when the previous frame image is input into different types of moving object recognition models respectively, the recognition result is whether the moving object in the previous frame image is a moving object that can be recognized by the corresponding type of moving object recognition model.

[0098] The moving object recognition model may be a target object recognition algorithm.

[0099] S2420. Determine whether there is a moving object of one of the different types in the corresponding frame image according to the recognition result.

[0100] In this embodiment, the current frame image is input into different types of moving object recognition models to obtain recognition results. If it is determined according to the recognition results that there is a moving object of one of the different types in the current frame image, it is indicated that there is a moving object in the current frame image. Conversely, if it is determined according to the recognition results that there is no moving object of one of the different types in the current frame image, it is indicated that there is no moving object in the current frame image.

[0101] Correspondingly, the previous frame image is input into different types of moving object recognition models to obtain recognition results. When it is determined according to the results that there is a moving object of one of the different types in the previous frame image, it is indicated that there is a moving object in the previous frame image. Conversely, when it is determined according to the recognition results that there is no moving object of one of the different types in the previous frame image, it is indicated that there is no moving object in the previous frame image.

[0102] S2500: If present, perform a feature point extraction operation on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image.

[0103] Correspondingly, in the absence of such a condition, a traditional method is used to determine whether the posture of the head mounted device has changed.

[0104] In this embodiment, the external environment is usually unstable, that is, at different times, the current frame image collected may or may not include a moving object. On this basis, the head-mounted device first detects whether there is a moving object in at least one of the current frame image and the previous frame image. The above steps S2200 and S2300 are performed only if there is a moving object. In this way, it can be avoided that the head-mounted device still performs operations such as S2200 and S2300 when neither the current frame image nor the previous frame image includes a moving object. This reduces the computational overhead of the head-mounted device.

[0105] It can be understood that when the current frame image or the previous frame image does not include a moving object and the head mounted device still executes the above S2200, the obtained first feature point set or the second feature point set is approximately an empty set.

[0106] <Device Example>

[0107] like Figure 3 As shown, this embodiment provides a device 30 for determining the position and posture of a head mounted device. The device 30 includes: an acquisition module 31, an execution module 32 and a determination module 33. Among them:

[0108] An acquisition module 31 is used to acquire a current frame image and a frame image before the current frame image;

[0109] The execution module 32 is used to perform a feature extraction operation on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image; wherein the feature extraction operation includes: extracting an overall feature point set of the image to be processed; extracting a feature point set of a moving object in the image to be processed; and removing the feature point set of the moving object from the overall feature point set to obtain an output feature point set;

[0110] The determination module 33 is used to determine whether the posture of the head mounted device changes according to the first feature point set and the second feature point set.

[0111] In one embodiment, the execution module 32 is further used to: identify the type of the moving object in the image to be processed;

[0112] The moving object feature point set is extracted from the image to be processed using a feature point extraction model that matches the type.

[0113] In one embodiment, the acquisition module 31 is further used to: acquire a training sample set, the training sample set comprising multiple groups of training samples, one group of the training samples comprising a sample image and a feature point set of the type of moving object in the sample image;

[0114] The sample image is used as an input of a feature point extraction model, and a feature point set of the type of moving object in the sample image is used as supervision to train the feature point extraction model to obtain a feature point extraction model matching the type.

[0115] In one embodiment, the device 30 further includes a detection module, which is used to:

[0116] Detecting whether the moving object exists in at least one of the current frame image and the previous frame image;

[0117] If yes, the step of performing feature point extraction operations on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image is performed.

[0118] In one embodiment, the detection module is further used to: input each of the current frame image and the previous frame image into different types of moving object recognition models to obtain recognition results;

[0119] According to the recognition result, it is determined whether there is a moving object of one of the different types in the corresponding frame image.

[0120] In one embodiment, the feature point extraction model is a histogram of oriented gradients algorithm.

[0121] In one embodiment, the head mounted device is a virtual reality head mounted display device, an augmented reality head mounted display device, or a mixed reality head mounted display device.

[0122] <Equipment Embodiment>

[0123] like Figure 4 As shown, this embodiment provides a head mounted device 40, wherein the head mounted device 40 includes the device 30 as described in the above device embodiment; or,

[0124] It includes a memory 41 and a processor 42, wherein the memory 41 is used to store computer instructions, and the processor 42 is used to call the computer instructions from the memory 41 to execute the method for determining the posture of a head-mounted device as described in any one of the above method embodiments.

[0125] <Storage Medium Embodiment>

[0126] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for determining the posture of a head mounted device according to any one of the method embodiments is implemented.

[0127] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0128] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0129] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0130] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0131] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0134] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0135] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for determining the posture of a head mounted device, characterized in that: Used to detect whether the posture of the head-mounted device has changed, including: Acquire a current frame image and a frame image previous to the current frame image; Performing a feature point extraction operation on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image; The feature point extraction operation includes: extracting the overall feature point set of the image to be processed; extracting the feature point set of the moving object in the image to be processed; removing the feature point set of the moving object from the overall feature point set, and outputting the feature point set; Determining whether the posture of the head mounted device changes according to the first feature point set and the second feature point set; Wherein, before performing the feature point extraction operation on the current frame image and the previous frame image to obtain the first feature point set of the current frame image and the second feature point set of the previous frame image, the method further includes: Detecting whether the moving object exists in at least one of the current frame image and the previous frame image; If yes, the step of performing feature point extraction operations on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image is performed.

2. The method according to claim 1, characterized in that The step of extracting a moving object feature point set from the image to be processed comprises: Identifying the type of the moving object in the image to be processed; The moving object feature point set is extracted from the image to be processed using a feature point extraction model that matches the type.

3. The method according to claim 2, characterized in that The method further comprises the step of obtaining a feature point extraction model matching the type, comprising: Acquire a training sample set, wherein the training sample set includes multiple groups of training samples, and one group of the training samples includes a sample image and a feature point set of the type of moving object in the sample image; The sample image is used as an input of a feature point extraction model, and a feature point set of the type of moving object in the sample image is used as supervision to train the feature point extraction model to obtain a feature point extraction model matching the type.

4. The method according to claim 1, characterized in that: The detecting whether the moving object exists in at least one of the current frame image and the previous frame image comprises: Input each of the current frame image and the previous frame image into different types of moving object recognition models to obtain recognition results; According to the recognition result, it is determined whether there is a moving object of one of the different types in the corresponding frame image.

5. The method according to claim 3, characterized in that: The feature point extraction model is a directional gradient histogram algorithm.

6. The method according to any one of claims 1 to 5, characterized in that: The head mounted device is a virtual reality head mounted display device, an augmented reality head mounted display device or a mixed reality head mounted display device.

7. A device for determining the posture of a head-mounted device, characterized in that: The device is used to detect whether the posture of the head mounted device has changed, and the device includes: An acquisition module, used for acquiring a current frame image and a frame image previous to the current frame image; An execution module is used to perform a feature extraction operation on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image; wherein the feature extraction operation includes: extracting an overall feature point set of the image to be processed; extracting a feature point set of a moving object in the image to be processed; and removing the feature point set of the moving object from the overall feature point set to obtain an output feature point set; A determination module, configured to determine whether the posture of the head mounted device changes according to the first feature point set and the second feature point set; Wherein, the device further comprises: A detection module, used for detecting whether the moving object exists in at least one of the current frame image and the previous frame image; If yes, the step of performing feature point extraction operations on the current frame image and the previous frame image to obtain a first feature point set of the current frame image and a second feature point set of the previous frame image is performed.

8. A head mounted device, characterized in that: The head mounted device comprises the apparatus as claimed in claim 7; or, It comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the posture determination method of the head-mounted device as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for determining the posture of a head mounted device according to any one of claims 1 to 6 is implemented.

Images