Methods, host computer, and readable storage media for managing sensor data.
By using the host processor in a virtual reality system to determine the foreground portion of a screen frame, filter out abnormal data, and supplement missing data with a predictive model, the problem of unreliable sensor data is solved, thus improving the user's interactive experience with VR applications.
Patent Information
- Application Number
- CN202410266707.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2023-11-27
- Filing Date
- 2024-03-08
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-03-08
AI Technical Summary
In virtual reality technology, sensor data may be unreliable and undetectable due to noisy environments and occlusion issues, preventing users from properly controlling or interacting with VR applications.
The system obtains sensor data through the host processor, identifies the foreground portion of the screen frame, filters out abnormal data using statistical models and masks, and supplements missing data using predictive models, ensuring the reliability and integrity of the sensor data.
It improves the reliability of sensor data, reduces the impact of noise and occlusion on user interaction, and enhances the control and interaction experience of VR applications.
Smart Images

Figure CN120050318B_ABST
Abstract
Description
Technical Field
[0001] This invention generally relates to a mechanism for managing sensor data, and more specifically to a method, host, and computer-readable storage medium for managing sensor data. Background Technology
[0002] In virtual reality (VR) technology, users can interact with VR applications using devices such as handheld controllers and / or wearable devices (e.g., smart rings, smart bracelets, etc.). In this scenario, the VR system can measure sensor data provided by the devices and accordingly control / interact with the VR application.
[0003] However, sensor data may be unreliable and / or undetectable due to issues such as noisy environments and / or occlusion, which may prevent users from properly controlling or interacting with VR applications. Summary of the Invention
[0004] Therefore, the present invention relates to a method, host, and computer-readable storage medium for managing sensor data that can be used to solve the above-mentioned technical problems.
[0005] Embodiments of the present invention provide a method for managing sensor data, the method being applied to a host and comprising: the host obtaining a plurality of first screen frames of a first application and obtaining first sensor data associated with the plurality of first screen frames from a first sensor, wherein the first sensor data associated with the plurality of first screen frames is detected during the display of the plurality of first screen frames; the host determining a first foreground portion within each of the plurality of first screen frames; the host determining a first mask associated with the first sensor based on the first foreground portion within each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; the host obtaining a plurality of second screen frames of the first application and obtaining second sensor data associated with the plurality of second screen frames from the first sensor; the host using the first mask to manage the second sensor data associated with the plurality of second screen frames; and the host using the managed second sensor data to interact with the first application.
[0006] An embodiment of the present invention provides a host computer including a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code to perform: obtaining a plurality of first screen frames of a first application and obtaining first sensor data associated with the plurality of first screen frames from a first sensor, wherein the first sensor data associated with the plurality of first screen frames is detected during the display of the plurality of first screen frames; determining a first foreground portion within each of the plurality of first screen frames; determining a first mask associated with the first sensor based on the first foreground portion within each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; obtaining a plurality of second screen frames of the first application and obtaining second sensor data associated with the plurality of second screen frames from the first sensor; managing the second sensor data associated with the plurality of second screen frames using the first mask; and interacting with the first application using the managed second sensor data.
[0007] Embodiments of the present invention provide a computer-readable storage medium recording an executable computer program, the executable computer program being loaded by a host to perform: obtaining a plurality of first screen frames of a first application and obtaining first sensor data associated with the plurality of first screen frames from a first sensor, wherein the first sensor data associated with the plurality of first screen frames is detected during the display of the plurality of first screen frames; determining a first foreground portion within each of the plurality of first screen frames; determining a first mask associated with the first sensor based on the first foreground portion within each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; obtaining a plurality of second screen frames of the first application and obtaining second sensor data associated with the plurality of second screen frames from the first sensor; managing the second sensor data associated with the plurality of second screen frames using the first mask; and interacting with the first application using the managed second sensor data. Attached Figure Description
[0008] This document includes accompanying drawings to provide a further understanding of the invention, and the drawings are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
[0009] Figure 1 A schematic diagram of a host computer according to an embodiment of the present invention is shown.
[0010] Figure 2A flowchart illustrating a method for managing sensor data according to an embodiment of the present invention is shown.
[0011] Figure 3 Showing according to Figure 2 Another flowchart of a method for managing sensor data.
[0012] Figure 4 A schematic diagram illustrating the updating of second sensor data according to an embodiment of the present invention is shown.
[0013] [Explanation of Symbols]
[0014] 100: Host
[0015] 102: Storage circuit
[0016] 104: Processor
[0017] 410: Second sensor data
[0018] 411, 412, 413~41K, 421, 422, 423~42K: Data segments
[0019] 420: Predicted sensor data
[0020] S210, S220, S230, S240, S250, S260, S310, S320, S330, S340, S350, S360: Steps
[0021] t3, t4: Time points Detailed Implementation
[0022] The preferred embodiments of the present invention will now be described in detail, examples of which are illustrated in the accompanying drawings. The same reference numerals are used as far as possible in the drawings and description to refer to the same or similar parts.
[0023] See Figure 1 , Figure 1A schematic diagram of a host according to an embodiment of the present invention is shown. In various embodiments, the host 100 may be any intelligent device and / or computer device that can provide visual content for reality services such as virtual reality (VR) services, augmented reality (AR) services, mixed reality (MR) services, and / or extended reality (XR) services, but the invention is not limited thereto. In some embodiments, the host 100 may be a head-mounted display (HMD) capable of displaying / providing visual content (e.g., AR / VR content) for a wearer / user to view. For better understanding of the concept of the invention, it will be assumed that the host 100 is a VR device (e.g., a VR HMD) for providing VR content for a user to view, but the invention is not limited thereto.
[0024] In one embodiment, the host 100 may be equipped with a built-in display for displaying VR content for a user to view. Alternatively, the host 100 may be connected to one or more external displays, and the host 100 may transmit VR content to the external displays for display, but the invention is not limited thereto.
[0025] exist Figure 1 In this system, the host 100 includes a storage circuit 102 and a processor 104. The storage circuit 102 is one or a combination of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or any other similar device, and the storage circuit 102 records multiple modules and / or program code that can be executed by the processor 104.
[0026] Processor 104 may be coupled to storage circuitry 102, and processor 104 may be, for example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and similar devices.
[0027] In embodiments of the present invention, processor 104 may access modules and / or program code stored in storage circuit 102 to implement the method for managing sensor data provided in the present invention, which will be further discussed below.
[0028] See Figure 2 , Figure 2 A flowchart illustrating a method for managing sensor data according to an embodiment of the present invention is shown. The method of this embodiment can be... Figure 1 The host 100 in the middle executes, and the following will be combined with Figure 1 The components shown are used to illustrate Figure 2 Details of each step in the process.
[0029] In step S210, the processor 104 obtains a plurality of first screen frames of the first application and obtains first sensor data associated with the plurality of first screen frames from the first sensor.
[0030] In various embodiments of the present invention, the first sensor may include one or more sensors disposed on a device (e.g., a handheld controller and / or a wearable device (e.g., a smart ring, a smart wristband / smart ankleband, etc.) connected to the host 100.
[0031] In some embodiments, the one or more sensors may include, for example, an inertial measurement unit (IMU), optical finger navigation (OFN), an accelerometer, a gyroscope sensor, etc., and the corresponding first sensor data may be, for example, 6 degrees-of-freedom (6DOF) or similar. In some embodiments, the first sensor may be an environmental sensor (e.g., a tracking camera or similar sensor) that applies technologies such as Simultaneous Localization and Mapping (SLAM), and the corresponding first sensor data may be, for example, the coordinates / attitude of a device used to interact with the first application.
[0032] In some embodiments, the first application may be a VR application (e.g., a game application) or a similar application.
[0033] In an embodiment where the host 100 is an HMD, the first screen frame may include a screen frame displayed on the HMD's near-eye display for the user to view. In another embodiment where the host 100 is, for example, a two-dimensional (2D) game console connected to one or more external displays, the first screen frame may include a screen frame provided by the host 100 to the external display for the user to view, but the invention is not limited thereto.
[0034] In an embodiment of the invention, the first sensor data associated with the plurality of first screen frames may be detected during the display of the plurality of first screen frames.
[0035] For example, if a first screen frame is displayed during the duration from time point t1 to time point t2, the associated first sensor data may be the sensor data measured by the first sensor during the duration from time point t1 to time point t2.
[0036] In one embodiment, the first sensor data may include data segments corresponding to time points within a duration from time point t1 to time point t2.
[0037] To better understand the concept of the present invention, the first sensor data is assumed to be measured over a period of time from time point t1 to time point t2, but the present invention is not limited thereto.
[0038] In step S220, the processor 104 determines a first foreground portion within each of the plurality of first screen frames.
[0039] In various embodiments, the processor 104 may use any existing technology / algorithm to extract a first foreground portion from each of the plurality of first screen frames.
[0040] In some embodiments, the first foreground portion within each of the plurality of first screen frames may be regarded as the portion within each of the plurality of first screen frames corresponding to the user (e.g., a user representative object).
[0041] For example, if the first application is a racing game, each of the first screen frames may include a background portion displaying the racetrack and the track within it, and a foreground portion displaying the car controlled by the user. In this case, the first foreground portion within each of the plurality of first screen frames may be an image area corresponding to the car controlled by the user, but the invention is not limited thereto.
[0042] In another embodiment, if the first application is a racing game, the foreground portion may include user-representative objects (e.g., cars), a racetrack, and a track, and the background may include other portions besides the foreground portion. In this case, the first foreground portion within each of the plurality of first screen frames may be an image area corresponding to the car, racetrack, and track, but the invention is not limited thereto.
[0043] In step S230, the processor 104 determines a first mask associated with the first sensor based on a first foreground portion within each of the plurality of first screen frames and first sensor data associated with the plurality of first screen frames.
[0044] In one embodiment, processor 104 may use a statistical model to characterize the first sensor data. For example, processor 104 may use a statistical model (e.g., a Gaussian Mixture Model, GMM) to characterize the first sensor data.
[0045] In one embodiment, the first sensor data may be characterized by, for example, 3-axis coordinate data, and the distribution of the data components on the 3 axes (e.g., X-axis, Y-axis, Z-axis) may be characterized by GMM, but the invention is not limited thereto.
[0046] In one embodiment, since the first foreground portion corresponding to each first screen frame will be configured with a default movement range, the processor 104 can determine a first numerical range based on a statistical model and the default movement range of the first foreground portion within each of the plurality of first screen frames.
[0047] For example, if the first foreground portion corresponds to, for instance, a virtual car in a first application, then the virtual car may only be allowed to move within its corresponding default range of movement (e.g., a racetrack). In this case, first sensor data attempting to move the virtual car anywhere outside the racetrack can be considered invalid / meaningless / unreliable.
[0048] Since the virtual car typically moves horizontally in the first application, the change in the data component on the X-axis of the first sensor data will be greater than the change in the data component on the Y / Z axes of the first sensor data.
[0049] In one embodiment, when the virtual car has moved to the first boundary of the track (e.g., the left boundary), the processor 104 can determine the X component of the corresponding first sensor data as a first limit (e.g., the lower limit). Similarly, when the virtual car has moved to the second boundary of the track (e.g., the right boundary), the processor 104 can determine the X component of the corresponding first sensor data as a second limit (e.g., the upper limit). In this case, X component data that is not within the numerical range between the first and second limits will be considered invalid / meaningless / unreliable.
[0050] Therefore, the processor 104 can determine the numerical range between the first limit and the second limit as the first numerical range, but the present invention is not limited thereto.
[0051] In one embodiment, the processor 104 may determine a first numerical range as a first mask. In some embodiments, sensor data provided by the first sensor may be unreliable due to, for example, ambient noise and / or interference. In such cases, the processor 104 may use the first mask to determine whether the sensor data is valid / reliable and filter the sensor data provided by the first sensor accordingly.
[0052] Specifically, in step S240, the processor 104 obtains a plurality of second screen frames of the first application and obtains second sensor data associated with the plurality of second screen frames from the first sensor.
[0053] In an embodiment where host 100 is an HMD, the second screen frame may include a screen frame displayed on the near-eye display of the HMD for the user to view. In another embodiment where host 100 is connected to one or more external displays, the second screen frame may include a screen frame provided by host 100 to the external display for the user to view, but the invention is not limited thereto.
[0054] In an embodiment of the invention, the second sensor data associated with the plurality of second screen frames may be detected during the display of the plurality of second screen frames.
[0055] For example, if a second screen frame is displayed during the duration from time point t3 to time point t4, the associated second sensor data could be the sensor data measured by the first sensor during the duration from time point t3 to time point t4.
[0056] In one embodiment, the second sensor data may include data segments corresponding to time points within a duration from time point t3 to time point t4.
[0057] As mentioned above, the second sensor data may be unreliable due to noise and / or interference. Therefore, in step S250, the processor 104 uses a first mask (e.g., a first numerical range) to filter the second sensor data associated with the plurality of second screen frames.
[0058] In one embodiment, the processor 104 may determine whether the second sensor data includes outlier data that is outside the first numerical range. In response to determining that the second sensor data includes outlier data outside the first numerical range, the outlier data is filtered out from the second sensor data. Therefore, unreliable portions of the second sensor data (e.g., outlier data) are filtered out.
[0059] In step S260, the processor 104 uses the managed second sensor data to interact with the first application. Because unreliable portions of the second sensor data have been filtered out, the processor 104 can better control / interact with the first application, ensuring that the user experience is not affected by noisy or interfering environments.
[0060] In some embodiments, the processor 104 may determine different masks for different application scenarios and use the corresponding masks to filter the sensor data for the corresponding application scenario.
[0061] For example, in one embodiment, the processor 104 may determine whether a first application scenario corresponds to a second application scenario before performing step S250, wherein the first application scenario corresponds to the plurality of first screen frames and the second application scenario corresponds to the plurality of second screen frames.
[0062] In one embodiment, in response to determining that a first application scenario corresponds to a second application scenario, processor 104 may perform step S250 to filter the second sensor data associated with the plurality of second screen frames using a first mask. Conversely, in response to determining that a first application scenario does not correspond to a second application scenario, processor 104 may use a second mask corresponding to the second application scenario to filter the second sensor data associated with the plurality of second screen frames.
[0063] For example, if both the first application scenario and the second application scenario are determined to be a car racetrack, the processor 104 can determine that the first application scenario corresponds to the second application scenario and use a first mask to filter the second sensor data.
[0064] However, if the first application scenario and the second application scenario are determined to be a car racetrack and an airplane racetrack, respectively, the processor 104 can determine that the first application scenario does not correspond to the second application scenario and does not use the first mask to filter the second sensor data. Instead, the processor 104 can obtain a second mask corresponding to the second application scenario and use the second mask to filter the second sensor data.
[0065] In embodiments of the present invention, the second mask can be determined by following a process similar to that described in steps S210 to S230, which will not be repeated herein.
[0066] In some embodiments, the first mask may be dedicated to a first application and used only for filtering sensor data provided by the first sensor. That is, the first mask will not be applied to other applications and / or sensor data provided by other sensors.
[0067] In one embodiment, the processor 104 may perform steps S250 and S260 only when the signal quality is poor due to noise and / or interference.
[0068] Specifically, before step S250, the processor 104 may also determine whether the signal quality of the second sensor data meets predetermined conditions. In some embodiments, the signal quality of the second sensor data may be characterized by, for example, the signal-to-noise ratio (SNR) and signal-to-noise-and-interference ratio (SINR) of the second sensor data, but the present invention is not limited thereto.
[0069] In one embodiment, in response to determining that the signal quality of the second sensor data does not meet a predetermined condition, the processor 104 may perform step S250 to filter the second sensor data associated with the plurality of second screen frames using a first mask. For example, if the processor 104 determines that the SINR of the second sensor data is less than a predetermined threshold, the processor 104 may determine that the signal quality of the second sensor data does not meet the predetermined condition and perform step S250 accordingly.
[0070] In another embodiment, in response to determining that the signal quality of the second sensor data meets predetermined conditions, the processor 104 may maintain the second sensor data associated with the plurality of second screen frames and use the maintained second sensor data to interact with the first application.
[0071] For example, if the processor 104 determines that the SINR of the second sensor data is not less than a predetermined threshold, the processor 104 may maintain the second sensor data associated with the plurality of second screen frames and use the maintained second sensor data to interact with the first application, but the invention is not limited thereto.
[0072] In some embodiments, some of the second sensor data may be lost or undetectable due to, for example, occlusion, and embodiments of the present invention have provided solutions to address this problem. (The following text will be combined with...) Figure 3 To elaborate on the details.
[0073] See Figure 3 , Figure 3 Showing according to Figure 2 Another flowchart of a method for managing sensor data. In this embodiment, the processor 104 may also perform steps S310 to S330 prior to step S240.
[0074] In step S310, the processor 104 obtains a plurality of first user images associated with the first sensor data, wherein the plurality of first user images are captured during the detection of the first sensor data.
[0075] In an embodiment where the first sensor data is assumed to have been detected over a period from time t1 to time t2, the first user image may be a user image captured over a period from time t1 to time t2, but the invention is not limited thereto.
[0076] In embodiments of the present invention, one or more external cameras may be set up in the environment of the user experience host 100 for a real-world service (e.g., a VR service), and the external cameras may capture images of the user experiencing the real-world service (e.g., a VR service) as user images. In some embodiments, the user image may be a tracking image of the user's body, but the present invention is not limited thereto.
[0077] Next, in step S320, the processor 104 determines training data by labeling the plurality of first user images using associated first sensor data, and in step S330, the training data is used to train the prediction model. In various embodiments, the prediction model may be a neural network (e.g., a convolutional neural network (CNN), a recurrent neural network (RNN), etc.) or any other machine learning model, but the invention is not limited thereto.
[0078] In embodiments of the present invention, the prediction model can learn the correspondence between user images and sensor data during the training process. That is, when a user in a user image is in a specific pose, the prediction model can learn about the sensor data. In this case, when the trained prediction model receives another user image, the trained prediction model can generate / output / predict the corresponding sensor data, but the present invention is not limited to this.
[0079] Next, processor 104 may execute steps S240 and S250, and then execute steps S340 to S360. In another embodiment, processor 104 may also execute steps S340 to S360 first, and then execute steps S240 and S250. In some embodiments, processor 104 may execute steps S240, S250, S340 to S360 in any order based on the designer's requirements, and some of the steps may even be executed simultaneously, but the invention is not limited thereto.
[0080] In step S340, the processor 104 obtains a plurality of second user images associated with the second sensor data.
[0081] In an embodiment where the second sensor data is assumed to be detected over a period from time t3 to time t4, the second user image may be a user image captured by an external camera over a period from time t3 to time t4, but the invention is not limited thereto.
[0082] In step S350, the processor 104 uses a prediction model to determine predicted sensor data based on the plurality of second user images. In embodiments of the invention, the processor 104 may input each of the second user images into the prediction model, and the prediction model may output / generate / predict predicted sensor data corresponding to the second user image.
[0083] In an embodiment where the second user image is assumed to have been captured over a duration from time point t3 to time point t4, the prediction model can output predicted sensor data corresponding to the duration from time point t3 to time point t4.
[0084] In one embodiment, the predicted sensor data may include data segments corresponding to time points within a duration from time point t3 to time point t4.
[0085] In step S360, processor 104 updates the second sensor data by replacing the first data segment of the second sensor data with the second data segment of the predicted sensor data. In some embodiments, processor 104 can be considered as updating the second sensor data by filling / supplementing the first data segment of the second sensor data with the second data segment of the predicted sensor data. In some embodiments, step S360 may be performed when the second sensor data is determined to be abnormal / missing, but the invention is not limited thereto. To better understand the concept of step S360, the following will be used Figure 4 As an illustrative example.
[0086] See Figure 4 , Figure 4 A schematic diagram illustrating the updating of second sensor data according to an embodiment of the present invention is shown.
[0087] exist Figure 4 In this context, it is assumed that the second sensor data 410 is measured over a duration from time point t3 to time point t4 and includes data segments 411 to 41K, where data segments 411 to 41K correspond to K (e.g., K is an integer) time points over the duration from time point t3 to time point t4.
[0088] In this embodiment, when the processor 104 obtains the second user images corresponding to the K time points respectively, the processor 104 can input each second user image into the trained prediction model, and the prediction model can output predicted sensor data 420 in response to the second user image.
[0089] exist Figure 4 In the predicted sensor data 420, data segments 421 to 42K are included, where data segments 421 to 42K correspond to the K time points within the duration from time point t3 to time point t4. From another perspective, data segments 421 to 42K in the predicted sensor data 420 correspond to data segments 411 to 41K in the second sensor data 410.
[0090] In an embodiment of the present invention, when the processor 104 intends to update the second sensor data 410, the processor 104 may use the second data segment corresponding to the data segments 421 to 42K to replace the first data segment among the data segments 411 to 41K.
[0091] For example, if processor 104 determines that data segment 411 in the second sensor data 410 needs to be replaced, processor 104 can replace data segment 411 with the corresponding data segment 421 in the predicted sensor data 420. In another instance, if processor 104 determines that data segment 41K in the second sensor data 410 needs to be replaced, processor 104 can replace data segment 41K with the corresponding data segment 42K in the predicted sensor data 420.
[0092] In one embodiment, the first data segment in the second sensor data 410 may be a data segment determined to be missing / absent / abnormal. For example, if the processor 104 determines that data segment 412 in the second sensor data 410 is missing / absent / abnormal, the processor 104 may determine that data segment 412 needs to be replaced / filled / supplemented, and use data segment 422 from the predicted sensor data 420 to replace / fill / supplement data segment 412, such as... Figure 4 As shown in the diagram. In this case, the updated second sensor data 410 includes data segments 411, 422, 413 to 41K.
[0093] Next, the processor 104 can use the updated second sensor data 410 to perform step S260. In one embodiment, since the processor 104 has already performed steps S240 and S250 prior to step S260, some data segments in the data segments of the second sensor data 410 may have been filtered out. In this case, the processor 104 can use the updated and / or managed second sensor data 410 to perform step S260.
[0094] For example, if data segment 411 in the second sensor data 410 has been filtered out in step S250, then Figure 4 The second sensor data 410 obtained may include data segments 422, 413 to 41K, but the invention is not limited thereto.
[0095] In some embodiments, the processor 104 may perform step S360 only when the data loss rate is too high.
[0096] For example, before performing step S360, the processor 104 may determine whether the data loss rate of the first application is higher than the loss rate threshold.
[0097] In one embodiment, in response to determining that the data loss rate of the first application is higher than a loss rate threshold, the processor 104 may perform step S350 to update the second sensor data by replacing the first data segment of the second sensor data with a second data segment of the predicted sensor data.
[0098] On the other hand, in response to determining that the data loss rate of the first application is not higher than a loss rate threshold, it means that the lost data segments in the second sensor data do not affect the interaction with the first application. Therefore, the processor 104 can maintain the second sensor data. That is, when the data loss rate is acceptable, the processor 104 can keep the lost data segments in the second sensor data intact, but the invention is not limited to this.
[0099] In one embodiment, Figure 2 The method described herein can be considered to include a development phase and an operation phase, wherein the development phase includes steps S210 to S230, and the operation phase includes steps S240 to S260. The development phase can be performed before the scenario in which the user actually experiences the first application, and the operation phase can correspond to the scenario in which the user actually experiences the first application.
[0100] In another embodiment, Figure 2 The method described herein can be considered to include a first running phase and a second running phase, wherein the first running phase includes steps S210 to S230, and the second running phase includes steps S240 to S260. The first running phase may be a scenario in which the user previously experienced the first application, and the second running phase may be a scenario in which the user is currently experiencing the first application, but the present invention is not limited thereto.
[0101] In some embodiments, Figure 2 Step S250 can be understood as corresponding to the first process. Figure 3 In this process, the second process involving steps S310 to S360 can be combined into Figure 2 In the process shown. That is to say, Figure 3 The process described can be viewed as involving a first process and a second process. In another embodiment, it can be achieved by... Figure 3 Remove step S250 and Figure 3 The modification involves only the second process, but the invention is not limited thereto.
[0102] The present invention also provides a computer-readable storage medium for performing the method for managing sensor data. The computer-readable storage medium comprises a plurality of program instructions (e.g., setting program instructions and deploying program instructions) contained therein. These program instructions can be loaded into and executed by host 100 to perform the method for managing sensor data and the functions of host 100 described above.
[0103] In summary, embodiments of the present invention provide a solution for determining a first mask based on a first foreground in each first screen frame and corresponding first sensor data, and using the first mask to filter out, for example, anomalous data in second sensor data. In this scenario, the second sensor data is less affected by ambient noise and / or interference.
[0104] Furthermore, embodiments of the present invention can also train the prediction model to determine the corresponding predicted sensor data based on user images. In this case, when one or more data segments of the second sensor data are determined to be missing, the corresponding data segments from the predicted sensor data can be used to replace / fill / supplement the missing data segments. Therefore, the second sensor data can be less affected by, for example, occlusion problems.
[0105] It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the invention without departing from the scope or spirit of the invention. In summary, the present invention is intended to cover modifications and variations of the invention that fall within the scope of the following claims and their equivalents.
Claims
1. A method for managing sensor data, applied to a host computer, characterized in that, The method includes: The host obtains a plurality of first screen frames of a first application and obtains first sensor data associated with the plurality of first screen frames from a first sensor, wherein the first sensor data associated with the plurality of first screen frames is detected during the display of the plurality of first screen frames, wherein the first sensor includes an inertial measurement unit and the first sensor data includes 6 degrees of freedom data; The host determines a first foreground portion within each of the plurality of first screen frames; The host determines a first mask associated with the first sensor based on the first foreground portion within each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; The host obtains multiple second screen frames of the first application and obtains second sensor data associated with the multiple second screen frames from the first sensor; The host manages the second sensor data by performing at least one of a first process and a second process, wherein the first process includes the host managing the second sensor data associated with the plurality of second screen frames using the first mask, and the second process includes: Obtain a plurality of first user images associated with the first sensor data, wherein the plurality of first user images are captured during the detection of the first sensor data; Training data is determined by labeling the plurality of first user images using associated first sensor data; The prediction model is trained using the training data. Obtain multiple second user images associated with the second sensor data; The prediction model is used to determine the predicted sensor data based on the plurality of second user images; The second sensor data is updated by replacing the first data segment of the second sensor data with a second data segment of the predicted sensor data, wherein the second data segment corresponds to the first data segment; and The host uses the managed second sensor data to interact with the first application.
2. The method of claim 1, wherein the first mask is characterized by a first numerical range, and the step of using the first mask to manage the second sensor data associated with the plurality of second screen frames includes: In response to determining that the second sensor data includes abnormal data that is not within the first numerical range, the abnormal data is filtered out from the second sensor data.
3. The method of claim 1, wherein prior to the step of using the first mask to manage the second sensor data associated with the plurality of second screen frames, the method further comprises: Determine whether the first application scenario corresponds to the second application scenario, wherein the first application scenario corresponds to the plurality of first screen frames, and the second application scenario corresponds to the plurality of second screen frames; as well as In response to determining that the first application scenario corresponds to the second application scenario, the first mask is used to filter the second sensor data associated with the plurality of second screen frames.
4. The method according to claim 3, further comprising: In response to determining that the first application scenario does not correspond to the second application scenario, a second mask corresponding to the second application scenario is used to filter the second sensor data associated with the plurality of second screen frames.
5. The method of claim 1, wherein the step of determining the first mask associated with the first sensor based on the first foreground portion within each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames comprises: Statistical models are used to characterize the data from the first sensor. The first numerical range is determined based on the statistical model and the default movement range of the first foreground portion within each of the plurality of first screen frames; as well as The first numerical range is determined as the first mask.
6. The method of claim 1, wherein prior to the step of updating the second sensor data, the method further comprises: In response to determining that the data loss rate of the first application is higher than the loss rate threshold, the second sensor data is updated by replacing the first data segment of the second sensor data with the second data segment of the predicted sensor data; In response to determining that the data loss rate of the first application is not higher than the loss rate threshold, the second sensor data is maintained.
7. The method of claim 1, wherein the first data segment is a data segment determined to be lost.
8. The method of claim 1, wherein prior to the step of using the first mask to manage the second sensor data associated with the plurality of second screen frames, the method further comprises: Determine whether the signal quality of the second sensor data meets predetermined conditions; In response to determining that the signal quality of the second sensor data does not meet the predetermined condition, the first mask is used to manage the second sensor data associated with the plurality of second screen frames; In response to determining that the signal quality of the second sensor data satisfies the predetermined condition, the second sensor data associated with the plurality of second screen frames is maintained and the maintained second sensor data is used to interact with the first application.
9. A host computer, characterized in that, include: Storage circuitry stores program code; as well as The processor, coupled to the storage circuitry, accesses and executes the program code: A plurality of first screen frames of a first application are obtained and first sensor data associated with the plurality of first screen frames is obtained from a first sensor, wherein the first sensor data associated with the plurality of first screen frames is detected during the display of the plurality of first screen frames, wherein the first sensor includes an inertial measurement unit and the first sensor data includes 6 degrees of freedom data; Determine a first foreground portion within each of the plurality of first screen frames; A first mask associated with the first sensor is determined based on the first foreground portion within each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; Obtain multiple second screen frames of the first application and obtain second sensor data associated with the multiple second screen frames from the first sensor; The second sensor data is managed by performing at least one of a first process and a second process, wherein the first process includes managing the second sensor data associated with the plurality of second screen frames using the first mask, and the second process includes: Obtain a plurality of first user images associated with the first sensor data, wherein the plurality of first user images are captured during the detection of the first sensor data; Training data is determined by labeling the plurality of first user images using associated first sensor data; The prediction model is trained using the training data. Obtain multiple second user images associated with the second sensor data; The prediction model is used to determine the predicted sensor data based on the plurality of second user images; The second sensor data is updated by replacing the first data segment of the second sensor data with a second data segment of the predicted sensor data, wherein the second data segment corresponds to the first data segment; and The managed second sensor data is used to interact with the first application.
10. The host computer of claim 9, wherein the first mask is characterized by a first numerical range, and the processor performs: In response to determining that the second sensor data includes abnormal data that is not within the first numerical range, the abnormal data is filtered out from the second sensor data.
11. The host according to claim 9, wherein before using the first mask to manage the second sensor data associated with the plurality of second screen frames, the processor further performs: Determine whether the first application scenario corresponding to the plurality of first screen frames corresponds to the second application scenario corresponding to the plurality of second screen frames; and In response to determining that the first application scenario corresponds to the second application scenario, the first mask is used to manage the second sensor data associated with the plurality of second screen frames.
12. The host computer of claim 11, wherein the processor further performs: In response to determining that the first application scenario does not correspond to the second application scenario, a second mask corresponding to the second application scenario is used to manage the second sensor data associated with the plurality of second screen frames.
13. The host computer of claim 9, wherein the processor performs: Statistical models are used to characterize the data from the first sensor. A first numerical range is determined based on the statistical model and the default movement range of the first foreground portion within each of the plurality of first screen frames; and The first numerical range is determined as the first mask.
14. The host computer of claim 9, wherein before updating the second sensor data, the processor further performs: In response to determining that the data loss rate of the first application is higher than the loss rate threshold, the second sensor data is updated by replacing the first data segment of the second sensor data with the second data segment of the predicted sensor data; In response to determining that the data loss rate of the first application is not higher than the loss rate threshold, the second sensor data is maintained.
15. The host according to claim 9, wherein before using the first mask to manage the second sensor data associated with the plurality of second screen frames, the processor further performs: Determine whether the signal quality of the second sensor data meets predetermined conditions; In response to determining that the signal quality of the second sensor data does not meet the predetermined condition, the first mask is used to manage the second sensor data associated with the plurality of second screen frames; In response to determining that the signal quality of the second sensor data satisfies the predetermined condition, the second sensor data associated with the plurality of second screen frames is maintained and the maintained second sensor data is used to interact with the first application.
16. A computer-readable storage medium that records an executable computer program, the executable computer program being loaded by a host computer for execution: A plurality of first screen frames of a first application are obtained and first sensor data associated with the plurality of first screen frames is obtained from a first sensor, wherein the first sensor data associated with the plurality of first screen frames is detected during the display of the plurality of first screen frames, wherein the first sensor includes an inertial measurement unit and the first sensor data includes 6 degrees of freedom data; Determine a first foreground portion within each of the plurality of first screen frames; A first mask associated with the first sensor is determined based on the first foreground portion within each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; Obtain multiple second screen frames of the first application and obtain second sensor data associated with the multiple second screen frames from the first sensor; The second sensor data is managed by performing at least one of a first process and a second process, wherein the first process includes managing the second sensor data associated with the plurality of second screen frames using the first mask, and the second process includes: Obtain a plurality of first user images associated with the first sensor data, wherein the plurality of first user images are captured during the detection of the first sensor data; Training data is determined by labeling the plurality of first user images using associated first sensor data; The prediction model is trained using the training data. Obtain multiple second user images associated with the second sensor data; The prediction model is used to determine the predicted sensor data based on the plurality of second user images; The second sensor data is updated by replacing the first data segment of the second sensor data with a second data segment of the predicted sensor data, wherein the second data segment corresponds to the first data segment; and The managed second sensor data is used to interact with the first application.
Citation Information
Patent Citations
Neural network foreground separation for mixed reality
EP3435284A1
