Method for managing sensor data, host and readable storage medium

Abstract

The invention provides a method for managing sensor data, a host and a readable storage medium. The method includes: obtaining a plurality of first screen frames of a first application and obtaining first sensor data associated with the first screen frames from a first sensor; determining a first foreground portion within each of the first screen frames; determining a first mask associated with the first sensor based on a first foreground portion within each of the first screen frames and first sensor data associated with the first screen frames; obtaining a plurality of second screen frames of the first application and obtaining second sensor data associated with the second screen frames from the first sensor; managing second sensor data associated with the second screen frame using the first mask; and interacting with the first application using the managed second sensor data.

Description

Technical Field

[0001] The present invention generally relates to a mechanism for managing sensor data, and more particularly to a method, a host, and a computer-readable storage medium for managing sensor data. Background Art

[0002] In virtual reality (VR) technology, a user may use devices such as a handheld controller and / or a wearable device (e.g., a smart ring, a smart bracelet, etc.) to interact with a VR application. In such a case, a VR system may measure sensor data provided by the devices and accordingly control / interact with the VR application.

[0003] However, the sensor data may be unreliable and / or undetectable due to problems such as a noisy environment and / or occlusion, such that the user may not be able to correctly control / interact with the VR application. Summary of the Invention

[0004] Accordingly, the present invention relates to a method, a host, and a computer-readable storage medium for managing sensor data that can be used to solve the above technical problems.

[0005] Embodiments of the present invention provide a method for managing sensor data, which is applied to a host and includes: obtaining, by the host, a plurality of first screen frames of a first application and obtaining first sensor data associated with the plurality of first screen frames from a first sensor, wherein the first sensor data associated with the plurality of first screen frames is detected during the display of the plurality of first screen frames; determining, by the host, a first foreground portion in each of the plurality of first screen frames; determining, by the host, a first mask associated with the first sensor based on the first foreground portion in each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; obtaining, by the host, a plurality of second screen frames of the first application and obtaining second sensor data associated with the plurality of second screen frames from the first sensor; managing, by the host, the second sensor data associated with the plurality of second screen frames using the first mask; and interacting, by the host, with the first application using the managed second sensor data.

[0006] An embodiment of the present invention provides a host, which includes a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code to perform: obtaining a plurality of first screen frames of a first application and obtaining first sensor data associated with the plurality of first screen frames from a first sensor, where the first sensor data associated with the plurality of first screen frames is detected during the display of the plurality of first screen frames; determining a first foreground portion in each of the plurality of first screen frames; determining a first mask associated with the first sensor based on the first foreground portion in each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; obtaining a plurality of second screen frames of the first application and obtaining second sensor data associated with the plurality of second screen frames from the first sensor; using the first mask to manage the second sensor data associated with the plurality of second screen frames; and using the managed second sensor data to interact with the first application.

[0007] An embodiment of the present invention provides a computer-readable storage medium that records an executable computer program, and the executable computer program is loaded by a host to perform: obtaining a plurality of first screen frames of a first application and obtaining first sensor data associated with the plurality of first screen frames from a first sensor, where the first sensor data associated with the plurality of first screen frames is detected during the display of the plurality of first screen frames; determining a first foreground portion in each of the plurality of first screen frames; determining a first mask associated with the first sensor based on the first foreground portion in each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; obtaining a plurality of second screen frames of the first application and obtaining second sensor data associated with the plurality of second screen frames from the first sensor; using the first mask to manage the second sensor data associated with the plurality of second screen frames; and using the managed second sensor data to interact with the first application. Description of the Drawings

[0008] The present disclosure includes drawings to provide a further understanding of the present invention, and the drawings are incorporated into and constitute a part of this specification. The drawings illustrate embodiments of the present invention and, together with the description, are used to explain the principles of the present invention.

[0009] Figure 1 A schematic diagram showing a host according to an embodiment of the present invention.

[0010] Figure 2A flowchart showing a method for managing sensor data according to an embodiment of the present invention.

[0011] Figure 3 showing according to Figure 2 another flowchart of a method for managing sensor data.

[0012] Figure 4 A schematic diagram showing the update of second sensor data according to an embodiment of the present invention.

[0013] [Description of symbols]

[0014] 100: Host

[0015] 102: Storage circuit

[0016] 104: Processor

[0017] 410: Second sensor data

[0018] 411, 412, 413~41K, 421, 422, 423~42K: Data segments

[0019] 420: Predicted sensor data

[0020] S210, S220, S230, S240, S250, S260, S310, S320, S330, S340, S350, S360: Steps

[0021] t3, t4: Time points Detailed implementation manners

[0022] Now, reference will be made in detail to the preferred embodiments of the present invention, and examples of the preferred embodiments are shown in the accompanying drawings. As much as possible, the same reference numerals are used in the drawings and the description to refer to the same or similar components.

[0023] See Figure 1 , Figure 1A schematic diagram of a host according to an embodiment of the present invention is shown. In various embodiments, the host 100 can be any intelligent device and / or computer device that can provide visual content for reality services such as virtual reality (VR) services, augmented reality (AR) services, mixed reality (MR) services, and / or extended reality (XR) services, etc., but the present invention is not limited thereto. In some embodiments, the host 100 can be a head-mounted display (HMD) capable of displaying / providing visual content (e.g., AR / VR content) for a wearer / user to view. For a better understanding of the concept of the present invention, it will be assumed that the host 100 is a VR device (e.g., VR HMD) for providing VR content for a user to view, but the present invention is not limited thereto.

[0024] In one embodiment, the host 100 can be provided with a built-in display for displaying VR content for a user to view. Additionally or alternatively, the host 100 can be connected to one or more external displays, and the host 100 can transmit the VR content to the external displays so that the external displays display the VR content, but the present invention is not limited thereto.

[0025] In Figure 1 this, the host 100 includes a storage circuit 102 and a processor 104. The storage circuit 102 is one or a combination of a fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or any other similar device, and the storage circuit 102 records a plurality of modules and / or program codes executable by the processor 104.

[0026] The processor 104 can be coupled to the storage circuit 102, and the processor 104 can be, for example, a general-purpose processor, a dedicated processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and similar devices.

[0027] In an embodiment of the present invention, the processor 104 can access the modules and / or program codes stored in the storage circuit 102 to implement the method for managing sensor data provided in the present invention, and the method will be further discussed below.

[0028] See Figure 2 , Figure 2 which shows a flowchart of a method for managing sensor data according to an embodiment of the present invention. The method of this embodiment can be executed by the host 100 in Figure 1 and the details of each step will be described below in conjunction with the components shown in Figure 1 . Figure 2

[0029] In step S210, the processor 104 obtains a plurality of first screen frames of a first application and obtains first sensor data associated with the plurality of first screen frames from a first sensor.

[0030] In various embodiments of the present invention, the first sensor may include one or more sensors provided on a device (such as a handheld controller and / or a wearable device (such as a smart ring, a smart wristband / smart ankle band, etc.)) connected to the host 100.

[0031] In some embodiments, the one or more sensors may include, for example, an inertial measurement unit (IMU), an optical finger navigation (OFN), an accelerometer, a gyro sensor, etc., and the corresponding first sensor data may be, for example, 6 degrees-of-freedom (6DOF) or the like. In some embodiments, the first sensor may be an environmental sensor (such as a tracking camera or a similar sensor) that applies technologies such as Simultaneous localization and mapping (SLAM), and the corresponding first sensor data may be, for example, the coordinates / attitudes of a device for interacting with the first application.

[0032] In some embodiments, the first application may be a VR application (such as a game application) or a similar application.

[0033] In an embodiment where the host 100 is an HMD, the first screen frame may include a screen frame displayed on the near-eye display of the HMD for the user to view. In another embodiment where the host 100 is, for example, a two-dimensional (2D) game console and is connected to one or more external displays, the first screen frame may include a screen frame provided by the host 100 to the external display for the user to view, but the present invention is not limited thereto.

[0034] In an embodiment of the present invention, the first sensor data associated with the plurality of first screen frames may be detected during the display of the plurality of first screen frames.

[0035] For example, if the first screen frame is displayed during a duration from time point t1 to time point t2, the associated first sensor data may be sensor data measured by the first sensor during the duration from time point t1 to time point t2.

[0036] In one embodiment, the first sensor data may include data segments corresponding to time points during the duration from time point t1 to time point t2, respectively.

[0037] For better understanding of the concept of the present invention, the first sensor data is assumed to be measured during the duration from time point t1 to time point t2, but the present invention is not limited thereto.

[0038] In step S220, the processor 104 determines a first foreground portion within each of the plurality of first screen frames.

[0039] In various embodiments, the processor 104 may use any existing techniques / algorithms to extract the first foreground portion from each of the plurality of first screen frames.

[0040] In some embodiments, the first foreground portion within each of the plurality of first screen frames may be regarded as the portion corresponding to the user within each of the plurality of first screen frames (e.g., a user representative object).

[0041] For example, if the first application is a racing game, each of the first screen frames may include a background portion showing a racecourse and a track in the racecourse and a foreground portion showing a car controlled by the user. In such a case, the first foreground portion within each of the plurality of first screen frames may be an image area corresponding to the car controlled by the user, but the present invention is not limited thereto.

[0042] In another embodiment, if the first application is a racing game, the foreground portion may include a user representative object (e.g., a car), a racecourse, and a track, and the background may include other portions other than the foreground portion. In such a case, the first foreground portion in each of the plurality of first screen frames may be an image area corresponding to the car, the racecourse, and the track, but the present invention is not limited thereto.

[0043] In step S230, the processor 104 determines a first mask associated with the first sensor based on the first foreground portion in each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames.

[0044] In one embodiment, the processor 104 may use a statistical model to characterize the first sensor data. For example, the processor 104 may use a statistical model (e.g., a Gaussian Mixture Model (GMM)) to characterize the first sensor data.

[0045] In one embodiment, the first sensor data may be characterized by, for example, 3-axis coordinate data, and the distribution of the data components on the 3 axes (e.g., the X-axis, the Y-axis, and the Z-axis) may be characterized by a GMM, but the present invention is not limited thereto.

[0046] In one embodiment, since the first foreground portion corresponding to each first screen frame will be configured with a default movement range, the processor 104 may determine a first numerical range based on the statistical model and the default movement range of the first foreground portion in each of the plurality of first screen frames.

[0047] For example, if the first foreground portion corresponds to a virtual car in the first application, for example, the virtual car may only be allowed to move within the corresponding default movement range (e.g., the track). In such a case, the first sensor data attempting to move the virtual car anywhere outside the track may be considered invalid / meaningless / unreliable.

[0048] Since the virtual car generally moves horizontally in the first application, the change in the data component on the X-axis of the first sensor data will be greater than the change in the data components on the Y / Z axes of the first sensor data.

[0049] In one embodiment, when the virtual car has moved to the first boundary (e.g., the left boundary) of the track, the processor 104 may determine the X component of the corresponding first sensor data as the first limit (e.g., the lower limit). Similarly, when the virtual car has moved to the second boundary (e.g., the right boundary) of the track, the processor 104 may determine the X component of the corresponding first sensor data as the second limit (e.g., the upper limit). In such a case, the X component data that is not within the numerical range between the first limit and the second limit will be regarded as invalid / meaningless / unreliable.

[0050] Therefore, the processor 104 may determine the numerical range between the first limit and the second limit as the first numerical range, but the present invention is not limited thereto.

[0051] In one embodiment, the processor 104 may determine the first numerical range as the first mask. In some embodiments, the sensor data provided by the first sensor may be unreliable due to, for example, environmental noise and / or interference. In such a case, the processor 104 may use the first mask to determine whether the sensor data is valid / reliable and accordingly filter the sensor data provided by the first sensor.

[0052] Specifically, in step S240, the processor 104 obtains a plurality of second screen frames of the first application and obtains second sensor data associated with the plurality of second screen frames from the first sensor.

[0053] In an embodiment where the host 100 is an HMD, the second screen frames may include the screen frames displayed on the near-eye display of the HMD for the user to view. In another embodiment where the host 100 is connected to one or more external displays, the second screen frames may include the screen frames provided by the host 100 to the external displays for the user to view, but the present invention is not limited thereto.

[0054] In an embodiment of the present invention, the second sensor data associated with the plurality of second screen frames may be detected during the display of the plurality of second screen frames.

[0055] For example, if the second screen frames are displayed during the duration from time point t3 to time point t4, the associated second sensor data may be the sensor data measured by the first sensor during the duration from time point t3 to time point t4.

[0056] In one embodiment, the second sensor data may include data segments corresponding to time points during the duration from time point t3 to time point t4 respectively.

[0057] As mentioned above, the second sensor data may be unreliable due to noise and / or interference. Thus, in step S250, the processor 104 uses a first mask (e.g., a first numerical range) to filter the second sensor data associated with the plurality of second screen frames.

[0058] In one embodiment, the processor 104 may determine whether the second sensor data includes outlier data that is not within the first numerical range. Responsive to determining that the second sensor data includes outlier data that is not within the first numerical range, the outlier data is filtered out from the second sensor data. Thus, the unreliable portion (e.g., outlier data) of the second sensor data will be filtered out.

[0059] In step S260, the processor 104 uses the managed second sensor data to interact with a first application. Since the unreliable portion of the second sensor data has been filtered out, the processor 104 can better control / interact with the first application such that the user experience will not be affected by a noisy / interfering environment.

[0060] In some embodiments, the processor 104 may determine different masks for different application scenarios and use the corresponding mask to filter the sensor data for the corresponding application scenario.

[0061] For example, in one embodiment, the processor 104 may determine whether a first application scenario corresponds to a second application scenario before performing step S250, where the first application scenario corresponds to the plurality of first screen frames and the second application scenario corresponds to the plurality of second screen frames.

[0062] In one embodiment, responsive to determining that the first application scenario corresponds to the second application scenario, the processor 104 may perform step S250 to filter the second sensor data associated with the plurality of second screen frames using the first mask. On the other hand, responsive to determining that the first application scenario does not correspond to the second application scenario, the processor 104 may filter the second sensor data associated with the plurality of second screen frames using a second mask corresponding to the second application scenario.

[0063] For example, if both the first application scenario and the second application scenario are determined to be a race track, the processor 104 may determine that the first application scenario corresponds to the second application scenario and use the first mask to filter the second sensor data.

[0064] However, if the first application scenario and the second application scenario are determined to be a car racing track and an airplane racing track respectively, the processor 104 may determine that the first application scenario does not correspond to the second application scenario and does not use the first mask to filter the second sensor data. Instead, the processor 104 may obtain a second mask corresponding to the second application scenario and use the second mask to filter the second sensor data.

[0065] In an embodiment of the present invention, the second mask may be determined by following a process similar to the process described in steps S210 to S230, which will not be repeated herein.

[0066] In some embodiments, the first mask may be dedicated to the first application and only used to filter the sensor data provided by the first sensor. That is, the first mask will not be applied to other applications and / or the sensor data provided by other sensors.

[0067] In one embodiment, the processor 104 may perform steps S250 and S260 only when the signal quality is poor due to noise and / or interference.

[0068] Specifically, before step S250, the processor 104 may further determine whether the signal quality of the second sensor data meets a predetermined condition. In some embodiments, the signal quality of the second sensor data may be characterized by, for example, the signal-to-noise ratio (SNR) or the signal-to-noise-and-interference ratio (SINR) of the second sensor data, but the present invention is not limited thereto.

[0069] In one embodiment, in response to determining that the signal quality of the second sensor data does not meet the predetermined condition, the processor 104 may perform step S250 to filter the second sensor data associated with the plurality of second screen frames using the first mask. For example, if the processor 104 determines that the SINR of the second sensor data is less than a predetermined threshold, the processor 104 may determine that the signal quality of the second sensor data does not meet the predetermined condition and accordingly perform step S250.

[0070] In another embodiment, in response to determining that the signal quality of the second sensor data meets the predetermined condition, the processor 104 may maintain the second sensor data associated with the plurality of second screen frames and use the maintained second sensor data to interact with the first application.

[0071] For example, if the processor 104 determines that the SINR of the second sensor data is not less than a predetermined threshold, the processor 104 may maintain the second sensor data associated with the plurality of second screen frames and use the maintained second sensor data to interact with the first application, but the present invention is not limited thereto.

[0072] In some embodiments, some of the second sensor data in the second sensor data may be lost / undetectable due to, for example, occlusion, and embodiments of the present invention have provided solutions to address such problems. This will be discussed in conjunction with Figure 3 the details.

[0073] See Figure 3 , Figure 3 which shows another flowchart of a method for managing sensor data according to Figure 2 . In this embodiment, the processor 104 may also perform steps S310 to S330 before step S240.

[0074] In step S310, the processor 104 obtains a plurality of first user images associated with the first sensor data, where the plurality of first user images are captured during the detection of the first sensor data.

[0075] In an embodiment where the first sensor data is assumed to be detected during a duration from time point t1 to time point t2, the first user image may be a user image captured during the duration from time point t1 to time point t2, but the present invention is not limited thereto.

[0076] In embodiments of the present invention, one or more external cameras may be set in the environment of the reality service (e.g., VR service) of the user experience host 100, and the external cameras may capture images of the user experiencing the reality service (e.g., VR service) as user images. In some embodiments, the user image may be a tracking image of the user's body, but the present invention is not limited thereto.

[0077] Next, in step S320, the processor 104 determines training data by labeling the plurality of first user images using the associated first sensor data, and in step S330, trains the prediction model using the training data. In various embodiments, the prediction model may be a neural network (e.g., convolutional neural network (CNN), recurrent neural network (RNN), etc.) or any other machine learning model, but the present invention is not limited thereto.

[0078] In an embodiment of the present invention, the prediction model can learn the correspondence between user images and sensor data during the training process. That is, when the user in the user image presents a specific pose, the prediction model can learn about the sensor data. In such a case, when the trained prediction model receives another user image, the trained prediction model can generate / output / predict the corresponding sensor data, but the present invention is not limited thereto.

[0079] Next, the processor 104 can execute steps S240 and S250, and then execute steps S340 to S360. In another embodiment, the processor 104 can also first execute steps S340 to S360, and then execute steps S240 and S250. In some embodiments, the processor 104 can execute steps S240, S250, S340 to S360 in any order based on the requirements of the designer, and some of these steps can even be executed simultaneously, but the present invention is not limited thereto.

[0080] In step S340, the processor 104 obtains a plurality of second user images associated with the second sensor data.

[0081] In an embodiment where the second sensor data is assumed to be detected during the duration from time point t3 to time point t4, the second user image can be a user image captured by an external camera during the duration from time point t3 to time point t4, but the present invention is not limited thereto.

[0082] In step S350, the processor 104 uses the prediction model to determine the predicted sensor data based on the plurality of second user images. In an embodiment of the present invention, the processor 104 can input each of the second user images into the prediction model, and the prediction model can output / generate / predict the predicted sensor data corresponding to the second user image.

[0083] In an embodiment where the second user image is assumed to be captured during the duration from time point t3 to time point t4, the prediction model can output the predicted sensor data corresponding to the duration from time point t3 to time point t4.

[0084] In one embodiment, the predicted sensor data can include data segments corresponding to time points during the duration from time point t3 to time point t4, respectively.

[0085] In step S360, the processor 104 updates the second sensor data by replacing a first data segment of the second sensor data with a second data segment of the predicted sensor data. In some embodiments, the processor 104 can be regarded as updating the second sensor data by filling / supplementing the first data segment of the second sensor data with the second data segment of the predicted sensor data. In some embodiments, step S360 can be implemented when the second sensor data is determined to be abnormal / missing, but the present invention is not limited thereto. To better understand the concept of step S360, Figure 4 will be used as an illustrative example.

[0086] See Figure 4 , Figure 4 which shows a schematic diagram of updating the second sensor data according to an embodiment of the present invention.

[0087] In Figure 4 , it is assumed that the second sensor data 410 is measured during the duration from time point t3 to time point t4 and includes data segments 411 to 41K, where the data segments 411 to 41K respectively correspond to K (e.g., K is an integer) time points during the duration from time point t3 to time point t4.

[0088] In this embodiment, when the processor 104 obtains second user images respectively corresponding to the K time points, the processor 104 can input each second user image into the trained prediction model, and the prediction model can output the predicted sensor data 420 in response to the second user image.

[0089] In Figure 4 , the predicted sensor data 420 includes data segments 421 to 42K, where the data segments 421 to 42K respectively correspond to the K time points during the duration from time point t3 to time point t4. From another perspective, the data segments 421 to 42K in the predicted sensor data 420 respectively correspond to the data segments 411 to 41K in the second sensor data 410.

[0090] In an embodiment of the present invention, when the processor 104 intends to update the second sensor data 410, the processor 104 can use the corresponding second data segment among the data segments 421 to 42K to replace the first data segment among the data segments 411 to 41K.

[0091] For example, if the processor 104 determines that a data segment 411 in the second sensor data 410 needs to be replaced, the processor 104 may use the corresponding data segment 421 in the predicted sensor data 420 to replace the data segment 411. For another example, if the processor 104 determines that a data segment 41K in the second sensor data 410 needs to be replaced, the processor 104 may use the corresponding data segment 42K in the predicted sensor data 420 to replace the data segment 41K.

[0092] In one embodiment, the first data segment in the second sensor data 410 may be a data segment determined to be lost / missing / anomalous. For example, if the processor 104 determines that the data segment 412 in the second sensor data 410 is lost / missing / anomalous, the processor 104 may determine that the data segment 412 needs to be replaced / filled / supplemented, and use the data segment 422 in the predicted sensor data 420 to replace / fill / supplement the data segment 412, as Figure 4 shown. In such a case, the updated second sensor data 410 includes data segments 411, 422, 413 to 41K.

[0093] Next, the processor 104 may use the updated second sensor data 410 to perform step S260. In one embodiment, since the processor 104 has performed steps S240 and S250 before step S260, some data segments in the data segments of the second sensor data 410 may have been filtered out. In such a case, the processor 104 may use the updated and / or managed second sensor data 410 to perform step S260.

[0094] For example, if the data segment 411 in the second sensor data 410 has been filtered out in step S250, then Figure 4 the resulting second sensor data 410 in may include data segments 422, 413 to 41K, but the present invention is not limited thereto.

[0095] In some embodiments, the processor 104 may perform step S360 only when the data loss rate is too high.

[0096] For example, before performing step S360, the processor 104 may determine whether the data loss rate of the first application is higher than a loss rate threshold.

[0097] In one embodiment, in response to determining that the data loss rate of the first application is higher than the loss rate threshold, the processor 104 may perform step S350 to update the second sensor data by using a second data segment of the predicted sensor data to replace a first data segment of the second sensor data.

[0098] On the other hand, in response to determining that the data loss rate of the first application is not higher than the loss rate threshold, it indicates that the lost data segments in the second sensor data may not affect the interaction with the first application. Therefore, the processor 104 may maintain the second sensor data. That is to say, when the data loss rate is acceptable, the processor 104 may keep the lost data segments in the second sensor data lost, but the present invention is not limited thereto.

[0099] In one embodiment, Figure 2 the method in may be regarded as including a development stage and an operation stage, where the development stage includes steps S210 to S230, and the operation stage includes steps S240 to S260. The development stage may be implemented before the user actually experiences the scenario of the first application, and the operation stage may correspond to the scenario where the user actually experiences the first application.

[0100] In another embodiment, Figure 2 the method in may be regarded as including a first operation stage and a second operation stage, where the first operation stage includes steps S210 to S230, and the second operation stage includes steps S240 to S260. The first operation stage may be the scenario where the user previously experienced the first application, and the second operation stage may be the scenario where the user currently experiences the first application, but the present invention is not limited thereto.

[0101] In some embodiments, Figure 2 step S250 in may be understood as corresponding to a first process. In Figure 3 a second process involving steps S310 to S360 may be combined into Figure 2 the process shown. That is to say, Figure 3 the process in may be regarded as involving a first process and a second process. In another embodiment, Figure 3 by removing step S250 from Figure 3 can be modified to only involve the second process, but the present invention is not limited thereto.

[0102] The present invention also provides a computer-readable storage medium for executing the method for managing sensor data. The computer-readable storage medium consists of a plurality of program instructions included therein (for example, setting program instructions and deployment program instructions). These program instructions can be loaded into the host 100 and executed by the host 100 to execute the above method for managing sensor data and the functions of the host 100.

[0103] In summary, the embodiments of the present invention provide a solution for determining a first mask based on the first foreground in each first screen frame and the corresponding first sensor data and using the first mask to filter out abnormal data in, for example, the second sensor data. In such a case, the second sensor data may be less affected by noise and / or interference in the environment.

[0104] In addition, embodiments of the present invention can also train a prediction model for determining corresponding predicted sensor data based on a user image. In such a case, when one or more data segments of the second sensor data are determined to be missing, the corresponding data segments in the predicted sensor data can be used to replace / fill / supplement the missing data segments. Therefore, the second sensor data can be less affected by, for example, occlusion problems.

[0105] It will be apparent to those skilled in the art that various modifications and changes can be made to the structure of the present invention without departing from the scope or spirit of the present invention. In summary, the present invention is intended to cover modifications and variations of the present invention that fall within the scope of the following claims and their equivalents.

Claims

1. A method for managing sensor data, applied to a host, characterized in that: The method comprises: obtaining, by the host, a plurality of first screen frames of a first application and obtaining, from a first sensor, first sensor data associated with the plurality of first screen frames, wherein the first sensor data associated with the plurality of first screen frames is detected during display of the plurality of first screen frames; determining, by the host, a first foreground portion within each of the plurality of first screen frames; determining, by the host, a first mask associated with the first sensor based on the first foreground portion within each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; obtaining, by the host, a plurality of second screen frames of the first application and obtaining, from the first sensor, second sensor data associated with the plurality of second screen frames; The host manages the second sensor data by performing at least one of a first process and a second process, wherein the first process includes managing the second sensor data associated with the plurality of second screen frames using the first mask by the host, and the second process includes: obtaining a plurality of first user images associated with the first sensor data, wherein the plurality of first user images are captured during detection of the first sensor data; determining training data by labeling the plurality of first user images using the associated first sensor data; Using the training data to train the prediction model; obtaining a plurality of second user images associated with the second sensor data; determining predicted sensor data based on the plurality of second user images using the prediction model; updating the second sensor data by replacing a first data segment of the second sensor data with a second data segment of the predicted sensor data, wherein the second data segment corresponds to the first data segment; and The managed second sensor data is used by the host to interact with the first application.

2. The method according to claim 1, wherein the first mask is characterized by a first range of values, and the step of using the first mask to manage the second sensor data associated with the plurality of second screen frames comprises: In response to determining that the second sensor data includes abnormal data that is not within the first value range, the abnormal data is filtered out from the second sensor data.

3. The method according to claim 1, wherein before the step of using the first mask to manage the second sensor data associated with the plurality of second screen frames, the method further comprises: Determining whether a first application scenario corresponds to a second application scenario, wherein the first application scenario corresponds to the plurality of first screen frames, and the second application scenario corresponds to the plurality of second screen frames; as well as In response to determining that the first application scenario corresponds to the second application scenario, the second sensor data associated with the plurality of second screen frames is filtered using the first mask.

4. The method according to claim 3, further comprising: In response to determining that the first application scenario does not correspond to the second application scenario, the second sensor data associated with the plurality of second screen frames is filtered using a second mask corresponding to the second application scenario.

5. The method of claim 1 , wherein the step of determining the first mask associated with the first sensor based on the first foreground portion within each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames comprises: characterizing the first sensor data using a statistical model; determining a first value range based on the statistical model and a default movement range of the first foreground portion within each of the plurality of first screen frames; as well as The first value range is determined as the first mask.

6. The method according to claim 1, wherein before the step of updating the second sensor data, the method further comprises: in response to determining that the data loss rate of the first application is above a loss rate threshold, updating the second sensor data by replacing the first data segment of the second sensor data with the second data segment of the predicted sensor data; In response to determining that the data loss rate of the first application is not above the loss rate threshold, maintaining the second sensor data. The method according to claim 1 , wherein the first data segment is a data segment determined to be lost.

8. The method according to claim 1, wherein before the step of using the first mask to manage the second sensor data associated with the plurality of second screen frames, the method further comprises: determining whether the signal quality of the second sensor data meets a predetermined condition; In response to determining that the signal quality of the second sensor data does not satisfy the predetermined condition, managing the second sensor data associated with the plurality of second screen frames using the first mask; In response to determining that the signal quality of the second sensor data satisfies the predetermined condition, the second sensor data associated with the plurality of second screen frames is maintained and the maintained second sensor data is used to interact with the first application.

9. A host, characterized in that: include: A storage circuit for storing program codes; as well as A processor coupled to the memory circuit and accessing the program code to perform: obtaining a plurality of first screen frames of a first application and obtaining first sensor data associated with the plurality of first screen frames from a first sensor, wherein the first sensor data associated with the plurality of first screen frames is detected during display of the plurality of first screen frames; determining a first foreground portion within each of the plurality of first screen frames; determining a first mask associated with the first sensor based on the first foreground portion within each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; obtaining a plurality of second screen frames of the first application and obtaining second sensor data associated with the plurality of second screen frames from the first sensor; The second sensor data is managed by performing at least one of a first process and a second process, wherein the first process includes managing the second sensor data associated with the plurality of second screen frames using the first mask, and the second process includes: obtaining a plurality of first user images associated with the first sensor data, wherein the plurality of first user images are captured during detection of the first sensor data; determining training data by labeling the plurality of first user images using the associated first sensor data; Using the training data to train the prediction model; obtaining a plurality of second user images associated with the second sensor data; determining predicted sensor data based on the plurality of second user images using the prediction model; updating the second sensor data by replacing a first data segment of the second sensor data with a second data segment of the predicted sensor data, wherein the second data segment corresponds to the first data segment; and The managed second sensor data is used to interact with the first application.

10. The host of claim 9, wherein the first mask is characterized by a first range of values, and the processor performs: In response to determining that the second sensor data includes abnormal data that is not within the first value range, the abnormal data is filtered out from the second sensor data.

11. The host of claim 9, wherein before using the first mask to manage the second sensor data associated with the plurality of second screen frames, the processor further performs: determining whether a first application scenario corresponding to the plurality of first screen frames corresponds to a second application scenario corresponding to the plurality of second screen frames; and In response to determining that the first application scenario corresponds to the second application scenario, the second sensor data associated with the plurality of second screen frames is managed using the first mask.

12. The host of claim 11, wherein the processor further performs: In response to determining that the first application scenario does not correspond to the second application scenario, the second sensor data associated with the plurality of second screen frames is managed using a second mask corresponding to the second application scenario.

13. The host of claim 9, wherein the processor performs: characterizing the first sensor data using a statistical model; determining a first value range based on the statistical model and a default movement range of the first foreground portion within each of the plurality of first screen frames; and The first value range is determined as the first mask.

14. The host of claim 9, wherein before updating the second sensor data, the processor further performs: in response to determining that the data loss rate of the first application is above a loss rate threshold, updating the second sensor data by replacing the first data segment of the second sensor data with the second data segment of the predicted sensor data; In response to determining that the data loss rate of the first application is not above the loss rate threshold, maintaining the second sensor data.

15. The host of claim 9, wherein before using the first mask to manage the second sensor data associated with the plurality of second screen frames, the processor further performs: determining whether the signal quality of the second sensor data meets a predetermined condition; In response to determining that the signal quality of the second sensor data does not satisfy the predetermined condition, managing the second sensor data associated with the plurality of second screen frames using the first mask; In response to determining that the signal quality of the second sensor data satisfies the predetermined condition, the second sensor data associated with the plurality of second screen frames is maintained and the maintained second sensor data is used to interact with the first application.

16. A computer-readable storage medium recording an executable computer program, wherein the executable computer program is loaded by a host to perform: obtaining a plurality of first screen frames of a first application and obtaining first sensor data associated with the plurality of first screen frames from a first sensor, wherein the first sensor data associated with the plurality of first screen frames is detected during display of the plurality of first screen frames; determining a first foreground portion within each of the plurality of first screen frames; determining a first mask associated with the first sensor based on the first foreground portion within each of the plurality of first screen frames and the first sensor data associated with the plurality of first screen frames; obtaining a plurality of second screen frames of the first application and obtaining second sensor data associated with the plurality of second screen frames from the first sensor; The second sensor data is managed by performing at least one of a first process and a second process, wherein the first process includes managing the second sensor data associated with the plurality of second screen frames using the first mask, and the second process includes: obtaining a plurality of first user images associated with the first sensor data, wherein the plurality of first user images are captured during detection of the first sensor data; determining training data by labeling the plurality of first user images using the associated first sensor data; Using the training data to train the prediction model; obtaining a plurality of second user images associated with the second sensor data; determining predicted sensor data based on the plurality of second user images using the prediction model; updating the second sensor data by replacing a first data segment of the second sensor data with a second data segment of the predicted sensor data, wherein the second data segment corresponds to the first data segment; and The managed second sensor data is used to interact with the first application.

Images