Inertial measurement unit data processing method and device and electronic equipment

Abstract

The invention discloses an inertial measurement unit data processing method and device and electronic equipment, and belongs to the technical field of electronic equipment. The method comprises the following steps: acquiring first IMU data, second IMU data and third IMU data of an inertial measurement unit IMU; predicting fourth IMU data corresponding to a fourth moment based on the first IMU data, the second IMU data and the third IMU data; sending the second IMU data and the fourth IMU data to the first application; wherein the second IMU data is IMU data which is transmitted through a first path of the IMU and is to be sent to the first application; the first IMU data is historical IMU data transmitted through a first path; the third IMU data is IMU data which is transmitted through a second path of the IMU and is used for realizing optical anti-vibration; the acquisition moment of the first IMU data is a first moment, the acquisition moment of the second IMU data is a second moment, the acquisition moment of the third IMU data is a third moment, the second moment is later than the first moment, and the third moment is later than the second moment; the fourth moment is later than the third moment.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to an inertial measurement unit data processing method, device and electronic equipment. Background Art

[0002] As a sensor used in electronic devices to detect and measure acceleration, tilt, shock, vibration, rotation, and multi-degree-of-freedom (DoF) motion, an inertial measurement unit (IMU) can be used to assist users in operating applications. For example, in shooting game applications or adventure game applications, IMU devices can be used to accurately detect changes in user hand movements, and the IMU data detected by the IMU device can be transmitted to the application. The application performs a series of processing on the IMU data to calculate the posture information of the device or user, including position, direction, angle, etc., and then adjusts the position and direction of the role, perspective, or object in the game according to the processing results, and then updates the game screen. In this way, the fun and authenticity of the game can be increased.

[0003] However, since the transmission and processing of IMU data requires a certain amount of time, it is difficult for applications to update interactive operation results in a timely manner based on IMU data. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an inertial measurement unit data processing method, device and electronic device, which can improve the timeliness of updating the results of interactive operations performed by applications based on IMU data.

[0005] In a first aspect, an embodiment of the present application provides an inertial measurement unit data processing method, comprising:

[0006] Acquire first IMU data, second IMU data, and third IMU data of an inertial measurement unit IMU;

[0007] Predicting fourth IMU data corresponding to a fourth moment based on the first IMU data, the second IMU data, and the third IMU data;

[0008] Sending the second IMU data and the fourth IMU data to the first application;

[0009] Among them, the second IMU data is the IMU data transmitted through the first channel of the IMU and will be sent to the first application; the first IMU data is the historical IMU data transmitted through the first channel; the third IMU data is the IMU data for realizing optical image stabilization transmitted through the second channel of the IMU; the collection time of the first IMU data is the first moment, the collection time of the second IMU data is the second moment, and the collection time of the third IMU data is the third moment, the second moment is later than the first moment, and the third moment is later than the second moment; the fourth moment is later than the third moment.

[0010] In a second aspect, an embodiment of the present application provides an inertial measurement unit data processing device, including:

[0011] An acquisition module is used to acquire first IMU data, second IMU data and third IMU data of an inertial measurement unit IMU;

[0012] A processing module, configured to predict fourth IMU data corresponding to a fourth moment based on the first IMU data, the second IMU data, and the third IMU data;

[0013] A transmission module, used for sending the second IMU data and the fourth IMU data to the first application;

[0014] Among them, the second IMU data is the IMU data transmitted through the first channel of the IMU and will be sent to the first application; the first IMU data is the historical IMU data transmitted through the first channel; the third IMU data is the IMU data for realizing optical image stabilization transmitted through the second channel of the IMU; the collection time of the first IMU data is the first moment, the collection time of the second IMU data is the second moment, and the collection time of the third IMU data is the third moment, the second moment is later than the first moment, and the third moment is later than the second moment; the fourth moment is later than the third moment.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0019] In the embodiment of the present application, the first IMU data, the second IMU data and the third IMU data are obtained; based on the first IMU data, the second IMU data and the third IMU data, the fourth IMU data corresponding to the fourth moment in the future is predicted; the second IMU data and the fourth IMU data are sent to the first application. According to the embodiment of the present application, the predicted fourth IMU data is transmitted to the first application, so that the first application can receive and process the fourth IMU data corresponding to the fourth moment in advance, thereby offsetting the time required for the transmission and processing of the fourth IMU data, so that the first application can update the interactive operation results more timely. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of an inertial measurement unit data processing method provided in some embodiments of the present application;

[0021] Figure 2 is a flowchart of an inertial measurement unit data processing method provided in other embodiments of the present application;

[0022] Figure 3 is a schematic diagram of a prediction algorithm provided by some embodiments of the present application;

[0023] Figure 4 is a schematic diagram of an inertial measurement unit data processing device provided in some embodiments of the present application;

[0024] Figure 5 is a schematic block diagram of an electronic device provided in some embodiments of the present application;

[0025] Figure 6 It is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] In the related art, when using IMU to assist users in operating applications, there is a problem that the interactive operation results are not updated in time, resulting in a poor user experience. For example, in a shooting game application or an adventure game application, the user has turned the phone or the handle, but due to the delay in IMU data transmission and processing, the perspective of the game protagonist in the application cannot be switched in time, thus affecting the user's control experience.

[0029] The applicant has found that in electronic devices, in addition to assisting user operations, IMU data can also be used to implement optical image stabilization of electronic devices. Since the requirements for assisting user operations and implementing optical image stabilization are different, but there is usually only one IMU device in an electronic device, in order to implement the above two functions respectively, two data links are usually set in the electronic device to transmit the IMU data of the IMU device. For ease of distinction, the two data links are respectively referred to as the first channel and the second channel. Among them, the first channel is specifically used to transmit IMU data to the user interface to assist user operations. The first channel is also commonly referred to as the UI (User Interface) channel, and the second channel is specifically used to transmit IMU data to the optical image stabilization system to implement optical image stabilization. The second channel is also commonly referred to as the OIS (Optical Image Stabilization) channel. In the related art, when using IMU data to assist user operations, the IMU data in the first channel is reported to the application, so that the application updates the interactive operation results based on the IMU data transmitted by the first channel.

[0030] In view of this, in order to solve the problem that when using IMU data to assist user operations, the interactive operation results cannot be updated in time due to the delay in IMU data transmission and processing, the embodiment of the present application provides an inertial measurement unit data processing method, device, equipment, medium and product, using the IMU data in the first channel and the IMU data in the second channel to predict the IMU data at a specified time in the future, and then report the IMU data in the first channel and the predicted IMU data combination to the application, so that the application updates the interactive operation results based on the IMU data in the first channel and the predicted IMU data. In this way, the IMU data at the future time is transmitted to the application in advance, so that the application can process the IMU data in advance, thereby offsetting the delay caused by the IMU data transmission and processing, so that the interactive operation results are updated more timely, thereby improving the user control experience.

[0031] In combination with the accompanying drawings, the inertial measurement unit data processing method, device, equipment, medium and product provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0032] The inertial measurement unit data processing method provided in the embodiment of the present application can be applied to application scenarios such as games and entertainment. Figure 1-Figure 4 The inertial measurement unit data processing method provided in the embodiment of the present application is described in detail. It should be noted that the inertial measurement unit data processing method provided in the embodiment of the present application can be executed by an electronic device, and the electronic device includes but is not limited to a smart phone, a smart watch, a smart bracelet, and a tablet computer. In the embodiment of the present application, the inertial measurement unit data processing method provided in the embodiment of the present application is described by taking the execution of the inertial measurement unit data processing method by an electronic device as an example.

[0033] See also Figure 1 , is a flow chart of an inertial measurement unit data processing method provided in some embodiments of the present application, such as Figure 1 As shown, the method includes the following steps 110 to 130, which are described in detail below.

[0034] Step 110: Obtain first IMU data, second IMU data, and third IMU data of an inertial measurement unit IMU.

[0035] Here, the inertial measurement unit IMU refers to an IMU device installed in an electronic device for measuring IMU data. Exemplarily, the IMU device includes, but is not limited to, a gyroscope for measuring the rotational angular velocity of an electronic device and an accelerometer for measuring the acceleration of an electronic device. IMU data is data obtained by an IMU device, and the IMU data includes an IMU value and an acquisition time corresponding to the IMU value, wherein the IMU value refers to a digital quantity obtained by the IMU device. For example, in the case where the IMU device includes a gyroscope and an accelerometer, the IMU value includes a digital quantity of the rotational angular velocity of the electronic device obtained by the gyroscope and a digital quantity of the acceleration of the electronic device obtained by the accelerometer. Since the output of the IMU device is usually an analog quantity, the IMU value can be obtained by converting the analog quantity output by the IMU device through an analog-to-digital converter. For example, the digital quantity of the rotational angular velocity of the electronic device can be obtained by converting the rotational angular velocity of the electronic device output by the gyroscope after analog-to-digital conversion, and the digital quantity of the acceleration of the electronic device can be obtained by converting the acceleration of the electronic device output by the accelerometer after analog-to-digital conversion.

[0036] As mentioned above, the electronic device is provided with a first channel and a second channel for transmitting IMU data. In this embodiment, the second IMU data is the IMU data transmitted through the first channel of the IMU to be sent to the first application, the first IMU data is the historical IMU data transmitted through the first channel, the first IMU data is collected at the first moment, the second IMU data is collected at the second moment, and the second moment is later than the first moment. The third IMU data is the IMU data for optical image stabilization transmitted through the second channel of the IMU, and the third IMU data is collected at the third moment, and the third moment is later than the second moment.

[0037] In some embodiments of the present application, the electronic device may execute the above step 110 to obtain the first IMU data, the second IMU data, and the third IMU data when the first application needs to use the IMU data. The first application may be any application running in the foreground of the electronic device. Exemplarily, the first application includes but is not limited to game applications, fitness and sports applications, augmented reality AR and virtual reality VR applications, etc., and game applications include but are not limited to shooting game applications, racing game applications, parkour game applications, etc.

[0038] In some embodiments of the present application, the electronic device includes a SensorHub subsystem, and the electronic device can obtain the first IMU data, the second IMU data, and the third IMU data through the SensorHub subsystem. The SensorHub subsystem is a system module that integrates multiple sensors and performs data processing. It brings together multiple different types of sensors and can collect data from different types of sensors, such as accelerometers, gyroscopes, magnetometers, environmental sensors, etc., and fuse, analyze and process these data. The SensorHub subsystem also acts as a middleware or interface to interact with the operating system and other applications of the electronic device. It provides the processed sensor data to the operating system and applications in a standardized format, so that they can easily obtain and use sensor information without directly performing complex underlying communications and data processing with each sensor. At the same time, it can also receive instructions from the system and applications to configure and adjust the working mode, parameters, etc. of the sensor. Based on this, when the first application needs to use the IMU data, it can register the IMU data at a specified sampling rate with the SensorHub subsystem in the electronic device. The first application registers the IMU data at a specified sampling rate with the SensorHub subsystem, which means that the first application sends a request to the SensorHub subsystem, requesting the SensorHub subsystem to collect and provide IMU data at a specified frequency. For example, the first application registers IMU data with a sampling rate of 50Hz to the SensorHub subsystem, which means that the first application requests the SensorHub subsystem to collect and provide IMU data at a frequency of 50Hz. In this way, the SensorHub subsystem can obtain the first IMU data, the second IMU data, and the third IMU data after the first application completes the registration.

[0039] In some embodiments of the present application, the SensorHub subsystem can set the ODR (output data rate) of the first channel based on the above-mentioned specified sampling rate after the first application completes registration, so as to obtain the second IMU data to be sent to the first application from the first channel based on the ODR of the first channel. Wherein, when setting the ODR of the first channel based on the specified sampling rate, the ratio of the specified sampling rate to N can be used as the ODR of the first channel. Wherein N is a positive integer value, and the specific value of N can be pre-set according to actual conditions. For example, the larger the ODR of the first channel, the higher the frequency of the SensorHub subsystem transmitting IMU data to the first application, and the first application can receive the IMU data more timely, thereby updating the interactive operation results more timely. Therefore, the value of N can be 1, that is, the ODR of the first channel can be consistent with the specified sampling rate. For example, if the specified sampling rate is 50Hz, the ODR of the first channel is set to 50Hz. In some embodiments of the present application, the first channel adopts a FIFO (First In First Out) interrupt mode, that is, the IMU data transmitted in the first channel is first cached in the FIFO, and when the FIFO is full, an interrupt signal is generated, and the SensorHub subsystem reads the IMU data stored in the FIFO based on the interrupt signal to transmit it to the first application. Among them, the full state of the FIFO means that the amount of data stored in the FIFO has reached the maximum amount of data it can accommodate. The maximum amount of data that the FIFO can accommodate is the ratio of the specified sampling rate to the ODR of the first channel, that is, N. For example, if the specified sampling rate and the ODR of the first channel are both 50Hz, the maximum amount of data that the FIFO can accommodate is 1, and the FIFO generates an interrupt every time it stores an IMU data to allow the SensorHub subsystem to read the IMU data stored in the FIFO. Based on this, after the SensorHub subsystem sets the ODR of the first channel based on the specified sampling rate registered by the first application, it can detect in real time whether the FIFO of the first channel is full. When the interrupt signal of the FIFO is detected, it can determine that the FIFO is full, and then read the IMU data stored in the FIFO, and use the read IMU data as the second IMU data. If no interrupt signal is detected, it is determined that the FIFO is not full, and then continue to detect whether the FIFO is full.

[0040] In some embodiments of the present application, the first IMU data may be the IMU data most recently read from the FIFO of the first path before the second IMU data is read. To facilitate the acquisition of the first IMU data, the electronic device may store the IMU data read from the FIFO each time to a specified location, so that in the above step 110, the IMU data most recently sent to the first application may be directly acquired from the specified location as the first IMU data.

[0041] In some embodiments of the present application, when preparing to report the second IMU data to the first application, the latest IMU data transmitted by the second channel can be obtained, and the obtained IMU data can be used as the third IMU data, thereby obtaining the third IMU data whose collection time is later than the second time.

[0042] In some embodiments of the present application, a data register may be used in the second path to store IMU data. Based on this, when preparing to send the second IMU data to the first application, the latest IMU data stored in the data register may be read, and the read IMU data may be used as the third IMU data.

[0043] Step 120. Based on the first IMU data, the second IMU data and the third IMU data, predict the fourth IMU data corresponding to the fourth moment.

[0044] Here, the fourth moment is a future moment that is later than the third moment.

[0045] In some embodiments of the present application, after acquiring the first IMU data, the second IMU data, and the third IMU data, the SensorHub subsystem predicts the fourth IMU data at the fourth moment in the future based on the first IMU data, the second IMU data, and the third IMU data. When predicting the fourth IMU data, the SensorHub subsystem can analyze the first IMU data, the second IMU data, and the third IMU data to determine the law of the IMU data based on time changes, and then predict the fourth IMU data corresponding to the fourth moment based on the law.

[0046] In this embodiment, compared with predicting the fourth IMU data corresponding to the fourth moment based on the second IMU data and other IMU data collected earlier than the second IMU data, predicting the fourth IMU data based on the third IMU data collected closer to the fourth moment will result in a more accurate prediction result.

[0047] Step 130. Send the second IMU data and the fourth IMU data to the first application.

[0048] In some embodiments of the present application, after predicting the fourth IMU data corresponding to the fourth moment, the SensorHub subsystem reports the fourth IMU data and the second IMU data to the first application. In this way, the fourth inertial unit data of the fourth moment in the future can be transmitted to the first application in advance, thereby offsetting the time consumed in transmitting the fourth inertial unit data, the first application receives the fourth inertial unit data in advance, and the first application can process the fourth inertial unit data in advance and obtain the processing result in advance, thereby offsetting the time consumed in processing the fourth inertial unit data. In this way, when the first application needs to use the fourth inertial unit data, it can directly obtain the processing result corresponding to the fourth inertial unit data without waiting for the transmission and processing of the fourth inertial unit data, so that the first application can update the interactive operation results in time.

[0049] The IMU data processing method provided in the embodiment of the present application obtains the first IMU data, the second IMU data and the third IMU data; based on the first IMU data, the second IMU data and the third IMU data, predicts the fourth IMU data corresponding to the fourth moment in the future; and sends the second IMU data and the fourth IMU data to the first application. According to the embodiment of the present application, the predicted fourth IMU data is transmitted to the first application, so that the first application can receive and process the fourth IMU data corresponding to the fourth moment in advance, thereby offsetting the time required for the transmission and processing of the fourth IMU data, so that the first application can update the interactive operation results more timely.

[0050] In some embodiments, in order to enhance the user's freedom, a control for turning the data prediction mode on and off may be provided in the electronic device, and the user may operate the control according to actual needs to turn the data prediction mode on or off. When the data prediction mode is on, when the first application needs to use the IMU data, the IMU value at a certain moment in the future will be predicted. When the data prediction mode is not on, when the first application needs to use the IMU data, the IMU value at a certain moment in the future will not be predicted. Based on this, before the above step 110, the electronic device may first detect whether the data prediction mode is on, and then execute steps 110 and 130 if it is determined that the data prediction mode is on. In the case where the data prediction mode is not on, steps 110-130 do not need to be executed.

[0051] In some embodiments of the present application, when the data prediction mode is not enabled, the traditional IMU data processing method can be used to obtain only the second IMU data and directly transmit the second IMU data to the first application. For example, when the data prediction mode is not enabled, the SensorHub subsystem can read the stored IMU data in the FIFO when the FIFO is full, and then directly transmit the read IMU data to the first application without performing IMU data prediction.

[0052] In some embodiments, Figure 2 As shown, in the above step 120, the fourth IMU data corresponding to the fourth moment can be predicted through the following steps 210-230.

[0053] Step 210. Based on the first IMU data, the second IMU data and the third IMU data, determine a fitting function with time as the independent variable and the IMU value as the dependent variable.

[0054] In some embodiments of the present application, the first IMU data, the second IMU data, and the third IMU data can be regarded as coordinate points in a custom coordinate system, wherein the custom coordinate system uses time as the horizontal axis and the IMU value as the vertical axis. By performing curve fitting on the coordinate points corresponding to the first IMU data, the second IMU data, and the third IMU data, a fitting function corresponding to the first IMU data, the second IMU data, and the third IMU data can be obtained, wherein the fitting function uses time as the independent variable and the IMU value as the dependent variable.

[0055] For example, see Figure 3 , where (t1,d1), (t2,d2), (t3,d3)…(t n ,d n ) represents the IMU data transmitted by the first channel, which means that the IMU value collected at time t1 is d1, the IMU value collected at time t2 is d2, the IMU value collected at time t3 is d3, and the IMU value collected at time t4 is d5. n The IMU value collected at each moment is d n , where the time interval between two adjacent acquisition moments is ODR represents the ODR of the first pass; (ot2,od2)…(ot n ,od n ) indicates the IMU data transmitted by the second channel, which means that the IMU value collected at time ot2 is od2, ot n The IMU value collected at all times is od n ;(pt2,pd2)…(pt n ,pd n ) represents the predicted IMU data, which means that the IMU predicted value corresponding to the pt2 moment is pd2, ptn The IMU prediction value corresponding to the moment is pd n Taking (t2, d2) as the second IMU data, (t1, d1) as the first IMU data, and (ot2, od2) as the third IMU data as an example, substituting (t1, d1), (t2, d2) and (ot2, od2) into the following formula (1), the corresponding fitting function f(t) can be obtained.

[0056]

[0057] Among them, the f(t) function takes time t as the independent variable and the IMU value f(t) as the dependent variable.

[0058] Step 220. According to the fourth moment and the fitting function, obtain the fourth IMU data corresponding to the fourth moment.

[0059] In some embodiments of the present application, after obtaining a fitting function with time as the independent variable and IMU value as the dependent variable based on the first IMU data, the second IMU data and the third IMU data, the value of the fourth moment as the independent variable is substituted into the fitting function to obtain the function value corresponding to the fourth moment, which is the predicted IMU value corresponding to the fourth moment. For example, taking the fourth moment as the pt2 moment as an example, pt2 is substituted into formula (1) as the value of t to obtain f(pt2), and f(pt2) is taken as the IMU predicted value corresponding to the pt2 moment, which is pd2. After predicting the IMU value corresponding to the fourth moment, the fourth moment and the predicted IMU value are combined into an array, and the array is used as the fourth IMU data corresponding to the fourth moment.

[0060] Since the fitting function is obtained by curve fitting based on the first IMU data, the second IMU data and the third IMU data, the IMU value corresponding to the fourth moment predicted based on the fitting function conforms to the IMU value change law determined based on the first IMU data, the second IMU data and the third IMU data. Therefore, the IMU value corresponding to the fourth moment predicted based on the fitting function has higher accuracy.

[0061] Through the above technical solution, the IMU value corresponding to the fourth moment is predicted based on the fitting function corresponding to the first IMU data, the second IMU data and the third IMU data, so that the predicted IMU value conforms to the change law of the IMU values ​​corresponding to the first IMU data, the second IMU data and the third IMU data, and has high accuracy.

[0062] In some embodiments, the fourth moment is determined before the above step 120. The fourth moment can be set to a moment later than the second moment and earlier than the next acquisition moment of the second IMU data, so that the predicted fourth IMU data can be sent to the first application between the two transmissions of the second IMU data to the first application.

[0063] From the above content, it can be seen that the SensorHub subsystem reads the second IMU data to be transmitted to the first application from the FIFO based on the ODR of the first channel. Therefore, the SensorHub subsystem reads the second IMU data to be transmitted to the first application every Read the second IMU data from the FIFO once. For example, taking the ODR as 50Hz, the SensorHub subsystem reads the second IMU data from the FIFO every 20ms. Based on this, the fourth moment can be determined by the following steps:

[0064] Obtain a first time difference between a first moment and a second moment;

[0065] Multiplying the first time difference by a preset coefficient to obtain a second time difference, where the preset coefficient is greater than 0 and less than 1;

[0066] The sum of the second time difference and the second moment is taken as the fourth moment.

[0067] Based on the above method, the fourth moment can be calculated according to the following formula (2):

[0068] pt2=a*(t2-t1)+t2 (2)

[0069] Wherein, pt2 represents the fourth moment, a represents the preset coefficient, the value range of a is (0,1), t2 represents the second moment, and t1 represents the first moment.

[0070] In some embodiments of the present application, the value of a can be 0.5, so that the interval between the fourth moment and the second moment can be equal to the interval between the fourth moment and the next acquisition moment of the second IMU data, thereby making the time intervals between adjacent IMU data received in the first application equal.

[0071] Through the above technical solution, the fourth moment can be set to a moment between the collection moment of obtaining the second IMU data this time and the collection moment of obtaining the second IMU data next time. In this way, the predicted fourth moment can be sent to the first application within the interval between two adjacent times of sending the second IMU data to the first application, thereby doubling the frequency of outputting IMU data to the first application without increasing power consumption. For example, taking the specified sampling rate registered by the first application as 50Hz and the ODR of the first channel as 50Hz, according to the traditional IMU data processing solution, the SensorHub subsystem reads a second IMU data from the FIFO every 20ms and transmits it to the first application. It can be seen that the frequency of outputting IMU data to the first application is 50Hz. According to this embodiment, in addition to reading a second IMU data from the FIFO every 20ms and transmitting it to the first application, the SensorHub subsystem also predicts a fourth IMU data with an acquisition time before the acquisition time of the second IMU data read this time and the acquisition time of the second IMU data read next time after each reading of the second IMU data from the FIFO, and transmits the fourth IMU data to the first application. In this way, within the same unit time, the amount of IMU data output to the first application based on this embodiment is doubled compared with the traditional solution, that is, the frequency of outputting IMU data to the first application is doubled.

[0072] In some embodiments, before acquiring the third IMU data, the output data rate of the second channel can be set to a preset maximum output data rate. The maximum output data rate is determined by the performance of the IMU device, and different IMU devices have different maximum output data rates. For example, if the maximum output data rate supported by the IMU device in the electronic device is 6600Hz, the output data rate ODR of the second channel is set to 6600Hz. If the maximum output data rate supported by the IMU device in the electronic device is 5500Hz, the output data rate ODR of the second channel is set to 5500Hz.

[0073] The maximum output data rate supported by the IMU device is usually greater than the ODR of the first path, that is, the frequency of the second path outputting IMU data is greater than the frequency of the first path outputting IMU data. Based on this, by setting the ODR of the second path to the maximum output data rate supported by the IMU device, it can be ensured that after the second IMU data is acquired, when preparing to transmit the second IMU data to the first application, the third IMU data with a collection time later than the second time can be successfully acquired.

[0074] In some embodiments, considering that the first application can only parse one IMU data at a time, if the second IMU data and the fourth IMU data are sent to the first application at the same time, the first application will not be able to successfully parse the second IMU data and the fourth IMU data. Therefore, in the above step 130, the second IMU data and the fourth IMU data can be sent to the first application in sequence, that is, the second IMU data is first sent to the first application, and then the fourth IMU data is sent to the first application. In this way, the first application can accurately parse the second IMU data and the fourth IMU data.

[0075] In some embodiments of the present application, the second IMU data and the fourth IMU data can be sent to the first application in sequence through the system framework layer of the electronic device. The system framework layer is an intermediate layer in the electronic device that provides infrastructure and services for the application. It is between the operating system and the application. Taking an electronic device using the Android operating system as an example, the system framework layer refers to the sensor framework layer, that is, the SensorFramework layer.

[0076] In some embodiments of the present application, sending the second IMU data and the fourth IMU data to the first application in sequence through the system framework layer of the electronic device includes the following steps:

[0077] Combining the second IMU data and the fourth IMU data to obtain combined data;

[0078] Send combined data to the system framework layer;

[0079] The combined data is parsed by the system framework layer to obtain the second IMU data and the fourth IMU data;

[0080] The second IMU data and the fourth IMU data are sent to the first application in sequence through the system framework layer.

[0081] After predicting the fourth IMU data, the SensorHub subsystem combines the second IMU data and the fourth IMU data in a preset format to obtain combined data. The preset format may be a format that can be parsed by the system framework layer, and the specific format may be pre-set according to actual conditions.

[0082] For example, the preset format may be as follows:

[0083] A1 B1 C1 A2 B2 C2

[0084] Among them, the A1 position corresponds to the timestamp in the second IMU data, that is, the second moment, the B1 position corresponds to the digital quantity corresponding to the rotation angular velocity in the second IMU data, the C1 position corresponds to the digital quantity corresponding to the acceleration in the second IMU data, the A2 position corresponds to the timestamp in the fourth IMU data, that is, the fourth moment, the B1 position corresponds to the digital quantity corresponding to the rotation angular velocity in the fourth IMU data, and the C1 position corresponds to the digital quantity corresponding to the acceleration in the fourth IMU data.

[0085] After obtaining the combined data, the SensorHub subsystem sends the combined data to the system framework layer. After receiving the combined data, the system framework layer parses the combined data to obtain the second IMU data and the fourth IMU data. As mentioned above, the SensorHub subsystem combines the second IMU data and the fourth IMU data based on a preset format that can be parsed by the system framework layer to obtain the combined data. Therefore, after receiving the combined data, the system framework layer can parse the combined data based on the preset format to obtain the second IMU data and the fourth IMU data.

[0086] After parsing the second IMU data and the fourth IMU data, the system framework layer first sends the second IMU data to the first application, and then sends the fourth IMU data to the first application. The smaller the time interval between the two transmissions, the better. In this way, the first application can receive the fourth inertial measurement data earlier, and thus has more sufficient time to process the fourth IMU data, so that the interactive operation results can be updated more timely based on the fourth IMU data.

[0087] Based on the IMU data processing method provided in the above embodiment, the present application also provides a specific implementation of the IMU data processing device. Please refer to the following embodiment.

[0088] See also Figure 4 , a schematic diagram of an IMU data processing device provided in some embodiments of the present application, such as Figure 4 As shown, the device 400 includes the following modules:

[0089] An acquisition module 410 is used to acquire first IMU data, second IMU data and third IMU data of an inertial measurement unit IMU;

[0090] A processing module 420, configured to predict fourth IMU data corresponding to a fourth moment based on the first IMU data, the second IMU data, and the third IMU data;

[0091] The transmission module 430 is used to send the second IMU data and the fourth IMU data to the first application.

[0092] The second IMU data is the IMU data transmitted through the first path of the IMU and to be sent to the first application;

[0093] The second IMU data is the historical IMU data transmitted through the first path;

[0094] The third IMU data is IMU data for implementing optical image stabilization and transmitted through the second path of the IMU;

[0095] The collection time of the first IMU data is the first moment, the collection time of the second IMU data is the second moment, and the collection time of the third IMU data is the third moment. The second moment is later than the first moment, and the third moment is later than the second moment.

[0096] The fourth moment is later than the third moment.

[0097] The IMU data processing device provided in the embodiment of the present application obtains the first IMU data, the second IMU data and the third IMU data; based on the first IMU data, the second IMU data and the third IMU data, predicts the fourth IMU data corresponding to the fourth moment in the future; and sends the second IMU data and the fourth IMU data to the first application. According to the embodiment of the present application, the predicted fourth IMU data is transmitted to the first application, so that the first application can receive and process the fourth IMU data corresponding to the fourth moment in advance, thereby offsetting the time required for the transmission and processing of the fourth IMU data, so that the first application can update the interactive operation results more timely.

[0098] In some embodiments, the processing module 420 is specifically configured to:

[0099] Based on the first IMU data, the second IMU data and the third IMU data, determine a fitting function with time as an independent variable and the IMU value as a dependent variable;

[0100] According to the fourth moment and the fitting function, obtaining fourth IMU data corresponding to the fourth moment;

[0101] The fourth IMU data includes the fourth moment and the IMU value corresponding to the fourth moment.

[0102] In some embodiments, the processing module 420 is further configured to:

[0103] Before predicting fourth IMU data corresponding to a fourth moment based on the first IMU data, the second IMU data, and the third IMU data, obtaining a first time difference between the first moment and the second moment;

[0104] Multiplying the first time difference by a preset coefficient to obtain a second time difference, where the preset coefficient is greater than 0 and less than 1;

[0105] The sum of the second time difference and the second moment is taken as the fourth moment.

[0106] In some embodiments, the processing module 420 is further configured to:

[0107] Before acquiring the third IMU data, the output data rate of the second channel is set to a preset maximum output data rate.

[0108] In some embodiments, the transmission module 430 is specifically configured to:

[0109] The second IMU data and the fourth IMU data are sent to the first application in sequence.

[0110] The IMU data processing device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0111] The IMU data processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0112] The IMU data processing device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0113] Alternatively, if Figure 5As shown, an embodiment of the present application further provides an electronic device 500, including a processor 501 and a memory 502, wherein the memory 502 stores programs or instructions that can be executed on the processor 501, and when the program or instructions are executed by the processor 501, the various steps of the above-mentioned IMU data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0114] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0115] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0116] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610.

[0117] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0118] The processor 610 is used to obtain first IMU data, second IMU data and third IMU data of an inertial measurement unit IMU;

[0119] The processor 610 is configured to predict fourth IMU data corresponding to a fourth moment based on the first IMU data, the second IMU data, and the third IMU data;

[0120] Processor 610, configured to send the second IMU data and the fourth IMU data to the first application;

[0121] The second IMU data is the IMU data transmitted through the first path of the IMU and to be sent to the first application;

[0122] The first IMU data is the historical IMU data transmitted through the first path;

[0123] The third IMU data is IMU data for implementing optical image stabilization and transmitted through the second path of the IMU;

[0124] The collection time of the first IMU data is the first moment, the collection time of the second IMU data is the second moment, and the collection time of the third IMU data is the third moment. The second moment is later than the first moment, and the third moment is later than the second moment.

[0125] The fourth moment is later than the third moment.

[0126] The electronic device provided in the embodiment of the present application obtains the first IMU data, the second IMU data and the third IMU data; based on the first IMU data, the second IMU data and the third IMU data, predicts the fourth IMU data corresponding to the fourth moment in the future; and sends the second IMU data and the fourth IMU data to the first application. According to the embodiment of the present application, the predicted fourth IMU data is transmitted to the first application, so that the first application can receive and process the fourth IMU data corresponding to the fourth moment in advance, thereby offsetting the time required for the transmission and processing of the fourth IMU data, so that the first application can update the interactive operation results more timely.

[0127] In some embodiments, the processor 610 is specifically configured to:

[0128] Based on the first IMU data, the second IMU data and the third IMU data, determine a fitting function with time as an independent variable and the IMU value as a dependent variable;

[0129] According to the fourth moment and the fitting function, obtaining fourth IMU data corresponding to the fourth moment;

[0130] The fourth IMU data includes the fourth moment and the IMU value corresponding to the fourth moment.

[0131] In some embodiments, the processor 610 is further configured to:

[0132] Before predicting fourth IMU data corresponding to a fourth moment based on the first IMU data, the second IMU data, and the third IMU data, obtaining a first time difference between the first moment and the second moment;

[0133] Multiplying the first time difference by a preset coefficient to obtain a second time difference, where the preset coefficient is greater than 0 and less than 1;

[0134] The sum of the second time difference and the second moment is taken as the fourth moment.

[0135] In some embodiments, the processor 610 is further configured to:

[0136] Before acquiring the third IMU data, the output data rate of the second channel is set to a preset maximum output data rate.

[0137] In some embodiments, the processor 610 is specifically configured to:

[0138] The second IMU data and the fourth IMU data are sent to the first application in sequence.

[0139] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0140] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory x09 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0141] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 610.

[0142] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned IMU data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0143] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0144] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned IMU data processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0145] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0146] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned IMU data processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0147] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for processing inertial measurement unit data, characterized in that: include: Acquire first IMU data, second IMU data, and third IMU data of an inertial measurement unit IMU; Predicting fourth IMU data corresponding to a fourth moment based on the first IMU data, the second IMU data, and the third IMU data; Sending the second IMU data and the fourth IMU data to the first application; Among them, the second IMU data is the IMU data transmitted through the first channel of the IMU and will be sent to the first application; the first IMU data is the historical IMU data transmitted through the first channel; the third IMU data is the IMU data for realizing optical image stabilization transmitted through the second channel of the IMU; the collection time of the first IMU data is the first moment, the collection time of the second IMU data is the second moment, and the collection time of the third IMU data is the third moment, the second moment is later than the first moment, and the third moment is later than the second moment; the fourth moment is later than the third moment.

2. The method according to claim 1, characterized in that The predicting, based on the first IMU data, the second IMU data, and the third IMU data, fourth IMU data corresponding to a fourth moment includes: Based on the first IMU data, the second IMU data and the third IMU data, determine a fitting function with time as an independent variable and IMU value as a dependent variable; Acquire fourth IMU data corresponding to the fourth moment according to the fourth moment and the fitting function; The fourth IMU data includes the fourth moment and the IMU value corresponding to the fourth moment.

3. The method according to claim 1 or 2, characterized in that: Before predicting fourth IMU data corresponding to a fourth moment based on the first IMU data, the second IMU data, and the third IMU data, the method further includes: Obtaining a first time difference between the first moment and the second moment; Multiplying the first time difference by a preset coefficient to obtain a second time difference, wherein the preset coefficient is greater than 0 and less than 1; The sum of the second time difference and the second moment is taken as the fourth moment.

4. The method according to claim 1 or 2, characterized in that: Before acquiring the third IMU data, the method further includes: The output data rate of the second path is set to a preset maximum output data rate.

5. The method according to claim 1 or 2, characterized in that: The sending the second IMU data and the fourth IMU data to the first application includes: The second IMU data and the fourth IMU data are sent to the first application in sequence.

6. An inertial measurement unit data processing device, characterized in that: include: An acquisition module is used to acquire first IMU data, second IMU data and third IMU data of an inertial measurement unit IMU; A processing module, configured to predict fourth IMU data corresponding to a fourth moment based on the first IMU data, the second IMU data, and the third IMU data; A transmission module, configured to send the second IMU data and the fourth IMU data to the first application; Among them, the second IMU data is the IMU data transmitted through the first channel of the IMU and will be sent to the first application; the first IMU data is the historical IMU data transmitted through the first channel; the third IMU data is the IMU data for realizing optical image stabilization transmitted through the second channel of the IMU; the collection time of the first IMU data is the first moment, the collection time of the second IMU data is the second moment, and the collection time of the third IMU data is the third moment, the second moment is later than the first moment, and the third moment is later than the second moment; the fourth moment is later than the third moment.

7. The device according to claim 6, characterized in that The processing module is specifically used for: Based on the first IMU data, the second IMU data and the third IMU data, determine a fitting function with time as an independent variable and IMU value as a dependent variable; Acquire fourth IMU data corresponding to the fourth moment according to the fourth moment and the fitting function; The fourth IMU data includes the fourth moment and the IMU value corresponding to the fourth moment.

8. The device according to claim 6 or 7, characterized in that The processing module is further used for: Before predicting the fourth IMU data corresponding to the fourth moment based on the first IMU data, the second IMU data and the third IMU data, obtaining a first time difference between the first moment and the second moment; Multiplying the first time difference by a preset coefficient to obtain a second time difference, where the preset coefficient is greater than 0 and less than 1; The sum of the second time difference and the second moment is taken as the fourth moment.

9. The device according to claim 6 or 7, characterized in that The processing module is further used for: Before acquiring the third IMU data transmitted through the second channel, the output data rate of the second channel is set to a preset maximum output data rate.

10. The device according to claim 6 or 7, characterized in that The transmission module is specifically used for: The second IMU data and the fourth IMU data are sent to the first application in sequence through the system framework layer.

11. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the inertial measurement unit data processing method as described in any one of claims 1 to 5 are implemented.

Images