Data processing method and device and electronic equipment

Abstract

The embodiment of the invention discloses a data processing method and device and electronic equipment. The method comprises the following steps: acquiring motion learning features of wearable equipment in a vehicle relative to the vehicle; obtaining visual learning features of the wearable device relative to the vehicle; and fusing the motion learning features and the visual learning features to obtain position attitude features of the wearable device relative to the vehicle, and taking the position attitude features as degree-of-freedom positioning estimation parameters of the wearable device in the vehicle. According to the method, various deep learning features are comprehensively utilized, so that the pose estimation result of the wearable device in the vehicle can be more accurate, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and more specifically, to a data processing method, device and electronic device. Background Art

[0002] State estimation algorithm is a mathematical method used to estimate the state of a system. It can predict the future state of the system by analyzing the input and output of the system. State estimation algorithms are widely used in many fields, such as robot control, aircraft navigation, self-driving cars, etc.

[0003] SLAM (Simultaneous localization and mapping) is a state estimation algorithm that can be used to estimate the trajectory (or posture) of a robot equipped with the SLAM algorithm and the position of map features. However, for self-driving cars, the external environment of the vehicle will be in a changing state due to the movement of the vehicle, which makes the state estimation algorithm of the MR (Mixed Reality) device inside the vehicle unable to run, affecting the user experience. Summary of the invention

[0004] In view of the above problems, the present application proposes a data processing method, device and electronic device to improve the above problems.

[0005] In a first aspect, the present application provides a data processing method, comprising: obtaining motion learning features of a wearable device inside a vehicle relative to the vehicle; obtaining visual learning features of the wearable device relative to the vehicle; fusing the motion learning features with the visual learning features to obtain position and posture features of the wearable device relative to the vehicle as degree-of-freedom positioning estimation parameters of the wearable device inside the vehicle.

[0006] In a second aspect, the present application provides a data processing device, comprising: a motion feature acquisition module, used to acquire motion learning features of a wearable device inside a vehicle relative to the vehicle; a visual feature acquisition module, used to acquire visual learning features of the wearable device relative to the vehicle; and a degree of freedom positioning estimation parameter acquisition module, used to fuse the motion learning features with the visual learning features to obtain position and posture features of the wearable device relative to the vehicle as degree of freedom positioning estimation parameters of the wearable device inside the vehicle.

[0007] In a third aspect, the present application provides an electronic device comprising one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium, in which program code is stored, wherein the above method is executed when the program code is run.

[0009] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0010] The present application provides a data processing method, device, vehicle and storage medium, which obtains the motion learning features of the wearable device inside the vehicle relative to the vehicle; obtains the visual learning features of the wearable device relative to the vehicle; and fuses the motion learning features with the visual learning features to obtain the position and posture features of the wearable device relative to the vehicle as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle. Thus, by comprehensively utilizing multiple deep learning features, the position and posture estimation results of the wearable device inside the vehicle can be more accurate, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A flow chart of a data processing method proposed in an embodiment of the present application is shown.

[0013] Figure 2 A flow chart of a data processing method proposed in another embodiment of the present application is shown.

[0014] Figure 3 A flow chart of a data processing method proposed in yet another embodiment of the present application is shown.

[0015] Figure 4 An example flow chart of a data processing method provided in an embodiment of the present application is shown.

[0016] Figure 5 A structural block diagram of a data processing device proposed in an embodiment of the present application is shown.

[0017] Figure 6 A structural block diagram of an electronic device proposed in this application is shown. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] In the embodiment of the present application, the inventor proposes a data processing method, device and electronic device, which obtains the motion learning features of the wearable device inside the vehicle relative to the vehicle; obtains the visual learning features of the wearable device relative to the vehicle; and fuses the motion learning features with the visual learning features to obtain the position and posture features of the wearable device relative to the vehicle as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle. Thus, by comprehensively utilizing multiple deep learning features, the position and posture estimation results of the wearable device inside the vehicle can be more accurate, thereby improving the user experience.

[0020] See also Figure 1 , is a data processing method provided in an embodiment of the present application, the data processing method can be applied to estimate the positioning information of a wearable device, and can be specifically used to perform high-frequency and high-precision positioning of a wearable device (such as MR glasses) inside a vehicle, the method comprising:

[0021] Step S110: Acquire motion learning characteristics of the wearable device inside the vehicle relative to the vehicle.

[0022] The wearable device may be a MR device, and may specifically include MR glasses and / or head mounted displays, etc., and the specific type may not be limited. The embodiment of the present application is described by taking the wearable device as MR glasses as an example.

[0023] When the vehicle is in motion, the wearable device will also shake due to the movement of the vehicle, so that its (wearable device) position is in a constantly changing state. In this way, if you want to obtain the positioning estimation information of the wearable device relative to the vehicle, you can first obtain the motion learning features of the wearable device relative to the vehicle. Optionally, the motion learning feature can be understood as the expression of the motion feature of the wearable device relative to the vehicle, and the motion learning feature can be obtained by learning the motion information of the wearable device and the vehicle through a deep learning model. It is worth noting that the deep learning model here is a deep learning model based on motion information. For example, the deep learning model can be a Deep Kalman Filter, and the specific type can be not limited.

[0024] Optionally, the road section on which the moving vehicle travels may be a straight road, a curved road, or a combination of the two.

[0025] Step S120: Acquire visual learning features of the wearable device relative to the vehicle.

[0026] In order to more accurately obtain the positioning estimation information of the wearable device relative to the vehicle, the visual learning features of the wearable device relative to the vehicle can also be obtained, and then the positioning estimation information of the wearable device relative to the vehicle can be obtained based on the motion learning features of the wearable device relative to the vehicle and the visual learning features of the wearable device relative to the vehicle.

[0027] The visual learning feature can be understood as the visual feature expression of the wearable device relative to the vehicle, and the visual learning feature can be learned by a deep learning model based on the image information of the wearable device and the sensor information of the wearable device. It should be noted that the deep learning model can be a convolutional neural network (CNN), a recurrent neural network (RNN), or a Transformer (deep learning model architecture).

[0028] Optionally, the visual learning features of the wearable device relative to the vehicle may be obtained while the motion learning features of the wearable device relative to the vehicle are obtained; or the visual learning features of the wearable device relative to the vehicle may be obtained after the motion learning features of the wearable device relative to the vehicle are obtained. The specific execution order of the two may not be limited.

[0029] Step S130: The motion learning feature is integrated with the visual learning feature to obtain a position and posture feature of the wearable device relative to the vehicle as a degree of freedom positioning estimation parameter of the wearable device inside the vehicle.

[0030] As an implementation method, by fusing the motion learning features of the wearable device relative to the vehicle with the visual learning features of the wearable device relative to the vehicle, the position and posture features of the wearable device relative to the vehicle can be obtained as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle, that is, the position and posture features can be regarded as the positioning estimation information of the wearable device relative to the vehicle.

[0031] Among them, the commonalities and features between motion learning features and visual learning features can be learned through a deep learning model, and then the position and posture features of the wearable device relative to the vehicle can be analyzed. Optionally, the deep learning model is a degree of freedom state estimation model. For example, the deep learning model in this embodiment can be a 6-degree-of-freedom state estimation model.

[0032] This embodiment provides a data processing method, which obtains the motion learning features of the wearable device inside the vehicle relative to the vehicle; obtains the visual learning features of the wearable device relative to the vehicle; and fuses the motion learning features with the visual learning features to obtain the position and posture features of the wearable device relative to the vehicle as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle. Thus, by comprehensively utilizing multiple deep learning features, the position and posture estimation results of the wearable device inside the vehicle can be more accurate, thereby improving the user experience.

[0033] See also Figure 2 , is a data processing method provided in another embodiment of the present application, the method comprising:

[0034] Step S210: Obtain the motion parameters of the vehicle relative to the ground.

[0035] In the process of acquiring the motion learning characteristics of the wearable device inside the vehicle relative to the vehicle, as an implementation method, the motion parameters of the vehicle relative to the ground can be acquired first.

[0036] In this embodiment, the motion parameters of the vehicle relative to the ground may include at least one of the vehicle's motion speed, acceleration, angular velocity, position, and rotation information.

[0037] As an implementation method, in order to reduce the dependence of wearable devices on vehicle computing resources, the wearable device can obtain the vehicle's motion parameters relative to the ground through a network (which can be a local area network inside the vehicle or a mobile network). Optionally, the vehicle can be equipped with sensors for collecting the above-mentioned multiple motion parameters. In this way, the vehicle's sensors can collect the relevant motion parameters and transmit them to the wearable device through the network (actively or passively).

[0038] Step S220: Obtain motion parameters of the wearable device inside the vehicle relative to the vehicle.

[0039] The motion parameters of the wearable device inside the vehicle relative to the vehicle can be understood as at least one type of motion parameters of the wearable device. It is understandable that during the movement of the vehicle, the images taken by the wearable device inside the vehicle, the angle and position of the wearable device, etc. will change. In this way, in the process of obtaining the motion parameters of the wearable device inside the vehicle relative to the vehicle, the first motion parameters obtained by the IMU (Inertial Measurement Unit) of the wearable device and the second motion parameters obtained by observing the camera of the wearable device can be obtained; then the first motion parameters are fused with the second parameters to obtain the motion parameters of the wearable device relative to the vehicle.

[0040] The first motion parameter and the second motion parameter are observations of different sources of the wearable device, and the confidence levels of the two are different. By fusing the first motion parameter and the second motion parameter, the error can be reduced.

[0041] Step S230: input the motion parameters of the vehicle relative to the ground and the motion parameters of the wearable device relative to the vehicle into a first deep learning model, and output a motion learning feature of the wearable device relative to the vehicle, wherein the first deep learning model is a deep learning model based on motion information.

[0042] It can be understood that the motion parameters of the vehicle relative to the ground are relative to the world coordinate system, while the motion parameters of the wearable device relative to the vehicle are relative to the vehicle as a reference system. In order to unify the coordinate system to reduce errors, the motion parameters of the vehicle relative to the ground and the motion parameters of the wearable device relative to the vehicle can be input into the first deep learning model, so that the two types of data with different coordinate systems can be converted into data in the same coordinate system through the first deep learning model, and then the motion learning features of the wearable device relative to the vehicle are output. Optionally, the first deep learning model is a deep learning model based on motion information.

[0043] In this embodiment, since the motion parameters of the vehicle relative to the ground and the motion parameters of the wearable device relative to the vehicle are in different coordinate systems, there is a time delay. In order to eliminate the delay error as much as possible, the process of obtaining the motion parameters of the vehicle relative to the ground and the process of obtaining the motion parameters of the wearable device relative to the vehicle need to comply with the same time synchronization protocol, that is, the timestamps of the parameters obtained by the two processes need to have a synchronized starting point. Optionally, the time synchronization protocol can be a Network Time Protocol (NTP), Precision Time Protocol (PTP) or other protocols. Optionally, the error of the timestamp in this embodiment can be in ns, and the expected synchronization error is less than 10ns.

[0044] Optionally, the first deep learning model in this embodiment can use the deep learning technology of time series to more accurately integrate the motion information of the vehicle and the motion information of the wearable device. Among them, the deep learning technology of time series: has the ability of end-to-end learning, can directly learn the dynamic evolution law of the state from the original data without the need to model the system in advance; can flexibly model non-Gaussian noise, which is more suitable for various situations in the real world; and is suitable for the processing of sequence data, and can perform state estimation and prediction on time series data.

[0045] Step S240: Acquire at least one image feature collected by at least one sensor of the wearable device.

[0046] In this implementation, the wearable device may be equipped with at least one sensor, and the at least one sensor may include an RGB camera, a laser sensor, a structured light sensor, and the like.

[0047] In the process of acquiring the visual learning features of the wearable device relative to the vehicle, in order to more completely acquire the visual features of the wearable device relative to the vehicle, at least one image feature collected by at least one sensor of the wearable device may be acquired.

[0048] Optionally, different types of sensors may acquire different image features, and the same sensor may acquire different image features at different times. The at least one image feature acquired by at least one sensor of the wearable device may be of the same time period (time) or different time periods (times).

[0049] Step S250: input the at least one image feature into a second deep learning model, and output a visual learning feature of the wearable device relative to the vehicle, wherein the second deep learning model is a deep learning model obtained based on sensor information encoding.

[0050] Among them, the image features collected by different sensors may be MxN images. In order to facilitate subsequent processing, at least one of the above image features can be input into the second deep learning model to learn the commonalities between the image features collected by different sensors through the second deep learning model, and extract the information that can be processed by the mathematical model accordingly. Then, the visual learning features of the wearable device relative to the vehicle output by the second deep learning model can be obtained. It should be noted that the second deep learning model is a deep learning model pre-encoded based on sensor information.

[0051] Optionally, the output of the convolutional layer or the output of the fully connected layer of the second deep learning model can be used as a visual learning feature of the wearable device relative to the vehicle.

[0052] Optionally, the second deep learning model and the first deep learning model are different deep learning models.

[0053] Step S260: Input the motion learning features and the visual learning features into a third deep learning model, and output the position and posture features of the wearable device relative to the vehicle as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle. The third deep learning model is a state estimation model based on 6 degrees of freedom.

[0054] In the process of fusing the above-mentioned motion learning features with the above-mentioned visual learning features, as an implementation method, the motion learning features and the visual learning features can be input into a third deep learning model, so as to obtain the position and posture features of the wearable device relative to the vehicle by combining the two types of deep learning features through the third deep learning model, and then use the position and posture features as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle. Among them, the third deep learning model can be a state estimation model based on 6 degrees of freedom.

[0055] Optionally, the third deep learning model and the first deep learning model are different deep learning models, and the third deep learning model and the second deep learning model are different deep learning models.

[0056] It should be noted that the above differences can be understood as the first deep learning model, the second deep learning model and the third deep learning model have different data analysis capabilities because they have received different training, that is, the first deep learning model is used to analyze motion information, the second deep learning model is used to analyze visual information, and the third deep learning model is used to analyze position and posture.

[0057] The present embodiment provides a data processing method, which includes obtaining the motion parameters of the vehicle relative to the ground; obtaining the motion parameters of the wearable device inside the vehicle relative to the vehicle; inputting the motion parameters of the vehicle relative to the ground and the motion parameters of the wearable device relative to the vehicle into a first deep learning model, and outputting the motion learning features of the wearable device relative to the vehicle, wherein the first deep learning model is a deep learning model based on motion information; obtaining at least one image feature collected by at least one sensor of the wearable device; inputting the at least one image feature into a second deep learning model, and outputting the visual learning features of the wearable device relative to the vehicle, wherein the second deep learning model is a deep learning model obtained based on sensor information encoding; inputting the motion learning features and the visual learning features into a third deep learning model, and outputting the position and posture features of the wearable device relative to the vehicle as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle, wherein the third deep learning model is a state estimation model based on 6 degrees of freedom. Thus, by comprehensively utilizing multiple deep learning features, the position and posture estimation results of the wearable device inside the vehicle can be more accurate, thereby improving the user experience.

[0058] By using deep learning models with different data analysis capabilities to analyze information from wearable devices or vehicles, the rigor, accuracy, and reliability of data processing can be improved.

[0059] See also Figure 3 , is a data processing method provided in yet another embodiment of the present application, the method comprising:

[0060] Step S310: Obtain the motion parameters of the vehicle relative to the ground.

[0061] The specific implementation of step S310 may refer to the relevant description of step S210 in the aforementioned embodiment, which will not be repeated here.

[0062] Step S320: Obtain motion parameters of the wearable device inside the vehicle relative to the vehicle.

[0063] The specific implementation of step S320 may refer to the relevant description of step S220 in the aforementioned embodiment, which will not be repeated here.

[0064] Step S330: input the motion parameters of the vehicle relative to the ground and the motion parameters of the wearable device relative to the vehicle into a first deep learning model, and output a motion learning feature of the wearable device relative to the vehicle, wherein the first deep learning model is a deep learning model based on motion information.

[0065] The specific implementation of step S330 may refer to the relevant description of step S230 in the aforementioned embodiment, which will not be repeated here.

[0066] Step S340: Acquire at least one image feature collected by at least one sensor of the wearable device.

[0067] The specific implementation of step S340 may refer to the relevant description of step S240 in the aforementioned embodiment, which will not be repeated here.

[0068] Step S350: input the at least one image feature into a second deep learning model, and output a visual learning feature of the wearable device relative to the vehicle, wherein the second deep learning model is a deep learning model obtained based on sensor information encoding.

[0069] The specific implementation of step S350 may refer to the relevant description of step S250 in the aforementioned embodiment, which will not be repeated here.

[0070] Step S360: Input the motion learning features and the visual learning features into a third deep learning model, and output the position and posture features of the wearable device relative to the vehicle as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle. The third deep learning model is a state estimation model based on 6 degrees of freedom.

[0071] The specific implementation of step S360 may refer to the relevant description of step S260 in the aforementioned embodiment, which will not be repeated here.

[0072] Step S370: Obtain 6-DOF position and posture training data.

[0073] In order to enable the above-mentioned deep learning models with different data analysis capabilities to have more accurate reasoning capabilities, as an implementation method, in the process of obtaining the position and posture characteristics of the wearable device relative to the vehicle, 6-degree-of-freedom position and posture training data can be simultaneously obtained to perform end-to-end model training on the above-mentioned deep learning models based on the training data.

[0074] As an implementation method, a posture calibration system based on a motion capture model can be used (which can be provided by a third party) to obtain high-precision motion capture data as the 6DOF true value for training (i.e., 6-degree-of-freedom position and posture training data).

[0075] Step S380: Obtain error parameters between the 6-DOF position and posture training data and the DOF positioning estimation parameters.

[0076] In this implementation, it is assumed that the degree of freedom positioning estimation parameters output by the third deep learning model have been preliminarily obtained. In this way, in order to ensure the effectiveness of model training, the first deep learning model, the second deep learning model and the third deep learning model can use the same error function (i.e., loss function). In this way, in order to further improve the effectiveness of model training, the error parameters between the 6-degree-of-freedom position and posture training data and the degree-of-freedom positioning estimation parameters can be obtained, and the error parameters can be used as the loss of the above-mentioned loss function.

[0077] Step S390: Based on the error parameter, the parameters of the first deep learning model, the second deep learning model and the third deep learning model are adjusted respectively to obtain a first target model, a second target model and a third target model.

[0078] Then, as an implementation method, the parameters of the first deep learning model, the second deep learning model, and the third deep learning model can be adjusted respectively based on the error parameters, so that the parameters of the first deep learning model, the second deep learning model, and the third deep learning model can be optimized in the same training system.

[0079] Optionally, during the training process, the model can automatically learn the correlation between image depth features and motion sensor information, thereby achieving more accurate prediction or inference.

[0080] In this way, the first deep learning model when the loss function converges can be determined as the first target model, the second deep learning model when the loss function converges can be determined as the second target model, and the third deep learning model when the loss function converges can be determined as the third target model. In this way, the above process can be re-executed with the first target model, the second target model, and the third target model to obtain new degree of freedom positioning estimation parameters.

[0081] Optionally, in order to improve the accuracy of data processing, after obtaining the first target model, the second target model and the third target model, some evaluation indicators (such as accuracy, precision, recall rate, etc.) can be used to evaluate the effectiveness of the model. After the evaluation is passed, these models can be deployed to actual applications to predict or infer new images and motion data to meet application needs.

[0082] In a specific application scenario, see Figure 4 , shows an example flow chart of the data processing method provided by the embodiment of the present application. Figure 4As shown, under the condition of complying with the time synchronization protocol, the motion information of the vehicle and the motion information of the wearable device can be input into the deep learning model 1 to learn the deep learning features of the motion (i.e., motion learning features); at the same time, the acquired wearable device image and other sensor information of the wearable device are input into the deep learning model 2 to learn the deep learning features of other sensors (i.e., visual learning features); in this way, the output of the deep learning model 1 and the output of the deep learning model 2 can be input into the deep learning model 3 together to learn the 6-DOF position and posture of the wearable device inside the vehicle. In this way, in order to improve the accuracy of data processing, the 6-DOF position and posture training data can be obtained, and the deep learning model 1, the deep learning model 2, and the deep learning model 3 are end-to-end trained based on the training data and the output of the deep learning model 3 to improve the reasoning ability of each deep learning model, thereby improving the accuracy of data processing.

[0083] The present embodiment provides a data processing method, which obtains motion parameters of a vehicle relative to the ground; obtains motion parameters of a wearable device inside the vehicle relative to the vehicle; inputs the motion parameters of the vehicle relative to the ground and the motion parameters of the wearable device relative to the vehicle into a first deep learning model, and outputs motion learning features of the wearable device relative to the vehicle, wherein the first deep learning model is a deep learning model based on motion information; obtains at least one image feature collected by at least one sensor of the wearable device; inputs the at least one image feature into a second deep learning model, and outputs visual learning features of the wearable device relative to the vehicle, wherein the second deep learning model is a deep learning model based on sensor information encoding. The obtained deep learning model; input the motion learning features and the visual learning features into the third deep learning model, and output the position and posture features of the wearable device relative to the vehicle as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle, and the third deep learning model is a state estimation model based on 6 degrees of freedom; obtain the 6-degree-of-freedom position and posture training data; obtain the error parameters between the 6-degree-of-freedom position and posture training data and the degree-of-freedom positioning estimation parameters; adjust the parameters of the first deep learning model, the second deep learning model and the third deep learning model based on the error parameters, and obtain the first target model, the second target model and the third target model. Therefore, by comprehensively utilizing multiple deep learning features, the position and posture estimation results of the wearable device inside the vehicle can be more accurate, thereby improving the user experience.

[0084] End-to-end model training based on comprehensive deep learning features can improve the reasoning ability of each deep learning model and thus improve the accuracy of data processing.

[0085] See also Figure 5 The present application provides a data processing device 400, the data processing device 400 comprising:

[0086] The motion feature acquisition module 410 is used to acquire motion learning features of a wearable device inside a vehicle relative to the vehicle.

[0087] As an implementation method, the motion feature acquisition module 410 can be used to obtain motion parameters of the vehicle relative to the ground; obtain motion parameters of a wearable device inside the vehicle relative to the vehicle; input the motion parameters of the vehicle relative to the ground and the motion parameters of the wearable device relative to the vehicle into a first deep learning model, and output motion learning features of the wearable device relative to the vehicle, wherein the first deep learning model is a deep learning model based on motion information.

[0088] Optionally, the motion parameters of the vehicle relative to the ground may include at least one of the vehicle's motion speed, acceleration, angular velocity, position and rotation information.

[0089] Among them, obtaining the motion parameters of the wearable device inside the vehicle relative to the vehicle may include: obtaining a first motion parameter obtained by the IMU of the wearable device and a second motion parameter observed by the camera of the wearable device; fusing the first motion parameter with the second parameter to obtain the motion parameter of the wearable device relative to the vehicle.

[0090] The visual feature acquisition module 420 is used to acquire the visual learning features of the wearable device relative to the vehicle.

[0091] As an implementation mode, the visual feature acquisition module 420 can be specifically used to acquire at least one image feature collected by at least one sensor of the wearable device; input the at least one image feature into a second deep learning model, and output a visual learning feature of the wearable device relative to the vehicle, wherein the second deep learning model is a deep learning model obtained based on sensor information encoding.

[0092] The degree of freedom positioning estimation parameter acquisition module 430 is used to fuse the motion learning feature with the visual learning feature to obtain the position and posture feature of the wearable device relative to the vehicle as the degree of freedom positioning estimation parameter of the wearable device inside the vehicle.

[0093] As an implementation mode, the degree of freedom positioning estimation parameter acquisition module 430 can be specifically used to input the motion learning features and the visual learning features into a third deep learning model, and output the position and posture features of the wearable device relative to the vehicle as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle. The third deep learning model is a state estimation model based on 6 degrees of freedom.

[0094] In this embodiment, the device 400 can also protect a data adjustment module for obtaining 6-degree-of-freedom position and posture training data; obtaining error parameters between the 6-degree-of-freedom position and posture training data and the degree-of-freedom positioning estimation parameters; and adjusting the parameters of the first deep learning model, the second deep learning model, and the third deep learning model based on the error parameters to obtain a first target model, a second target model, and a third target model.

[0095] The following will be combined Figure 6 A vehicle provided in the present application is described.

[0096] See also Figure 6 Based on the above data processing method and device, the embodiment of the present application also provides another vehicle 100 that can execute the above data processing method. The vehicle 100 includes one or more (only one is shown in the figure) processors 102, a memory 104, and a data acquisition module 106 that are coupled to each other. Among them, the memory 104 stores a program that can execute the content of the above embodiment, and the processor 102 can execute the program stored in the memory 104.

[0097] Among them, the processor 102 may include one or more processing cores. The processor 102 uses various interfaces and lines to connect various parts in the entire vehicle 100, and executes various functions and processes data of the vehicle 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 104, and calling data stored in the memory 104. Optionally, the processor 102 can be implemented in at least one hardware form of a network processor (Neural network Processing Unit, NPU), a digital signal processing (Digital Signal Processing, DSP), a field programmable gate array (Field-Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). The processor 102 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU), a network processor (Neural network Processing Unit, NPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; the NPU is responsible for processing multimedia data such as video and images; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 102, but may be implemented separately through a communication chip.

[0098] The memory 104 may include a random access memory (RAM) or a read-only memory (ROM). The memory 104 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 104 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the vehicle 100 during use (such as a phone book, audio and video data, chat record data), etc.

[0099] The data acquisition module 106 can be used to obtain the driving status information of the vehicle 100. The data acquisition module 106 can be a laser radar, a camera, a sensor, a GPS (Global Positioning System) navigation, etc.

[0100] An embodiment of the present application provides a computer-readable storage medium in which a program code is stored, and the program code can be called by a processor to execute the method described in the above method embodiment.

[0101] The computer readable storage medium may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium includes a non-transitory computer-readable storage medium. The computer readable storage medium has storage space for program codes that execute any of the method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0102] In summary, the data processing method, device and electronic device provided by the present application obtain the motion learning features of the wearable device inside the vehicle relative to the vehicle; obtain the visual learning features of the wearable device relative to the vehicle; and fuse the motion learning features with the visual learning features to obtain the position and posture features of the wearable device relative to the vehicle as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle. Thus, by comprehensively utilizing multiple deep learning features, the position and posture estimation results of the wearable device inside the vehicle can be more accurate, thereby improving the user experience.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining motion learning characteristics of a wearable device inside a vehicle relative to the vehicle; Acquiring visual learning features of the wearable device relative to the vehicle; The motion learning feature is integrated with the visual learning feature to obtain a position and posture feature of the wearable device relative to the vehicle as a degree of freedom positioning estimation parameter of the wearable device inside the vehicle.

2. The method according to claim 1, characterized in that: The obtaining of motion learning characteristics of a wearable device inside a vehicle relative to the vehicle includes: Obtain the motion parameters of the vehicle relative to the ground; Obtaining motion parameters of the wearable device inside the vehicle relative to the vehicle; The motion parameters of the vehicle relative to the ground and the motion parameters of the wearable device relative to the vehicle are input into a first deep learning model, and the motion learning features of the wearable device relative to the vehicle are output. The first deep learning model is a deep learning model based on motion information.

3. The method according to claim 2, characterized in that The obtaining of the motion parameters of the wearable device inside the vehicle relative to the vehicle includes: Acquire a first motion parameter obtained by an IMU of the wearable device and a second motion parameter observed by a camera of the wearable device; The first motion parameter is fused with the second parameter to obtain a motion parameter of the wearable device relative to the vehicle.

4. The method according to claim 2, characterized in that: The obtaining of the visual learning feature of the wearable device relative to the vehicle includes: Acquire at least one image feature collected by at least one sensor of the wearable device; The at least one image feature is input into a second deep learning model, and a visual learning feature of the wearable device relative to the vehicle is output, wherein the second deep learning model is a deep learning model obtained based on sensor information encoding.

5. The method according to claim 4, characterized in that The fusing of the motion learning feature with the visual learning feature to obtain the position and posture feature of the wearable device relative to the vehicle as a degree of freedom positioning estimation parameter of the wearable device inside the vehicle includes: The motion learning features and the visual learning features are input into a third deep learning model, and the position and posture features of the wearable device relative to the vehicle are output as the degree of freedom positioning estimation parameters of the wearable device inside the vehicle. The third deep learning model is a state estimation model based on 6 degrees of freedom.

6. The method according to claim 5, characterized in that The method further comprises: Obtain 6-DOF position and posture training data; Obtaining error parameters between the 6-DOF position and posture training data and the DOF positioning estimation parameters; Based on the error parameter, the parameters of the first deep learning model, the second deep learning model and the third deep learning model are adjusted respectively to obtain a first target model, a second target model and a third target model.

7. The method according to claim 2, characterized in that The motion parameters of the vehicle relative to the ground include at least one of the vehicle's motion speed, acceleration, angular velocity, position, and rotation information.

8. A data processing device, characterized in that: The device comprises: A motion feature acquisition module, used to acquire motion learning features of a wearable device inside a vehicle relative to the vehicle; A visual feature acquisition module, used to acquire visual learning features of the wearable device relative to the vehicle; The degree of freedom positioning estimation parameter acquisition module is used to fuse the motion learning feature with the visual learning feature to obtain the position and posture feature of the wearable device relative to the vehicle as the degree of freedom positioning estimation parameter of the wearable device inside the vehicle.

9. An electronic device, characterized in that: including one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, wherein when the program code is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images