Electronic device and hybrid reality production method
By using the inertial sensing data mapping model in electronic devices, the problem of high-expenditure computing and inability to import virtual content in existing hybrid reality technologies is solved, and the functions of low computing volume and real-time virtual content import are realized.
Patent Information
- Application Number
- CN202311373104.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-10-11
- Filing Date
- 2023-10-23
- Publication Date
- 2025-04-11
AI Technical Summary
Existing hybrid reality (MR) technologies require a large number of spatial anchors and high-power computing, and cannot import virtual content in real space instantly.
An electronic device is provided, equipped with an inertial sensing element and a processor, to establish a hybrid real-life environment coordinate system through an inertial sensing data mapping model, calculate the starting coordinate point and spatial work point, and execute a three-dimensional scene editing program.
It reduces the need for computing volume, realizes the function of instantly importing virtual content in real space, and improves the intuition and convenience of hybrid real-life production.
Smart Images
Figure CN119806312A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an electronic device and a mixed reality production method, and in particular to an electronic device and a mixed reality production method that perform positioning through an inertial sensing data mapping model. Background Art
[0002] The establishment of the current mixed reality (MR) environment coordinate system often requires the assistance of a large number of spatial anchor points and real-time image recognition algorithms, which not only consumes a large amount of computing power, but also makes it impossible to import virtual content into the real space in real time.
[0003] Therefore, how to provide a mixed reality production method and device that has low computing power requirements and can instantly import virtual content into real space is one of the problems to be solved in the field. Summary of the invention
[0004] Therefore, the present invention provides an electronic device and a mixed reality production method for solving the above problems.
[0005] One aspect of the present invention is to provide an electronic device that can move in real space. The electronic device includes an inertial sensing element and a processor. The inertial sensing element is used to obtain first inertial sensing data when the electronic device moves. The processor is electrically coupled to the inertial sensing element and is used to perform the following steps: establish a corresponding mixed reality environment coordinate system based on the real space, and calculate the starting coordinate point of the electronic device in the mixed reality environment coordinate system; convert the first inertial sensing data into a first movement vector in the mixed reality environment coordinate system according to the inertial sensing data mapping model; calculate the first space operation point in the mixed reality environment coordinate system according to the starting coordinate point and the first movement vector in the mixed reality environment coordinate system; and execute a three-dimensional scene editing program with the first space operation point.
[0006] In some embodiments, the inertial sensing data mapping model is used to generate the first motion vector according to a normalized inertial sensing data obtained after the first inertial sensing data is normalized.
[0007] In some embodiments, it is characterized in that the training of the inertial sensing data mapping model includes the following: collecting a plurality of test inertial sensing data after a plurality of different test electronic devices move along the same test path, wherein the plurality of test inertial sensing data are respectively generated by different inertial sensing elements and have different data ranges; and training the inertial sensing data mapping model based on the plurality of test inertial sensing data, so that the inertial sensing data mapping model maps the plurality of test inertial sensing data to a test movement vector of the same value.
[0008] In some embodiments, it is characterized in that the processor is further used to perform the following steps: when the electronic device moves to a first position in the real space, the inertial sensing data mapping model converts the first inertial sensing data into the first movement vector in the mixed reality environment coordinate system, and obtains the first spatial operation point in the mixed reality environment coordinate system by adding the starting coordinate point and the first movement vector, and executes the three-dimensional scene editing program at the first spatial operation point at the moment or afterwards, wherein the three-dimensional scene editing program includes one of an event program, a special effect action program and a virtual object rendering program, and the three-dimensional scene editing program is stored corresponding to the first spatial operation point.
[0009] In some embodiments, it is characterized in that the inertial sensing element is further used to obtain a second inertial sensing data when the electronic device moves, and the processor is further used to perform the following steps: when the electronic device moves from the first position to a second position in the real space, the inertial sensing data mapping model converts the second inertial sensing data into a second movement vector in the mixed reality environment coordinate system, and obtains a second spatial operation point in the mixed reality environment coordinate system by adding the first spatial operation point and the second movement vector, executes the three-dimensional scene editing program at the second spatial operation point at the moment or afterwards, and stores the three-dimensional scene editing program corresponding to the second spatial operation point.
[0010] In some embodiments, it is characterized in that the electronic device further includes: an output device, electrically coupled to the processor, for outputting one of the event program, the special effect action program and the virtual object rendering program on the output device when the processor executes the three-dimensional scene editing program; and an input device, electrically coupled to the processor and a memory, for receiving and transmitting a selection signal to the processor after the processor executes the three-dimensional scene editing program.
[0011] In some embodiments, it is characterized in that the event program, the special effect action program and the virtual object rendering program included in the three-dimensional scene editing program are multiple event programs, multiple special effect action programs, and multiple virtual object rendering programs; the output device includes a screen device, and the screen device displays an edit list, and the content of the edit list is selected from a part of the multiple event programs, the special effect action program and the virtual object rendering program.
[0012] In some embodiments, it is characterized in that the processor is further used to perform the following steps: execute the three-dimensional scene editing program, and when storing two of the starting coordinate point, the first spatial operation point and the second spatial operation point, store the three-dimensional scene editing program together.
[0013] In some embodiments, it is characterized in that the processor further performs the following steps: extracting a feature anchor point from an image captured by a camera device; and establishing a corresponding mixed reality environment coordinate system based on the feature anchor point in the real space as the origin of a coordinate system of the mixed reality environment.
[0014] In some embodiments, the first movement vector at least includes a movement distance and a rotation angle, wherein the first movement vector includes a six degrees of freedom (6DOF) motion trajectory of the first spatial operating point relative to the starting coordinate point.
[0015] Another aspect of the present invention is to provide a mixed reality production method, which is applicable to electronic devices. The mixed reality production method includes the following steps: establishing a corresponding mixed reality environment coordinate system according to the feature anchor point of the real space, and calculating the starting coordinate point of the electronic device in the mixed reality environment coordinate system; obtaining the first inertial sensing data when the electronic device moves; converting the first inertial sensing data into a first movement vector in the mixed reality environment coordinate system according to the inertial sensing data mapping model; calculating the first space operation point in the mixed reality environment coordinate system according to the starting coordinate point and the first movement vector in the mixed reality environment coordinate system; and executing the three-dimensional scene editing program with the first space operation point.
[0016] In some embodiments, the method further comprises: generating the first movement vector based on the inertial sensing data mapping model according to a normalized inertial sensing data obtained after the first inertial sensing data is normalized.
[0017] In some embodiments, the method is characterized in that it further includes: collecting a plurality of test inertial sensing data after a plurality of different test electronic devices move along the same test path, wherein the plurality of test inertial sensing data are respectively generated by different inertial sensing elements and have different data ranges; and training the inertial sensing data mapping model based on the plurality of test inertial sensing data, so that the inertial sensing data mapping model maps the plurality of test inertial sensing data to a test movement vector of the same value.
[0018] In some embodiments, it is characterized in that it also includes: when the electronic device moves to a first position in the real space, the inertial sensing data mapping model converts the first inertial sensing data into the first movement vector in the mixed reality environment coordinate system; obtains a first spatial operation point in the mixed reality environment coordinate system according to the addition of the starting coordinate point and the first movement vector; and executes the three-dimensional scene editing program at the first spatial operation point at the moment or afterwards, wherein the three-dimensional scene editing program includes one of an event program, a special effect action program and a virtual object rendering program, and stores the three-dimensional scene editing program corresponding to the first spatial operation point.
[0019] In some embodiments, it is characterized in that it also includes: when the electronic device moves from the first position to a second position in the real space, the inertial sensing data mapping model converts a second inertial sensing data into a second movement vector in the mixed reality environment coordinate system; a second space operation point in the mixed reality environment coordinate system is obtained by adding the first space operation point and the second movement vector; and the three-dimensional scene editing program is executed at the second space operation point at the moment or afterwards, and the three-dimensional scene editing program is stored corresponding to the second space operation point.
[0020] In some embodiments, it is characterized in that it also includes: when executing the three-dimensional scene editing program, outputting one of the event program, the special effect action program and the virtual object rendering program on an output device; and after executing the three-dimensional scene editing program, receiving and transmitting a selection signal by an input device.
[0021] In some embodiments, it is characterized in that the event program, the special effect action program and the virtual object rendering program included in the three-dimensional scene editing program are multiple event programs, multiple special effect action programs, and multiple virtual object rendering programs, wherein the mixed reality production method further includes: a screen device of the output device displays an editing list, and the content of the editing list is selected from a part of the multiple event programs, the special effect action program and the virtual object rendering program.
[0022] In some embodiments, it is characterized in that it further includes: executing the three-dimensional scene editing program, and when storing two of the starting coordinate point, the first space operation point and the second space operation point, the three-dimensional scene editing program is stored together.
[0023] In some embodiments, the method further comprises: extracting a feature anchor point from an image captured by a camera device; and establishing a corresponding coordinate system of the mixed reality environment based on the feature anchor point in the real space as the origin of a coordinate system of the mixed reality environment.
[0024] In some embodiments, the first movement vector includes a movement distance and a rotation angle, wherein the first movement vector includes a six-degree-of-freedom (6DOF) motion trajectory of the first spatial operating point relative to the starting coordinate point.
[0025] As can be seen from the above, the present invention provides an electronic device and a mixed reality production method. Compared with the traditional mixed reality production method, there is no need to establish a large number of identification space anchor points, and there is no need to perform high-efficiency operations such as real-time matching and positioning (such as SLAM or feature point recognition). The implementation method of the present invention only needs to identify a single starting anchor, and use an inertial sensing data mapping model and inertial sensing data to calculate the starting coordinate point and the spatial operation point, which can greatly reduce the amount of calculation compared to the traditional mixed reality production method. In addition, the implementation method of the present invention can edit mixed reality events, special effects actions, and virtual object rendering programs in real time while the electronic device is moving, and can be previewed in real time on the electronic device, which is more intuitive and more convenient when performing mixed reality production. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to make the above and other objects, features, advantages and embodiments of the present disclosure more clearly understood, the accompanying drawings are described as follows:
[0027] Figure 1 is a schematic diagram of a real space of an electronic device and its operation according to some embodiments of the present invention;
[0028] Figure 2 is a schematic diagram of an electronic device according to some embodiments of the present invention;
[0029] Figure 3 is a flow chart of a mixed reality production method according to some embodiments of the present invention;
[0030] Figure 4 is a schematic diagram of an edit list according to some embodiments of the present invention;
[0031] Figure 5 is a schematic diagram of a mixed reality production system according to some embodiments of the present invention; and
[0032] Figure 6 is a schematic diagram of a mixed reality production system according to some embodiments of the present invention.
[0033]
Explanation of symbols
[0034] E: Real Space
[0035] ES: Mixed Reality Environment Coordinate System
[0036] EC: Environmental Characteristics
[0037] ANC: Feature Anchor
[0038] P0: origin of coordinate system
[0039] P1: Starting coordinate point
[0040] P2, P3: Space operation point
[0041] A1, A2, A3: Location
[0042] D1, D2: Inertial sensing data
[0043] T1, T2: moving vector
[0044] 100: Electronic devices
[0045] 110: Inertial sensing element
[0046] 120: Processor
[0047] 152: Camera Device
[0048] 140: Output device
[0049] 150: Input device
[0050] 160: Memory
[0051] 300:Mixed Reality Production Methods
[0052] S310, S320, S330, S340: Steps
[0053] S350, S355, S360, S370: Steps
[0054] S380, S385, S390: Steps
[0055] 400: Edit List
[0056] 500,600: Mixed reality production system
[0057] 510: Training device
[0058] 530A, 530B: Electronic devices
[0059] 610: APP production device
[0060] 630: Electronic devices DETAILED DESCRIPTION
[0061] The following disclosure provides many different embodiments or examples for implementing different features of the present invention. The elements and configurations in the specific examples are used to simplify the present invention in the following discussion. Any examples discussed are only used for illustrative purposes and do not limit the scope and significance of the present invention or its examples in any way.
[0062] See also Figure 1 . Figure 1 FIG. 1 is a schematic diagram of an electronic device 100 and a real space E in which the electronic device 100 and the electronic device 100 operate according to some embodiments of the present invention. Figure 1 As shown, the electronic device 100 can move in the real space E. Based on the real space E, the electronic device 100 establishes a mixed reality environment coordinate system ES corresponding to the real space E. Based on the position of the electronic device 100 in the real space E, the corresponding starting coordinate point P1 and spatial operation points P2 and P3 in the mixed reality environment coordinate system ES are calculated, and a three-dimensional scene editing program is executed based on the starting coordinate point P1 and the spatial operation points P2 and P3.
[0063] The details of establishing the starting coordinate point P1 and the spatial operation points P2 and P3 and executing the 3D scene editing program will be described below. Figures 2 to 3 Explain together.
[0064] See also Figure 2 In some embodiments, the electronic device 100 includes an inertial sensing element 110 , a processor 120 , an output device 140 , an input device 150 , and a memory 160 . In some embodiments, the input device 150 includes a camera device 152 .
[0065] In terms of connection, the inertial sensing element 110, the output device 140, the input device 150 and the memory 160 are respectively coupled to the processor 120. Figure 2 The detailed operation of the electronic device 100 will be described below. Figure 3 Explain together.
[0066] Please also read Figure 3 . Figure 3 is a flow chart of a mixed reality production method 300 according to some embodiments of the present invention.
[0067] The mixed reality production method 300 can be applied to Figure 1 and Figure 2 The electronic device 100 in the embodiment of the present invention has the same or similar structure as the electronic device 100 in the embodiment of the present invention. Figure 2 The operation method is described by taking the example of Figure 2 The application is limited.
[0068] It should be noted that, in some embodiments, the mixed reality production method 300 may also be implemented as a computer program and stored in a computer program such as Figure 2 In the memory 160, the computer, electronic device, or the aforementioned Figure 2 The processor 120 in the electronic device 100 reads the computer program and executes the operation method. The processor 120 may include one or more chips. The memory 160 may be a read-only memory, a flash memory, a floppy disk, a hard disk, an optical disk, a flash disk, a magnetic tape, a database accessible via a network, or a non-transitory computer-readable recording medium having the same function that can be easily thought of by those skilled in the art.
[0069] In addition, it should be understood that the operations of the mixed reality production method 300 mentioned in this embodiment, except for those whose order is specifically described, can be adjusted in sequence according to actual needs, and can even be executed simultaneously or partially simultaneously.
[0070] Furthermore, in different embodiments, these operations may be adaptively increased, replaced, and / or omitted.
[0071] See also Figure 3 The mixed reality production method 300 includes the following steps S310 to S390. Figure 1 and Figure 2 Provide explanation.
[0072] In step S310, an image of the real space is captured, and feature anchor points are extracted from the image. Figure 2 The camera device 152 shown in FIG. Figure 1 The image captured by the camera device 152 includes: Figure 1 Next, the processor 120 extracts a feature anchor point ACH according to the environment feature EC.
[0073] In step S320, a corresponding mixed reality environment coordinate system is established according to the feature anchor point in the real space, and the starting coordinate point of the electronic device in the mixed reality environment coordinate system is calculated. Figure 2 The processor 120 shown in FIG. 1 establishes a corresponding mixed reality environment coordinate system ES according to the feature anchor point ACH. In some embodiments, the mixed reality environment coordinate system ES uses the center of the feature anchor point ACH as the coordinate system origin P0, and the mixed reality environment coordinate system ES is established based on the coordinate system origin.
[0074] According to the coordinate system origin P0, the processor 120 calculates the starting coordinate point P1 of the electronic device 100 in the mixed reality environment coordinate system. In one embodiment, the processor 120 calculates the starting coordinate point P1 by simultaneous localization and mapping (SLAM), but the implementation of the present invention is not limited to the above.
[0075] In step S330, the first inertial sensing data of the electronic device when it moves is obtained. Figure 1 When moving in the real space E shown in Figure 2 The inertial sensing element 110 shown in FIG. 1 obtains inertial sensing data when the electronic device 100 moves. For example, when the electronic device 100 is moved by Figure 1 When the position A1 in the real space E shown in FIG. 1 moves to the position A2 in the real space E, the inertial sensing element 110 obtains the inertial sensing data D1 of the electronic device 100 moving from the position A1 in the real space E to the position A2 in the real space E.
[0076] In step S340, the first inertial sensing data is converted into a first motion vector in the mixed reality environment coordinate system according to the inertial sensing data mapping model. In some embodiments, the inertial sensing data mapping model is stored in a Figure 2 The memory 160 is shown for reading by the processor 120 .
[0077] In some embodiments, the inertial sensing data mapping model is trained by the training device based on different training electronic devices. Figure 5 Explain together.
[0078] In some embodiments, the processor 120 of the electronic device 100 is composed of Figure 5 The training device 510 in the embodiment loads the inertial sensing data mapping model.
[0079] In some embodiments, when the electronic device 100 moves from a position A1 in the real space E to a position A2 in the real space E, the inertial sensing data mapping model converts the inertial sensing data D1 into a movement vector T1 in the mixed reality environment coordinate system ES.
[0080] In some embodiments, the processor 120 of the electronic device 100 first normalizes the inertial sensing data D1 to map it to a uniform value range, and then inputs the normalized inertial sensing data D1 to the inertial sensing data mapping model to obtain the motion vector T1. In some embodiments, the uniform value range is a value range preset when the inertial sensing data mapping model is trained.
[0081] In some embodiments, the movement vector T1 includes a movement distance and a rotation angle. The movement vector T1 includes a six degrees of freedom (6DOF) motion trajectory of the spatial working point P2 relative to the starting coordinate point P1.
[0082] In step S350, a first spatial operation point in the mixed reality environment coordinate system is calculated according to the starting coordinate point and the first motion vector in the mixed reality environment coordinate system. For example, in some embodiments, the processor 120 adds the starting coordinate point P1 and the motion vector T1 to obtain the spatial operation point P2 in the mixed reality environment coordinate system ES.
[0083] In some embodiments, the processor 120 calculates the spatial operation point P2 according to the following formula (1).
[0084] Formula (1): P2 = P1 + T1.
[0085] In the above formula (1), P1 and P2 are spatial operation points, and T1 is the movement vector.
[0086] In step S355, a 3D scene editing program is executed at the first spatial operation point. In some embodiments, the 3D scene editing program includes one of an event program, a special effect action program, and a virtual object rendering program.
[0087] In some embodiments, step S355 is performed when the electronic device 100 moves to the position A2 and the processor 120 calculates the spatial operation point P2.
[0088] In some embodiments, the event program, special effect action program, and virtual object rendering program included in the three-dimensional scene editing program include multiple event programs, multiple special effect action programs, and multiple virtual object rendering programs.
[0089] In some embodiments, after the processor 120 executes the 3D scene editing program at the position A2 in the real space, the 3D scene editing program is stored in the corresponding space operation point P2.
[0090] In some embodiments, when the processor 120 executes the 3D scene editing program, the screen device of the output device 140 of the electronic device 100 displays the editing list. The content of the editing list is selected from a part of a plurality of event programs, special effect action programs, and virtual object rendering programs. Figure 4 . Figure 4 FIG. 4 is a schematic diagram of an edit list 400 according to some embodiments of the present invention. Figure 4The edit list 400 shown includes a selection list of operation points, events, virtual objects, special effects, etc. The user can select the 3D scene to be edited through the selection list, and the processor 120 executes part of the event program, special effects program, and virtual object rendering program in the 3D scene editing program according to the content of the edit list 400.
[0091] In some embodiments, after the processor 120 executes the 3D scene editing program, the input device 150 receives and transmits a selection signal to the processor 120. The processor 120 sequentially selects the operation point, event, virtual object, special effect action, etc. selected in the editing list 400 to execute the 3D scene editing program.
[0092] In some embodiments, when the processor 120 executes the 3D scene editing program, the output device 140 of the electronic device 100 outputs one of the event program, special effect action program, and virtual object rendering program in the 3D scene editing program. Figure 4 The screen device of the output device 140 of the electronic device 100 is further used to display a preview image in real time when the 3D scene editing program is executed.
[0093] Please come back Figure 1 and Figure 2 In some embodiments, after the processor 120 executes the 3D scene editing program based on the spatial operation point P2 , the processor 120 stores the starting coordinate point P1 and the spatial operation point P2 in the memory 160 when storing the 3D scene editing program.
[0094] In step S360 , second inertial sensing data of the electronic device when it moves is obtained.
[0095] Please also read Figure 1 In some embodiments, the electronic device 100 moves from position A2 to position A3 in the real space E. When the electronic device 100 moves from position A2 in the real space E to position A3 in the real space E, the inertial sensing element 110 obtains inertial sensing data D2 of the electronic device 100 moving from position A2 in the real space E to position A3 in the real space E.
[0096] In step S370 , the second inertial sensing data is converted into a second motion vector in the mixed reality environment coordinate system according to the inertial sensing data mapping model. After the inertial sensing element 110 obtains the inertial sensing data D2 , the inertial sensing data mapping model converts the inertial sensing data D2 into a motion vector T2 in the mixed reality environment coordinate system ES.
[0097] In some embodiments, the processor 120 of the electronic device 100 first normalizes the inertial sensing data D2 to map it to a uniform value range, and then inputs the normalized inertial sensing data D2 after normalization into the inertial sensing data mapping model to obtain the motion vector T2. In some embodiments, the uniform value range is a value range preset when the inertial sensing data mapping model is trained.
[0098] In some embodiments, the movement vector T2 includes a movement distance and a rotation angle. The movement vector T2 includes a six-degree-of-freedom (6DOF) motion trajectory of the spatial working point P3 relative to the spatial working point P2.
[0099] In step S380 , a second spatial operation point in the mixed reality environment coordinate system is calculated according to the first spatial operation point in the mixed reality environment coordinate system and the second motion vector.
[0100] In some embodiments, the processor 120 obtains a second spatial operation point P3 in the mixed reality environment coordinate system ES according to the addition of the spatial operation point P2 and the movement vector T2.
[0101] In some embodiments, the processor 120 calculates the spatial operation point P3 according to the following formula (2).
[0102] Formula (2): P3 = P2 + T2.
[0103] In the above formula (2), P2 and P3 are spatial operation points, and T2 is the movement vector.
[0104] In step S385, the 3D scene editing program is executed at the second spatial operation point. In some embodiments, step S385 is executed when the electronic device 100 moves to the position A3 and the processor 120 calculates the spatial operation point P3.
[0105] In some embodiments, after the processor 120 executes the three-dimensional scene editing program based on the spatial operation point P3, the three-dimensional scene editing program is stored corresponding to the spatial operation point P3.
[0106] In some embodiments, after the processor 120 executes the three-dimensional scene editing program based on the spatial operation point P3, the processor 120 stores the spatial operation point P2 and the spatial operation point P3 in the memory 160 when storing the three-dimensional scene editing program.
[0107] The detailed operation of step S385 is similar to that of step S355 and will not be described in detail here.
[0108] In step S390, a 3D scene editing program is subsequently executed.
[0109] In some embodiments, the processor 120 of the electronic device 100 may first store the starting coordinate point P1 and the spatial operation points P2 to P3, and then Figure 4 The operation points, events, virtual objects, special effects actions, etc. selected in the edit list 400 are edited in the 3D scene afterward.
[0110] It should be noted that in the implementation of the present invention, more spatial operation points and their corresponding three-dimensional scene editing programs can be established and stored. In addition, the starting coordinate point P1 and the spatial operation points P2 to P3 mentioned above respectively include the coordinates of the starting coordinate point P1 and the spatial operation points P2 to P3 in the mixed reality environment coordinate system ES.
[0111] See also Figure 5 . Figure 5 is a schematic diagram of a mixed reality production system 500 according to some embodiments of the present invention. The mixed reality production system 500 includes a training device 510, and electronic devices 530A and 530B.
[0112] In some embodiments, the electronic devices 530A and 530B are different test electronic devices. The training device 510 includes an inertial sensing data mapping model. The training device 510 is used to collect the electronic devices 530A and 530B along the same test path (for example, Figure 1 The training device 510 trains the inertial sensing data mapping model based on the test inertial sensing data of the electronic devices 530A and 530B, so that the inertial sensing data mapping model maps the test inertial sensing data of the electronic devices 530A and 530B to the test movement vectors of the same value.
[0113] That is, the electronic device 530A is composed of Figure 1 The inertial sensing data DA generated by moving from position A to position B in the electronic device 530B is Figure 1 The inertial sensing data DB generated by moving from position A to position B in FIG. 1 , after being input into the inertial sensing data mapping model, generates a movement vector T1 .
[0114] In some embodiments, the inertial sensing data mapping model is trained in a long short-term memory (LSTM) manner, but the embodiments of the present invention are not limited thereto.
[0115] In some embodiments, the training device 510 is further used to map different inertial sensing data generated by different inertial sensing elements of the test electronic devices 530A and 530B of different specifications to the same output value range through an inertial sensing data mapping model to generate calibrated inertial sensing data, and then input the calibrated inertial sensing data into the inertial sensing data mapping model to train the inertial sensing data mapping model.
[0116] For example, in one embodiment, the training device 510 pre-processes the different inertial sensing data generated by the different inertial sensing elements of the different test electronic devices 530A and 530B using a normalization technique, converts the different inertial sensing data generated by the different inertial sensing elements of the different test electronic devices 530A and 530B to the same data interval, such as inertial sensing data with values between 1 and -1, and then trains the inertial sensing data mapping model.
[0117] In this way, the movement vectors generated by the different inertial sensing data generated by the different inertial sensing elements of the different test electronic devices 530A and 530B are unified after the inertial sensing data mapping model is passed through, which can reduce the calculation error of the spatial operation point caused by the different inertial sensing elements of the different test electronic devices 530A and 530B.
[0118] In the above embodiment, only the test electronic devices 530A and 530B are used as examples for illustration, and more test electronic devices are also within the scope of the implementation of the present invention.
[0119] See also Figure 6 . Figure 6 is a schematic diagram of a mixed reality production system 600 according to some embodiments of the present invention.
[0120] The mixed reality production system 600 further includes an APP production device 610, an electronic device 100, and 630. The electronic device 100 transmits the starting coordinate point P1, the spatial operation points P2, P3, and the edited three-dimensional scene editing program generated by the mixed reality production method 300 to the APP production device 610. The APP production device 610 generates an application program APP according to the starting coordinate point P1, the spatial operation points P2, P3, and the edited three-dimensional scene editing program transmitted by the electronic device 100. In some embodiments, the electronic device 100 also transmits the inertial sensing data mapping model to the APP production device 610.
[0121] In some embodiments, the electronic device 630 can download the application APP and the inertial sensing data mapping model from the APP production device 610. Figure 1The position A1 in the real space E in the image is obtained according to the environmental feature EC, and the feature anchor point ACH is obtained and the mixed reality environment coordinate system ES is established, and the starting coordinate point P1 of the electronic device 630 is calculated. When the electronic device 630 moves from the position A1 to the position A2, after the inertial sensing element of the electronic device 630 obtains the inertial sensing data, the processor of the electronic device 630 inputs the inertial sensing data into the inertial sensing data mapping model to obtain the corresponding movement vector T1. Then, the processor of the electronic device 630 adds the starting coordinate point P1 and the movement vector T1 to obtain the space operation point P2 corresponding to the position A2. When the processor of the electronic device 630 determines that the electronic device 630 moves to the position A2 and its corresponding space operation point P2, the processor of the electronic device 630 executes a part of the events, special effects actions and virtual object rendering corresponding to the space operation point P2. The above-mentioned events, special effects actions and virtual object rendering corresponding to the space operation point P2 are established by the electronic device 100.
[0122] In some embodiments, the training device 510 and the APP production device 610 may be a server or other electronic devices. In some embodiments, the training device 510 and the APP production device 610 include a processor, a memory, an input device, an output device, and the like.
[0123] In some embodiments, the electronic device 100, 530A, 530B, 630 may be a server, circuit, central processing unit (CPU), microprocessor (MCU) or other device with equivalent functions, such as storage, calculation, data reading, receiving signals or messages, and sending signals or messages.
[0124] In some embodiments, the inertial sensing element 110 may be an element having a function of acquiring inertial sensing data or a similar function, such as a micro electro mechanical system (MEMS), an inertial measurement unit (IMU), etc. In some embodiments, the camera device 152 may be an element having a function of capturing an image or a similar function. In some embodiments, the output device 140 may be an element having a function of outputting a signal or a similar function, such as a screen, a projector, a speaker, a vibrator, etc., and the input device 150 may be an element having a function of inputting a signal or a similar function, such as a touch device, a lens, a microphone, a sensor, etc.
[0125] As can be seen from the above embodiments of the present invention, the embodiments of the present invention provide an electronic device and a mixed reality production method, which, compared with the traditional mixed reality production method, does not need to establish a large number of recognition space anchor points, and does not need to perform high-efficiency operations such as real-time matching and positioning (such as SLAM or feature point recognition). The embodiments of the present invention only need to identify a single starting anchor point (such as Figure 1 The feature anchor point ANC in the image is used to calculate the starting coordinate point and the spatial operation point, and the inertial sensing data mapping model and the inertial sensing data are used to calculate the starting coordinate point and the spatial operation point, which can greatly reduce the amount of calculation compared to the traditional mixed reality production method. In addition, the embodiments of the present invention can edit mixed reality events, special effects actions, and virtual object rendering programs in real time while the electronic device is moving, and can be previewed in real time on the electronic device, which is more intuitive and more convenient when performing mixed reality production.
[0126] In addition, the above examples include sequential exemplary steps, but the steps do not have to be performed in the order shown. Performing the steps in different orders is within the scope of the present disclosure. The steps may be added, replaced, changed in order, and / or omitted as appropriate within the spirit and scope of the embodiments of the present disclosure.
[0127] Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone familiar with this technology can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the definition of the attached claims.
Claims
1. An electronic device capable of moving in a real space, characterized in that: The electronic device comprises: an inertial sensing element, used to obtain a first inertial sensing data when the electronic device moves; and A processor is electrically coupled to the inertial sensing element, and is configured to execute the following steps: Establishing a corresponding mixed reality environment coordinate system according to the real space, and calculating a starting coordinate point of the electronic device in the mixed reality environment coordinate system; Converting the first inertial sensing data into a first motion vector in the mixed reality environment coordinate system according to an inertial sensing data mapping model; Calculating a first spatial operation point in the mixed reality environment coordinate system according to the starting coordinate point in the mixed reality environment coordinate system and the first motion vector; and A three-dimensional scene editing program is executed at the first spatial operation point.
2. The electronic device according to claim 1, wherein: The inertial sensing data mapping model is used to generate the first motion vector according to a normalized inertial sensing data obtained after the first inertial sensing data is normalized.
3. The electronic device according to claim 1, wherein: The training of the inertial sensing data mapping model includes the following: Collecting a plurality of test inertial sensing data after a plurality of different test electronic devices respectively move along the same test path, wherein the plurality of test inertial sensing data are respectively generated by different inertial sensing elements and have different data ranges; as well as The inertial sensing data mapping model is trained according to the plurality of test inertial sensing data, so that the inertial sensing data mapping model maps the plurality of test inertial sensing data to a test motion vector with the same value.
4. The electronic device according to claim 1, wherein: The processor is further configured to perform the following steps: When the electronic device moves to a first position in the real space, the inertial sensing data mapping model converts the first inertial sensing data into the first movement vector in the mixed reality environment coordinate system, and obtains the first spatial operation point in the mixed reality environment coordinate system by adding the starting coordinate point and the first movement vector. The three-dimensional scene editing program is executed at the first spatial operation point at the moment or afterwards, wherein the three-dimensional scene editing program includes one of an event program, a special effect action program and a virtual object rendering program, and the three-dimensional scene editing program is stored corresponding to the first spatial operation point.
5. The electronic device as claimed in claim 4, characterized in that: The inertial sensing element is further used to obtain a second inertial sensing data when the electronic device moves, and the processor is further used to perform the following steps: When the electronic device moves from the first position to a second position in the real space, the inertial sensing data mapping model converts the second inertial sensing data into a second movement vector in the mixed reality environment coordinate system, and obtains a second space operation point in the mixed reality environment coordinate system by adding the first space operation point and the second movement vector. The three-dimensional scene editing program is executed at the second space operation point at the moment or afterwards, and the three-dimensional scene editing program is stored corresponding to the second space operation point.
6. The electronic device as claimed in claim 5, characterized in that: The electronic device further comprises: an output device electrically coupled to the processor, for outputting one of the event program, the special effect action program and the virtual object rendering program on the output device when the processor executes the three-dimensional scene editing program; as well as An input device is electrically coupled to the processor and a memory, and is used for receiving and transmitting a selection signal to the processor after the processor executes the three-dimensional scene editing program.
7. The electronic device according to claim 6, wherein: The event program, the special effect action program and the virtual object rendering program included in the three-dimensional scene editing program are multiple event programs, multiple special effect action programs and multiple virtual object rendering programs; The output device includes a screen device, which displays an edit list. The content of the edit list is selected from a part of the multiple event programs, the special effect action program and the virtual object rendering program.
8. The electronic device as claimed in claim 5, characterized in that: The processor is further configured to perform the following steps: The three-dimensional scene editing program is executed, and when two of the starting coordinate point, the first spatial operation point and the second spatial operation point are stored, the three-dimensional scene editing program is stored together.
9. The electronic device as claimed in claim 1, wherein: The processor further performs the following steps: Extracting a feature anchor point from an image captured by a camera device; and The corresponding mixed reality environment coordinate system is established according to the characteristic anchor point of the real space as the origin of a coordinate system of the mixed reality environment.
10. The electronic device according to claim 1, wherein: The first movement vector at least includes a movement distance and a rotation angle, and the first movement vector includes a six-degree-of-freedom motion trajectory of the first spatial operation point relative to the starting coordinate point.
11. A mixed reality production method, applicable to an electronic device, characterized in that: Include: Establishing a corresponding mixed reality environment coordinate system according to a characteristic anchor point in a real space, and calculating a starting coordinate point of the electronic device in the mixed reality environment coordinate system; Acquire a first inertial sensing data when the electronic device moves; Converting the first inertial sensing data into a first motion vector in the mixed reality environment coordinate system according to an inertial sensing data mapping model; Calculating a first spatial operation point in the mixed reality environment coordinate system according to the starting coordinate point in the mixed reality environment coordinate system and the first motion vector; and A three-dimensional scene editing program is executed at the first spatial operation point.
12. The mixed reality production method according to claim 11, characterized in that: Also includes: The first movement vector is generated based on the inertial sensing data mapping model according to a normalized inertial sensing data obtained after the first inertial sensing data is normalized.
13. The mixed reality production method according to claim 11, wherein: Also includes: Collecting a plurality of test inertial sensing data after a plurality of different test electronic devices respectively move along the same test path, wherein the plurality of test inertial sensing data are respectively generated by different inertial sensing elements and have different data ranges; as well as The inertial sensing data mapping model is trained according to the plurality of test inertial sensing data, so that the inertial sensing data mapping model maps the plurality of test inertial sensing data to a test motion vector with the same value.
14. The mixed reality production method according to claim 11, wherein: Also includes: When the electronic device moves to a first position in the real space, the inertial sensing data mapping model converts the first inertial sensing data into the first movement vector in the mixed reality environment coordinate system; Obtaining a first spatial operation point in the mixed reality environment coordinate system by adding the starting coordinate point and the first movement vector; and The 3D scene editing program is executed at the first spatial operation point at present or afterwards, wherein the 3D scene editing program includes one of an event program, a special effect action program and a virtual object rendering program, and the 3D scene editing program is stored corresponding to the first spatial operation point.
15. The mixed reality production method according to claim 14, wherein: Also includes: When the electronic device moves from the first position to a second position in the real space, the inertial sensing data mapping model converts a second inertial sensing data into a second movement vector in the mixed reality environment coordinate system; Obtaining a second spatial operation point in the mixed reality environment coordinate system by adding the first spatial operation point and the second motion vector; and The three-dimensional scene editing program is executed at the second spatial operation point at present or later, and the three-dimensional scene editing program is stored corresponding to the second spatial operation point.
16. The mixed reality production method according to claim 15, characterized in that: Also includes: When executing the three-dimensional scene editing program, outputting one of the event program, the special effect action program and the virtual object rendering program on an output device; as well as After executing the three-dimensional scene editing program, an input device receives and transmits a selection signal.
17. The mixed reality production method according to claim 16, wherein: The event program, the special effect action program and the virtual object rendering program included in the three-dimensional scene editing program are multiple event programs, multiple special effect action programs, and multiple virtual object rendering programs, and the mixed reality production method further includes: A screen device of the output device displays an edit list, and the content of the edit list is selected from a part of the plurality of event programs, the special effect action program and the virtual object rendering program.
18. The mixed reality production method according to claim 15, wherein: Also includes: The three-dimensional scene editing program is executed, and when two of the starting coordinate point, the first spatial operation point and the second spatial operation point are stored, the three-dimensional scene editing program is stored together.
19. The mixed reality production method according to claim 11, wherein: Also includes: Extracting a feature anchor point from an image captured by a camera device; and The corresponding mixed reality environment coordinate system is established according to the characteristic anchor point of the real space as the origin of a coordinate system of the mixed reality environment.
20. The mixed reality production method of claim 11, wherein: The first movement vector includes a movement distance and a rotation angle, and the first movement vector includes a six-degree-of-freedom motion trajectory of the first spatial operation point relative to the starting coordinate point.