Third perspective recording method, system, device and storage medium

Abstract

The application provides a third perspective recording method, system, device and storage medium. The third perspective recording method comprises: in a third perspective recording scene, determining a pose of a terminal device in a coordinate system of an XR device; sending a first parameter to the XR device; the first parameter comprises a camera parameter of the terminal device and the pose of the terminal device in the coordinate system of the XR device; receiving a first virtual image from the XR device; wherein the XR device constructs a virtual camera corresponding to the terminal device in a corresponding virtual scene based on the first parameter; the first virtual image is a virtual image generated by rendering the virtual scene with the virtual camera as an observation perspective; synthesizing a real image photographed by the terminal device and the first virtual image to obtain a synthesized image; and displaying the synthesized image on a display interface of the terminal device, so that the terminal device can record the virtual scene from different perspectives.

Description

Technical Field

[0001] This application belongs to the XR field, and in particular relates to third-person perspective recording methods, systems, devices and storage media. Background Technology

[0002] Extended Reality (XR) devices include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) devices, which create environments that combine real and virtual scenes through hardware. XR devices offer high privacy and exclusivity; generally, only users wearing the XR device can see its content, while those not wearing it typically cannot. Currently, third-person recording can be used to record XR content and display it on a PC or mobile phone, allowing users without XR devices to experience a sense of participation in the virtual scene.

[0003] Existing third-person perspective recording solutions generally only support recording from a fixed perspective on terminal devices (such as mobile phones), which affects the user experience. Summary of the Invention

[0004] In view of this, embodiments of this application provide a third-person perspective recording method, system, device, and storage medium to solve the problem that the prior art only supports fixed-perspective recording of terminal devices.

[0005] The first aspect of this application provides a third-view recording method, applied to a terminal device, including:

[0006] In a third-person perspective recording scenario, determine the pose of the terminal device in the coordinate system of the XR device; Send a first parameter to the XR device; the first parameter includes the camera parameters of the terminal device and the pose of the terminal device in the coordinate system of the XR device; The XR device receives a first virtual image from the XR device; wherein the XR device constructs a virtual camera corresponding to the terminal device in a corresponding virtual scene based on the first parameter; the first virtual image is a virtual image generated by rendering the virtual scene with the virtual camera as the viewing angle; The real image captured by the terminal device is combined with the first virtual image to obtain a composite image; The composite image is displayed on the display interface of the terminal device.

[0007] In one embodiment, determining the pose of the terminal device in the coordinate system of the XR device includes: In response to a calibration command, at least one first calibration image is acquired, the first calibration image including at least the target action performed by the user; Calculate a first transformation matrix between the coordinate system of the target action and the coordinate system of the terminal device based on the at least one first calibration image; A second transformation matrix is ​​obtained between the coordinate system of the target action from the XR device and the coordinate system of the XR device. The second transformation matrix is ​​determined based on a second calibration image captured by the XR device. The second calibration image includes at least the target action captured by the XR device. The pose of the terminal device in the coordinate system of the XR device is determined based on the first transformation matrix and the second transformation matrix.

[0008] In one embodiment, any one of the first calibration images has time information, and the step of calculating the first transformation matrix between the coordinate system of the target action and the coordinate system of the terminal device based on the at least one first calibration image includes: Based on the time information corresponding to each first calibration image, a target calibration image is determined from the at least one first calibration image, wherein the time difference between the time information of the target calibration image and the time information of the second calibration image is within a preset range; Calculate the first transformation matrix between the coordinate system of the target action and the coordinate system of the terminal device based on the target calibration image.

[0009] In one embodiment, the target action is a target gesture; calculating a first transformation matrix between the coordinate system of the target action and the coordinate system of the terminal device based on the target calibration image includes: Based on the target calibration image, a first transformation matrix is ​​determined between the coordinate system of a preset number of joints on the target gesture and the coordinate system of the terminal device.

[0010] In one embodiment, the method further includes: Acquire the pose change data of the terminal device; The pose change data is sent to the XR device, and the pose change data is used to update the pose of the virtual camera in the coordinate system of the XR device.

[0011] In one embodiment, after the synthesized image is displayed on the display interface of the terminal device, the method further includes: In response to a first operation event triggered by the user on the display interface, the first operation event is converted into a second operation event in the coordinate system of the terminal device according to the camera parameters of the terminal device; The second operation event is sent to the XR device, which triggers the XR device to generate interactive objects in the virtual scene and renders the virtual scene according to the viewing angle of the virtual camera to generate a second virtual image.

[0012] In one embodiment, before synthesizing the real image captured by the terminal device with the first virtual image to obtain the synthesized image, the method further includes: The terminal device caches multiple frames of the real images it has collected, and associates corresponding time information with each frame of the real images. Based on the time information carried by the first virtual image, the real images whose time information is within a preset time difference range with the first virtual image are determined from multiple frames of the real images.

[0013] In one embodiment, before determining the pose of the terminal device in the coordinate system of the XR device, the method further includes: In response to a third-party recording instruction, the time of the terminal device and the XR device is synchronized according to the time information of the terminal device and the time information of the XR device.

[0014] A second aspect of this application provides a third-view recording method applied to an XR device, comprising: In a third-person perspective recording scenario, the first parameter of any one of one or more terminal devices is obtained. The first parameter includes at least the pose of the terminal device in the coordinate system of the XR device and the camera parameters of the terminal device. Based on the first parameter of any of the terminal devices, a virtual camera corresponding to any of the terminal devices is constructed in the corresponding virtual scene; The virtual scene is rendered using the virtual camera as the viewing angle to generate a first virtual image; The first virtual image is sent to any of the terminal devices.

[0015] In one embodiment, before obtaining the first parameter of any one of the one or more terminal devices, the method further includes: Detect coordinate calibration request; In response to the coordinate calibration request, a guidance prompt is output, which guides the user to perform the target action. Calculate a second transformation matrix between the coordinate system of the target action and the coordinate system of the XR device based on the second calibration image; The second transformation matrix is ​​sent to any of the terminal devices, the second transformation matrix being used to determine the pose of the terminal device in the coordinate system of the XR device.

[0016] In one embodiment, the method further includes: Acquire the pose change data of the terminal device; The virtual camera's viewing angle in the virtual scene is updated based on the pose change data; The virtual scene is rendered based on the updated viewing angle of the virtual camera in the virtual scene, generating an updated virtual image; The updated virtual image is sent to the terminal device.

[0017] In one embodiment, the method further includes: Obtain a second operation event from any of the terminal devices in the coordinate system of the terminal device; Based on the pose of the terminal device in the coordinate system of the XR device and the second operation event in the coordinate system of the terminal device, the interactive objects in the virtual scene are generated; The virtual scene is rendered based on the viewing angle of the virtual camera to generate a second virtual image; The second virtual image is sent to the terminal device.

[0018] In one embodiment, before acquiring the first parameter of any one of the one or more terminal devices in a third-person perspective recording scenario, the method further includes: The operation interface is displayed, and the operation interface includes a third-party recording control; In response to a user's triggering operation on the third-party recording control, the third-party recording service is started to establish a communication connection with at least one terminal device.

[0019] In one embodiment, before obtaining the first parameter of any one of the one or more terminal devices, the method further includes: performing time synchronization between the terminal device and the XR device based on the time information of the terminal device and the time information of the XR device.

[0020] A third aspect of this application provides a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the third-view recording method as described in the first aspect above.

[0021] A fourth aspect of this application provides an XR device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the third-view recording method as described in the second aspect above.

[0022] A fifth aspect of this application provides a third-view recording system, comprising: one or more terminal devices and an XR device, wherein the terminal device is the terminal device described in the third aspect above, and the XR device is the XR device described in the fourth aspect above.

[0023] A sixth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the third-view recording method as described in the first or second aspect above.

[0024] A seventh aspect of this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the third-view recording method as described in the first or second aspect above.

[0025] A seventh aspect of this application provides a chip including a processor and a memory, wherein the processor executes a computer program or instructions stored in the memory to implement the third-view recording method as described in the first or second aspect above.

[0026] The beneficial effects of this application embodiment compared with the prior art are as follows: In a third-person perspective recording scenario, the terminal device determines its pose in the coordinate system of the XR device and sends its first parameters to the XR device. These first parameters include the camera parameters of the terminal device and its pose in the XR device's coordinate system. This allows the XR device to construct a virtual camera corresponding to the terminal device within the corresponding virtual scene based on the first parameters, and to render the virtual scene using the virtual camera as the viewing perspective, generating a first virtual image. The terminal device receives the first virtual image from the XR device, synthesizes the real image captured by the terminal device with the first virtual image, and obtains a composite image, which is then displayed on the terminal device's display interface. Since the virtual camera corresponds to the position of the terminal device, the pose of the virtual camera in the virtual scene can be updated in real time according to the terminal device's pose, thereby updating the virtual camera's viewing perspective in the virtual scene. This allows for obtaining virtual images from different viewing perspectives, enabling the terminal device to record virtual scenes from different viewpoints. By combining the first virtual image with a real image captured by the terminal device, a combination of virtual content and real scene can be presented intuitively on the terminal device, enhancing the sense of participation in the virtual scene for users who are not wearing XR devices. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0028] Figure 1 This is a schematic diagram of a third-view recording system provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the implementation flow of a third-view recording method executed on a terminal device according to an embodiment of this application; Figure 3 This is a scene diagram of a terminal device provided in an embodiment of this application that synthesizes and displays a first virtual image and a real image; Figure 4 This is a schematic diagram illustrating the implementation flow of a third-view recording method performed on an XR device according to an embodiment of this application; Figure 5 This is an interactive flowchart of the third-view recording method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the XR device provided in the embodiments of this application. Detailed Implementation

[0029] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0030] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0031] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0032] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0033] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0034] Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0035] The third-person perspective recording method provided in this application is applied to a third-person perspective recording system, which includes an XR device and one or more terminal devices. The one or more terminal devices are communicatively connected to the XR device, for example, via Bluetooth or Wi-Fi. The XR device can be an AR device, VR device, or MR device. The terminal devices can be electronic devices with shooting capabilities, such as mobile phones, tablets, wearable devices, or personal computers.

[0036] In this embodiment of the application, the XR device can create a virtual camera corresponding to each terminal device in the virtual scene of the XR device based on the camera parameters of each of the one or more terminal devices.

[0037] For example, such as Figure 1 As shown, the XR device is AR glasses 11, and the terminal device is a mobile phone 12. The AR glasses 11 display a virtual scene, which the user can see while wearing them. When the mobile phone 12 is connected to the AR glasses 11, the mobile phone 12 receives the virtual image rendered from the AR glasses 11 and combines it with a real image captured by the mobile phone 12 to obtain a composite image. This composite image is then displayed on the mobile phone 12's screen, allowing the mobile phone 12 to be used as a third-person perspective for recording the virtual scene.

[0038] The third-person perspective recording method provided in this application will be illustrated below with reference to the aforementioned third-person perspective recording system.

[0039] Please see the appendix Figure 2 The third-view recording method provided in one embodiment of this application is applied to a terminal device and includes steps S201 to S205.

[0040] S201: In a third-person perspective recording scenario, determine the pose of the terminal device in the coordinate system of the XR device.

[0041] In one embodiment, a user enables a third-party recording service via an XR device. When the terminal device detects that the XR device has enabled the third-party recording service, it can determine that it has entered a third-person perspective recording scene. For example, after the XR device enables the third-party recording service, it periodically sends broadcast packets containing the XR device's identifier and information about enabling the third-party recording service. When the terminal device listens to the broadcast packets, it determines that it has entered a third-person perspective recording scene.

[0042] In another embodiment, the terminal device may have an application corresponding to the XR device installed. If the terminal device determines that the user clicks the third-view recording control in the application interface corresponding to the application, it determines that it has entered the third-view recording scene.

[0043] In one possible embodiment of this application, the pose of the terminal device in the coordinate system of the XR device includes the coordinates and rotation angle of the terminal device in the coordinate system of the XR device.

[0044] In one possible embodiment of this application, the pose of the terminal device in the coordinate system of the XR device can be input by the user on the terminal device, for example, set by the user in the application interface corresponding to the application. Alternatively, the pose of the terminal device in the coordinate system of the XR device can be calculated by the XR device or other third-party devices and then provided to the terminal device; this embodiment of the application does not limit this approach.

[0045] In another possible embodiment of this application, the pose of the terminal device in the coordinate system of the XR device can be calculated by the terminal device itself.

[0046] The following describes how the terminal device calculates its pose in the coordinate system of the XR device.

[0047] In one embodiment, upon detecting a calibration command input by a user, the terminal device, in response to the calibration command, acquires at least one first calibration image, the first calibration image including at least the target action performed by the user. Based on the at least one first calibration image, a first transformation matrix is ​​calculated between the coordinate system of the target action and the coordinate system of the terminal device. The terminal device then obtains a second transformation matrix between the coordinate system of the target action and the coordinate system of the XR device, determined based on a second calibration image captured by the XR device, the second calibration image including at least the target action captured by the XR device. Based on the first and second transformation matrices, the terminal device determines its pose in the coordinate system of the XR device.

[0048] It is understood that the user in the first calibration image in this application embodiment can refer to a user wearing an XR device.

[0049] The target action can be a specified hand gesture, eye movement, facial movement, or head movement, etc. For example, such as... Figure 1 As shown, the target action is a hand gesture of clasping hands together or spreading hands. When the user performs the target action, the terminal device acquires an image to obtain at least one first calibration image, which includes at least the target action performed by the user.

[0050] In one possible embodiment of this application, the method provided in this application may further include: the terminal device sending a calibration request to the XR device in response to a calibration command; and the XR device guiding the user to perform a target action in response to the calibration request.

[0051] The terminal device can begin image acquisition either after detecting a user's calibration input command, or upon detecting a command to begin image acquisition sent by the XR device. For example, the XR device may instruct the terminal device to acquire images when it detects that the user's action matches the target action.

[0052] After calculating a first transformation matrix based on at least one first calibration image, the terminal device receives a second transformation matrix sent by the XR device and determines the terminal device's pose in the XR device's coordinate system based on the first and second transformation matrices. Completing calibration through a target action simplifies the calibration process, is not limited to any particular calibration scenario, and is convenient for users.

[0053] In one embodiment, each of the at least one first calibration image acquired by the terminal device includes time information (also referred to as a timestamp). While receiving the second transformation matrix, the terminal device also receives the time information of the second calibration image. Based on the time information of the second calibration image sent by the XR device, the terminal device determines a first calibration image from the at least one first calibration image whose time difference with the time information of the second calibration image is within a preset range, and uses this first calibration image as the target calibration image. For example, the first calibration image whose timestamp is closest to the timestamp of the second calibration image is selected from the at least one first calibration image; this is the target calibration image. This ensures that the target calibration image and the second calibration image capture the same target action, avoiding the problem of low calculation accuracy caused by inconsistent actions. After determining the target calibration image, the terminal device calculates the first transformation matrix based on the target calibration image, and determines the pose of the terminal device in the XR device coordinate system based on the first transformation matrix and the second transformation matrix.

[0054] In another embodiment, the terminal device can calculate the corresponding first transformation matrix based on each first calibration image, add time information to each first transformation matrix based on the time information of each first calibration image, and after obtaining the time information of the second calibration image, select the first transformation matrix that is closest to the time information of the second calibration image based on the time information corresponding to the multiple first transformation matrices, and determine the pose of the terminal device in the XR device coordinate system based on the first transformation matrix and the second transformation matrix.

[0055] In one embodiment, the target action is a target gesture. After obtaining the target calibration image, the terminal device establishes a world coordinate system based on the target gesture and obtains the coordinates of a preset number of joints (at least three non-collinear joints) on the target gesture in the world coordinate system. For example, if there are 42 joints on the target gesture, one of the joints or the center of the target gesture is taken as the origin of the world coordinate system, and the coordinates of the 42 joints on the target gesture in the world coordinate system are obtained.

[0056] Next, the terminal device determines the coordinates of a preset number of key points in its coordinate system based on the target calibration image. Then, based on the coordinates of these key points in the world coordinate system and their respective coordinates in the terminal device's coordinate system, it obtains a first transformation matrix between the world coordinate system (i.e., the coordinate system containing the preset number of key points) and the terminal device's coordinate system. Finally, the terminal device determines a transformation matrix between its coordinate system and the XR device's coordinate system based on the first transformation matrix and a second transformation matrix obtained from the XR device. Specifically, the first transformation matrix is... The second transformation matrix is The transformation matrix between the coordinate system of the terminal device and the coordinate system of the XR device is determined as follows: × The transformation matrix between the coordinate system of the terminal device and the coordinate system of the XR device represents the initial pose of the terminal device in the coordinate system of the XR device. Calibration via gestures captures subtle gesture features through gesture joints, improving the accuracy of coordinate transformation compared to methods that calibrate based on a single feature point.

[0057] By calculating the pose of the terminal device in the coordinate system of the XR device, the computational consumption of the XR device can be minimized, avoiding virtual scene lag and screen delay caused by excessive computational resources, and ensuring that the immersive experience of the XR device wearer is not affected.

[0058] S202: Send the first parameter to the XR device.

[0059] As an example, the first parameter includes the camera parameters of the terminal device and the pose of the terminal device in the coordinate system of the XR device. The camera parameters of the terminal device include the field of view of the camera, the horizontal pixel size of the captured image, and the vertical pixel size of the captured image.

[0060] In one possible embodiment of this application, the terminal device may actively send the first parameter to the XR device, or it may send the first parameter to the XR device at the request of the XR device. This embodiment of the application does not limit this.

[0061] The XR device is used to construct a virtual camera corresponding to the terminal device in the virtual scene based on the first parameter. The coordinate system of the virtual scene is the coordinate system of the XR device, the camera parameters of the virtual camera are the same as those of the terminal device, and the pose of the virtual camera in the coordinate system of the XR device is the same as that of the terminal device in the coordinate system of the XR device.

[0062] S203: Receive the first virtual image from the XR device.

[0063] Specifically, the XR device uses the virtual camera as the viewing angle to render the virtual scene and generate a first virtual image, and the terminal device receives the first virtual image sent by the XR device.

[0064] Optionally, the first virtual image also carries time information, which indicates the generation time of the first virtual image and is used for subsequent synthesis with a real image at the corresponding time.

[0065] S204: Combine the real image captured by the terminal device with the first virtual image to obtain a composite image.

[0066] Specifically, the terminal device takes a picture of the scene in which it is located, obtaining at least one real image. Upon receiving the first virtual image, the real image and the first virtual image are combined to obtain a composite image. The first virtual image is not displayed as an independent frame alongside the real image in a spliced, split-screen, or picture-in-picture manner. Instead, based on the same viewing angle as the terminal device, virtual content from the first virtual image is superimposed on the real image according to spatial correspondence within the same display screen.

[0067] Specifically, the first virtual image corresponds to an image rendered from a virtual scene using a virtual camera as the viewing perspective. The pose of the virtual camera corresponds to the pose of the terminal device in the XR device coordinate system. Therefore, the virtual content in the first virtual image is consistent with the real scene captured by the terminal device in terms of spatial position and viewing perspective.

[0068] In this way, the terminal device can simultaneously present real scenes and virtual content on a single screen, achieving spatial overlay display of virtual content and real scenes, thus distinguishing it from existing technical solutions that splice virtual and real images as multiple independent screens.

[0069] In one embodiment, before receiving the first virtual image, the terminal device caches multiple frames of real images captured by the terminal device and associates corresponding time information with each frame of the real images. After receiving the first virtual image, based on the time information of the first virtual image, a real image whose time information is within a preset time difference range from the multiple frames of real images is determined. For example, the terminal device records the scene in real time and simultaneously caches 10 real images and the timestamp corresponding to each real image. When the first virtual image and its corresponding timestamp are received, the real image whose timestamp is closest to the first virtual image is determined from the 10 real images, and the real image and the first virtual image are synthesized, thereby making the time information of the virtual image and the real image consistent and improving the real-time recording effect of the terminal device.

[0070] S205: Display the composite image on the display interface of the terminal device.

[0071] Specifically, for each frame of the first virtual image received from the XR device, the terminal device synthesizes the real image and the first virtual image to obtain the corresponding synthesized image, and displays each frame of the synthesized image sequentially on the display interface, thereby realizing the recording of the virtual scene.

[0072] In this embodiment, the first virtual image is not the display screen seen by the XR device (AR glasses) wearer from a first-person perspective, but rather an image obtained by constructing a virtual camera in the virtual world corresponding to the position and orientation of the terminal device in the XR device coordinate system, and rendering the virtual scene using the virtual camera as the viewing perspective.

[0073] Therefore, the first virtual image corresponds to the virtual scene content seen when the terminal device is located at the corresponding position in the virtual world. Its viewing angle, field of view, and visible virtual objects are different from the display screen seen by the AR glasses wearer through the XR device in the first perspective.

[0074] For example, such as Figure 3As shown, the viewing angle of the virtual camera corresponding to the terminal device in the virtual scene is different from that of the XR device in the virtual scene. The terminal device synthesizes the real image 31 and the first virtual image 32 obtained by the virtual camera rendering the virtual scene, and displays it on the display interface. The display screen 33 seen by the XR device wearer in the first perspective is different from the first virtual image 32 displayed on the display interface of the terminal device.

[0075] Through the above method, the embodiments of this application realize the generation of a third-view virtual image independent of the first-view perspective of the XR device wearer, enabling terminal device users who are not wearing XR devices to observe the virtual scene from a position different from the wearer's perspective.

[0076] In one embodiment, the method provided in this application may further include: after the terminal device sends its pose in the coordinate system of the XR device to the XR device, the terminal device acquires pose change data of the terminal device. This pose change data reflects the change in the pose of the terminal device at the current moment relative to the pose at the previous moment. The terminal device sends the pose change data to the XR device, and the pose change data is used to update the pose of the virtual camera in the coordinate system of the XR device.

[0077] For example, the terminal device has a gyroscope or a six-axis sensor. The terminal device uses the gyroscope or six-axis sensor to detect its pose in real time, obtaining pose change data from the previous moment, and sends this pose change data to the XR device. The XR device, based on the terminal device's pose in its coordinate system at the previous moment and the pose change data, obtains the current pose of the terminal device in its coordinate system, and updates the pose of the virtual camera in its coordinate system. When the virtual camera's pose changes in its coordinate system, the virtual camera's shooting angle changes. Correspondingly, the terminal device receives the updated shooting angle from the XR device and renders the virtual image of the virtual scene. Therefore, the terminal device can record the virtual scene while moving.

[0078] In one embodiment, when the XR device enables a third-party recording service, the terminal device searches for the address of the XR device (such as an Internet Protocol address (IP address) or a Media Access Control address (MAC address)), or searches based on the address of the XR device entered by the user. When the XR device is found, a communication connection is established with the XR device.

[0079] In another embodiment, when the XR device enables a third-party recording service, the terminal device can send a connection establishment request message to the XR device. The connection establishment request message may carry the identification information of the terminal device and requests to establish a communication connection with the XR device.

[0080] In one possible embodiment of this application, the method provided in this application embodiment may further include: When a communication connection is established between the terminal device and the XR device, a time synchronization operation is performed between the terminal device and the XR device.

[0081] For example, the specific method for performing time synchronization between a terminal device and an XR device is as follows: In response to a third-party recording instruction, the terminal device sends its own time information to the XR device. For instance, after establishing a communication connection with the XR device, the terminal device outputs a prompt asking whether to start third-party recording. Upon detecting confirmation from the user, it confirms that third-party recording has begun and sends its own time information to the XR device. Subsequently, the terminal device receives the XR device's time information and synchronizes its time with the XR device's time information. For example, if the time information is a timestamp, the terminal device adjusts its own time information or instructs the XR device to adjust its time information based on the difference between the timestamps of the terminal device and the XR device, thus synchronizing their times and improving the real-time performance of the recording.

[0082] In one embodiment, when the terminal device detects that a user triggers a first operation event on the display interface, it converts the first operation event into a second operation event in the terminal device's coordinate system based on the terminal device's camera parameters. The terminal device then sends the second operation event in its coordinate system to the XR device, enabling the XR device to generate interactive objects in the virtual scene based on the second operation event. This allows the terminal device to participate in the virtual scene. For example, the virtual scene could be a game scene, and the interactive object corresponding to the virtual camera could be a water gun. The water gun is displayed at the virtual camera's position within the virtual scene. When the user clicks on the terminal device's display interface, the XR device generates water projectiles fired from the water gun in the virtual scene, allowing interaction with objects in the virtual scene and enhancing the fun of third-party recordings.

[0083] After generating interactive objects in the virtual scene, the XR device renders the virtual scene according to the viewing angle of the virtual camera, generating a second virtual image, which is then sent to the terminal device. The terminal device composites the second virtual image with a real image at a corresponding time to obtain a composite image, which is then displayed on the display interface.

[0084] In one embodiment, after obtaining the synthesized image, the terminal device can save each frame of the synthesized image and, according to the time information of each frame, obtain a recorded video synthesized from the virtual image and the real image. After obtaining the recorded video, the terminal device can also send the recorded video to other interactive devices, thereby realizing the sharing of the recorded video.

[0085] In one embodiment, the terminal device exits the third-view recording scene when it detects a user's command to exit the view recording scene triggered on the terminal device or a command to exit the view recording scene from the XR device.

[0086] In the above embodiments, the terminal device determines its pose in the coordinate system of the XR device, sends its pose and camera parameters to the XR device, and enables the XR device to create a virtual camera corresponding to the terminal device in the virtual scene. The XR device then receives a first virtual image rendered from the virtual camera's viewpoint, combines this first virtual image with a real image, and displays the composite image on the display interface. Therefore, as the terminal device moves, the virtual camera's viewpoint in the virtual scene moves accordingly, allowing the terminal device to record from a third-person perspective without being limited to a fixed viewpoint, thus increasing the enjoyment of third-person recording.

[0087] Please see the appendix Figure 4 Another embodiment of this application provides a third-view recording method applied to an XR device, including steps S401 to S404.

[0088] S401: In a third-person perspective recording scenario, acquire the first parameter of any one of the one or more terminal devices.

[0089] For example, the first parameter includes at least the pose of the terminal device in the coordinate system of the XR device and the camera parameters of the terminal device.

[0090] In one possible embodiment of this application, the XR device can obtain the first parameters of each terminal device from each terminal device. Alternatively, the XR device can also obtain the first parameters of each terminal device based on user input on the XR device. For example, the XR device may have a display interface with an input prompt window, where the user can input the identification information of each terminal device and the first parameters of each terminal device.

[0091] Of course, in this embodiment, the XR device may also have a communication connection with one of the terminal devices, and the terminal device may act as a relay for communication between other terminal devices and the XR device. The other terminal devices may have a communication connection with the terminal device, and the terminal device may obtain the first parameters of the other terminal devices and the terminal device itself and forward them to the XR device. This embodiment does not limit this.

[0092] Specifically, in response to a user's start or wake-up command, the XR device displays an interface that includes third-party recording controls. When a user triggers an action on the third-party recording controls (e.g., the user clicks on a third-party hybrid recording control), the XR device activates the third-party recording service and enters a third-person perspective recording scene.

[0093] In one possible implementation of this application, after the XR device enables a third-party recording service, the XR device can broadcast its address information so that it can receive communication connection requests from any terminal device. These communication connection requests are used to establish a communication connection with the XR device. When the XR device detects a communication connection request from any terminal device, it establishes a communication connection with the corresponding terminal device.

[0094] In one possible embodiment of this application, the method provided in this application may further include: performing a time synchronization operation between the XR device and the terminal device when a communication connection is established between the XR device and the terminal device.

[0095] As an example, performing time synchronization between an XR device and a terminal device may include: the XR device receiving a time synchronization command from the terminal device, and in response to the time synchronization command, sending its own time information to the terminal device. For example, the time synchronization command carries the terminal device's time information. The XR device can adjust its own time information based on the difference between the terminal device's time information and the XR device's time information, ensuring consistency between the XR device's time information and the terminal device's time information, thereby guaranteeing timing consistency between the terminal device and the XR device.

[0096] In one embodiment, after the terminal device and the XR device synchronize their time, if the XR device detects a coordinate calibration request sent by the terminal device or a calibration request input by the user, it outputs guidance prompts (for example, the guidance prompts can be output on the XR device's display interface or via the XR device's voice playback component). These prompts guide the user to perform a target action. The target action can be a specified gesture, eye movement, facial movement, or head movement. The XR device can play audio indicating the target action or display the target action on the display interface, thereby guiding the user to perform the target action. For example, the target action might be a gesture of clasping hands or spreading hands. The XR device can display a method for performing the target action or an image of the target action on the display interface, thereby guiding the user to perform the target action, ensuring the consistency of the target action, and improving the accuracy of subsequent calibration.

[0097] For example, the guidance prompts can be text prompts, voice prompts, or a combination of text prompts and voice prompts; this application embodiment does not limit this.

[0098] When the user performs the target action, the XR device can continuously acquire images of the target action. When it is determined from the acquired images that the user's action does not conform to the target action, an incorrect action prompt message is input. The process continues until it is determined that the user's action conforms to the target action, at which point a second calibration image containing at least the target action performed by the user is acquired.

[0099] In one embodiment, after detecting a coordinate calibration request, the XR device activates its gesture calibration function and then outputs guidance prompts. The gesture calibration function can remain always on, maintaining its activation after detecting the coordinate calibration request. Alternatively, the gesture calibration function can be off; if it is off upon detecting the coordinate calibration request, the XR device will activate its gesture calibration function.

[0100] In one possible implementation of this application, the XR device can disable the gesture calibration function after gesture calibration is completed.

[0101] In one embodiment, when the XR device receives the first parameter sent by the terminal device, or when it detects that the terminal device has received the second calibration image, it determines that the coordinate calibration is complete and disables the gesture calibration function.

[0102] After the XR device acquires the second calibration image, it determines the coordinates of a preset number of key points in the coordinate system of the XR device based on the second calibration image. Based on the coordinates of the preset number of key points in the world coordinate system and the coordinates of the key points in the coordinate system of the XR device, it obtains a second transformation matrix between the world coordinate system (i.e., the coordinate system in which the preset number of key points are located) and the coordinate system of the XR device. The second transformation matrix is ​​then sent to the corresponding terminal device. The terminal device uses the first transformation matrix and the second transformation matrix calculated by the terminal device to determine the pose of the terminal device in the coordinate system of the XR device.

[0103] In another embodiment, the XR device obtains the calibration method selected by the user, and cooperates with the terminal device to complete the coordinate calibration according to the calibration process corresponding to the calibration method, so that the terminal device can calculate the pose of the terminal device in the XR device coordinate system.

[0104] For example, if the calibration method is gesture calibration, then follow the above method to complete the gesture calibration process.

[0105] If the calibration method is QR code calibration, the XR device downloads and displays the QR code from the cloud. The terminal device can photograph the QR code, determine the transformation between its own coordinate system and the QR code's coordinate system based on the captured image, and simultaneously determine the transformation between the XR device's coordinate system and the QR code's coordinate system based on the QR code's position on the XR device. Based on these transformations, the terminal device's pose within the XR device's coordinate system is determined. Alternatively, the XR device instructs the terminal device to display the QR code, photographs it, and sends the resulting image to the terminal device. The terminal device then determines its pose within the XR device's coordinate system based on both the displayed and transmitted QR code images.

[0106] If the calibration method is marker calibration, when the XR device detects a calibration request from the terminal device, it instructs the terminal device to photograph a specified marker on the XR device and sends the marker's position information on the XR device to the terminal device. The terminal device determines the transformation relationship between the XR device's coordinate system and the marker's coordinate system based on the marker's position information. It then determines the transformation relationship between the terminal device's coordinate system and the marker's coordinate system based on the photographed image of the marker. Finally, based on the transformation relationships between the terminal device's coordinate system and the QR code's coordinate system, and between the XR device's coordinate system and the QR code's coordinate system, it determines the terminal device's pose within the XR device's coordinate system.

[0107] After each terminal device determines its pose in the XR device coordinate system, the XR device receives the first parameter sent by at least one terminal device.

[0108] In another embodiment, the XR device may also obtain the first parameters of each terminal device from the server, or obtain the first parameters input by the user.

[0109] S402: Based on the first parameter of any of the terminal devices, construct a virtual camera corresponding to any of the terminal devices in the corresponding virtual scene.

[0110] Specifically, the camera parameters of the virtual camera are the same as the camera parameters of the terminal device, and the pose of the virtual camera in the coordinate system of the XR device is the same as the pose of the terminal device in the coordinate system of the XR device. The pose of the virtual camera in the coordinate system of the XR device is the same as the pose of the virtual camera in the virtual scene.

[0111] After an XR device creates a virtual camera in a virtual scene, it can display the virtual camera or not, or it can display the virtual camera or the corresponding interactive object when the terminal device detects user operation events.

[0112] S403: Render the virtual scene using the virtual camera as the viewing angle to generate a first virtual image.

[0113] Specifically, the XR device uses a virtual camera as the viewing perspective. Based on the virtual camera's position in the virtual scene and its camera parameters, it determines the content that the virtual camera can capture in the virtual scene, renders the content in the virtual scene, and obtains a first virtual image. The effect of the first virtual image is that of an image obtained by taking a picture of the virtual scene using the virtual camera as the viewing perspective.

[0114] XR devices can generate a first virtual image that matches the resolution of the terminal device, thereby improving the effect of subsequent image compositing.

[0115] S404: Send the first virtual image to any of the terminal devices.

[0116] Specifically, after generating a first virtual image, the XR device sends the first virtual image to the corresponding terminal device. The terminal device then captures a picture of the scene in which it is located, obtaining a real image. Upon receiving the real image, the real image and the first virtual image are combined to obtain a composite image.

[0117] For example, if the XR device creates virtual cameras corresponding to terminal device a and terminal device b in a virtual scene, the XR device can use the virtual camera corresponding to terminal device a as the viewing angle to capture virtual image 1 of the virtual scene, and then send virtual image 1 to terminal device a. The XR device can use the virtual camera corresponding to terminal device b as the viewing angle to capture virtual image 2 of the virtual scene, and then send virtual image 2 to terminal device b.

[0118] In one embodiment, the XR device acquires pose change data of the terminal device, and obtains the current pose of the terminal device in the XR device coordinate system based on the previous pose and pose change data of the terminal device in the XR device coordinate system. For example, the pose of the terminal device in the XR device coordinate system at the previous moment was... × The pose change of the terminal device at the current moment relative to the previous moment is: Then the pose of the terminal device in the XR device coordinate system at the current moment is ( × ) After obtaining the current pose of the terminal device in the XR device coordinate system, the XR device updates the pose of the virtual camera in the same coordinate system, thereby updating the virtual camera's viewing angle in the virtual scene. Based on the updated viewing angle, the virtual scene is rendered to generate an updated virtual image. The XR device then sends the updated virtual image to the terminal device, which synthesizes the real image and the updated virtual image, displaying the synthesized image on the display interface. Therefore, the terminal device's display interface can present recorded images with changing shooting angles, enabling the recording of virtual images from any perspective.

[0119] In one embodiment, when the terminal device detects that a user triggers a first operation event on the display interface, it converts the first operation event into a second operation event in the terminal device's coordinate system based on the terminal device's camera parameters, and sends the second operation event in the terminal device's coordinate system to the XR device. The XR device, based on the second operation event in the terminal device's coordinate system and the terminal device's pose in the XR device's coordinate system, converts the second operation event in the terminal device's coordinate system into an operation event in the XR device's coordinate system. This operation event in the XR device's coordinate system is the operation event in the virtual scene. The XR device generates interactive objects in the virtual scene based on these operation events. For example, the XR device can generate a ray at a corresponding position in the virtual scene based on a click or swipe operation on the terminal device. Or, in a game scene, a water gun can be displayed at the location of the virtual camera, and a water cannonball can be generated based on a click operation on the terminal device, allowing the user to interact with objects in the virtual scene through the terminal device, thus enhancing the fun of third-party recording.

[0120] After generating interactive objects in the virtual scene, the XR device renders the virtual scene based on the pose of the virtual camera in the XR device's coordinate system, generating a second virtual image. The second virtual image is then sent to the terminal device, which synthesizes the real image and the second virtual image with the same time information to obtain a composite image.

[0121] In one embodiment, when the XR device detects a user input command to interrupt or terminate the third-view recording service, it ends the third-view recording and closes the third-view recording service.

[0122] In the above embodiments, the XR device generates a virtual camera in the virtual scene based on the pose of the terminal device in the XR device's coordinate system and the camera parameters of the terminal device. The virtual scene is rendered to obtain a first virtual image, and the first virtual image is sent to the terminal device. This allows the position of the virtual camera to be updated according to the position of the terminal device, enabling recording of the virtual scene from any angle and enhancing the fun and sense of participation in recording the virtual scene.

[0123] In one embodiment, the third-view recording method flows as follows: Figure 5 As shown.

[0124] The XR device initiates a third-party recording service based on user actions triggered by the third-party recording control. Upon detecting this, the terminal device establishes a communication connection with the XR device using its IP address. After this connection is established, the XR device sends time information to the terminal device, which then synchronizes their times accordingly.

[0125] After the terminal device and the XR device synchronize their time, the terminal device sends a calibration request to the XR device when it detects a calibration command input by the user. The XR device guides the user to perform the target action and captures a second calibration image. Based on the second calibration image, it determines a second transformation matrix and sends the second transformation matrix to the terminal device. The terminal device determines a first transformation matrix based on the first calibration image captured by the terminal device, which includes the target action. Based on the first and second transformation matrices, it determines the pose of the terminal device in the coordinate system of the XR device, thus completing the coordinate system synchronization.

[0126] After coordinate system synchronization is complete, the terminal device sends the first parameter to the XR device. The first parameter includes the terminal device's pose in the XR device's coordinate system and the terminal device's camera parameters. The XR device then creates a virtual camera corresponding to the terminal device. When the terminal device's pose changes, the terminal device determines the pose change data and sends it to the XR device. The XR device then updates the virtual camera's pose in the XR device's coordinate system based on the pose change data.

[0127] When the terminal device detects an operation event triggered by the user on the display interface, it converts the operation event into an operation event in the terminal device's coordinate system and sends it to the XR device. The XR device, based on the operation event in its own coordinate system and the terminal device's pose within its own coordinate system, determines the operation event in the virtual scene and generates interactive objects within the virtual scene. The XR device renders the virtual scene according to the virtual camera's viewing angle, generating a virtual image, which is then sent to the terminal device. The terminal device composites the virtual image with a real image captured by the terminal device to obtain a composite image, which is then displayed on the display interface. Afterward, the terminal device determines whether to enable local recording. If local recording is enabled, it stores each frame of the composite image. If local recording is not enabled, it returns to the step of determining pose change data.

[0128] When the XR device detects a command to interrupt third-party recording, it closes the third-party recording service and ends third-party recording. If the XR device does not detect a command to interrupt third-party recording, it returns to the step of updating the virtual camera's pose in the XR device's coordinate system. When the terminal device detects that the XR device has interrupted the third-party recording service, it closes the third-party recording service and ends third-party recording. If the terminal device does not detect an interruption of the third-party recording service, it returns to the step of determining pose change data.

[0129] In the above embodiments, coordinate calibration is completed by guiding the user to perform the target action, which is simple to operate, improves calibration efficiency, and is not limited to any particular application scenario. The XR device creates a virtual camera based on the pose of the terminal device in the XR device's coordinate system, renders the virtual scene based on the virtual camera's viewing angle in the virtual scene, and generates a virtual image. It supports multiple terminal devices accessing the XR device, and can update the pose of the virtual camera in the virtual scene according to the pose of the terminal device, allowing the terminal device to record the virtual scene from any angle, improving the flexibility and fun of recording virtual scenes for users.

[0130] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0131] Figure 6 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application.

[0132] like Figure 6 As shown, the terminal device in this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the processor 61 executes the computer program 63, it implements the steps described in the third-view recording method embodiment, for example... Figure 2 Steps S201 to S205 are shown.

[0133] For example, the computer program 63 may be divided into one or more modules / units, which are stored in the memory 62 and executed by the processor 61 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 53 in the terminal device.

[0134] Those skilled in the art will understand that Figure 6 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0135] The processor 61 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0136] The memory 62 can be an internal storage unit of the terminal device, such as a hard drive or RAM. The memory 62 can also be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 62 can include both internal and external storage units. The memory 62 is used to store the computer program and other programs and data required by the terminal device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0137] Figure 7 This is a schematic diagram of the structure of the XR device provided in the embodiments of this application.

[0138] like Figure 7 As shown, the XR device in this embodiment includes: a processor 71, a memory 72, and a computer program 73 stored in the memory 72 and executable on the processor 71. When the processor 71 executes the computer program 73, it implements the steps described in the third-view recording method embodiment above, for example... Figure 4 Steps S401 to S404 are shown.

[0139] For example, the computer program 73 may be divided into one or more modules / units, which are stored in the memory 72 and executed by the processor 71 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 73 in the XR device.

[0140] Those skilled in the art will understand that Figure 7This is merely an example of an XR device and does not constitute a limitation on XR devices. It may include more or fewer components than shown, or combine certain components, or different components. For example, the XR device may also include input / output devices, network access devices, buses, etc.

[0141] The processor 71 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0142] The memory 72 can be an internal storage unit of the XR device, such as the hard drive or memory of the XR device. The memory 72 can also be an external storage device of the XR device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard equipped on the XR device. Furthermore, the memory 72 can include both internal and external storage units of the XR device. The memory 72 is used to store the computer program and other programs and data required by the XR device. The memory 72 can also be used to temporarily store data that has been output or will be output.

[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0144] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0145] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0148] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0149] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0150] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A third-person perspective recording method, applied to a terminal device, characterized in that, include: In a third-person perspective recording scenario, determine the pose of the terminal device in the coordinate system of the XR device; Send the first parameter to the XR device; The first parameter includes the camera parameters of the terminal device and the pose of the terminal device in the coordinate system of the XR device; The XR device receives a first virtual image from the XR device; wherein the XR device constructs a virtual camera corresponding to the terminal device in a corresponding virtual scene based on the first parameter; the first virtual image is a virtual image generated by rendering the virtual scene with the virtual camera as the viewing angle; The real image captured by the terminal device is combined with the first virtual image to obtain a composite image; The composite image is displayed on the display interface of the terminal device.

2. The method according to claim 1, characterized in that, Determining the pose of the terminal device in the coordinate system of the XR device includes: In response to a calibration command, at least one first calibration image is acquired, the first calibration image including at least the target action performed by the user; Calculate a first transformation matrix between the coordinate system of the target action and the coordinate system of the terminal device based on the at least one first calibration image; A second transformation matrix is ​​obtained between the coordinate system of the target action from the XR device and the coordinate system of the XR device. The second transformation matrix is ​​determined based on a second calibration image captured by the XR device. The second calibration image includes at least the target action captured by the XR device. The pose of the terminal device in the coordinate system of the XR device is determined based on the first transformation matrix and the second transformation matrix.

3. The method according to claim 2, characterized in that, Each of the first calibration images has time information. The step of calculating the first transformation matrix between the coordinate system of the target action and the coordinate system of the terminal device based on the at least one first calibration image includes: Based on the time information corresponding to each first calibration image, a target calibration image is determined from the at least one first calibration image, wherein the time difference between the time information of the target calibration image and the time information of the second calibration image is within a preset range; Calculate a first transformation matrix between the coordinate system of the target action and the coordinate system of the terminal device based on the target calibration image.

4. The method according to claim 3, characterized in that, The target action is the target gesture; Calculating a first transformation matrix between the coordinate system of the target action and the coordinate system of the terminal device based on the target calibration image includes: Based on the target calibration image, a first transformation matrix is ​​determined between the coordinate system of a preset number of joints on the target gesture and the coordinate system of the terminal device.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Acquire the pose change data of the terminal device; The pose change data is sent to the XR device, and the pose change data is used to update the pose of the virtual camera in the coordinate system of the XR device.

6. The method according to any one of claims 1 to 4, characterized in that, After the synthesized image is displayed on the display interface of the terminal device, the method further includes: In response to a first operation event triggered by the user on the display interface, the first operation event is converted into a second operation event in the coordinate system of the terminal device according to the camera parameters of the terminal device; The second operation event is sent to the XR device, which triggers the XR device to generate interactive objects in the virtual scene and renders the virtual scene according to the viewing angle of the virtual camera to generate a second virtual image.

7. The method according to any one of claims 1 to 4, characterized in that, Before combining the real image captured by the terminal device with the first virtual image to obtain the combined image, the method further includes: The terminal device caches multiple frames of the real images it has collected, and associates corresponding time information with each frame of the real images. Based on the time information carried by the first virtual image, the real images whose time information is within a preset time difference range with the first virtual image are determined from multiple frames of the real images.

8. The method according to any one of claims 1 to 4, characterized in that, Before determining the pose of the terminal device in the coordinate system of the XR device, the method further includes: In response to a third-party recording instruction, the time of the terminal device and the XR device is synchronized according to the time information of the terminal device and the time information of the XR device.

9. A third-person perspective recording method, applied to XR devices, characterized in that, include: In a third-person perspective recording scenario, the first parameter of any one of one or more terminal devices is obtained. The first parameter includes at least the pose of the terminal device in the coordinate system of the XR device and the camera parameters of the terminal device. Based on the first parameter of any of the terminal devices, a virtual camera corresponding to any of the terminal devices is constructed in the corresponding virtual scene; The virtual scene is rendered using the virtual camera as the viewing angle to generate a first virtual image; The first virtual image is sent to any of the terminal devices.

10. The method according to claim 9, characterized in that, Before obtaining the first parameter of any one of the one or more terminal devices, the method further includes: Detect coordinate calibration request; In response to the coordinate calibration request, a guidance prompt is output, which guides the user to perform the target action. Acquire a second calibration image, the second calibration image including at least the target action performed by the user; Calculate a second transformation matrix between the coordinate system of the target action and the coordinate system of the XR device based on the second calibration image; The second transformation matrix is ​​sent to any of the terminal devices, the second transformation matrix being used to determine the pose of the terminal device in the coordinate system of the XR device.

11. The method according to claim 9 or 10, characterized in that, The method further includes: Acquire the pose change data of the terminal device; The virtual camera's viewing angle in the virtual scene is updated based on the pose change data; The virtual scene is rendered based on the updated viewing angle of the virtual camera in the virtual scene, generating an updated virtual image; The updated virtual image is sent to the terminal device.

12. The method according to claim 9 or 10, characterized in that, The method further includes: Obtain a second operation event from any of the terminal devices in the coordinate system of the terminal device; Based on the pose of the terminal device in the coordinate system of the XR device and the second operation event in the coordinate system of the terminal device, the interactive objects in the virtual scene are generated; The virtual scene is rendered based on the viewing angle of the virtual camera to generate a second virtual image; The second virtual image is sent to the terminal device.

13. The method according to claim 9 or 10, characterized in that, In a third-person perspective recording scenario, before acquiring the first parameter of any one of the one or more terminal devices, the method further includes: The operation interface is displayed, and the operation interface includes a third-party recording control; In response to a user's triggering operation on the third-party recording control, the third-party recording service is started to establish a communication connection with at least one terminal device.

14. The method according to claim 9 or 10, characterized in that, Before obtaining the first parameter of any one of the one or more terminal devices, the method further includes: performing time synchronization between the terminal device and the XR device based on the time information of the terminal device and the time information of the XR device.

15. A terminal device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the method as claimed in any one of claims 1 to 8.

16. An XR device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the method as described in any one of claims 9 to 14.

17. A third-person perspective recording system, characterized in that, include: One or more terminal devices, and an XR device, wherein the terminal device is the terminal device as described in claim 15, and the XR device is the XR device as described in claim 16.

18. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 14.

Images