Method, host and computer readable storage medium for providing visual content

Abstract

Embodiments of the present disclosure provide a method, a host and a computer readable storage medium for providing visual content. The method comprises: providing a reference object at a first location to aim at a first point and to determine a first reference line associated with the first point, wherein the first point is associated with an external camera; providing the reference object at a second location to aim at the first point and to determine a second reference line associated with the first point; determining a camera position of the external camera based on the first reference line and the second reference line; obtaining a specific image captured by the external camera; and generating specific visual content by combining the specific image with a virtual scene based on the camera position.

Description

Technical Field

[0001] This invention generally relates to mechanisms for providing content, and more particularly to methods, hosts, and computer-readable storage media for providing visual content. Background Technology

[0002] Mixed reality capture (MRC) technology can be used to provide a third-person perspective to virtual reality (VR) users. MRC requires an external camera to capture images corresponding to the third-person viewpoint.

[0003] In image rendering processing for VR applications, application images presented by the application camera (which has the same viewpoint as the external camera) can be combined with camera images extracted from the external camera to form a mixed image for (real-time) broadcasting.

[0004] However, if the external camera is not positioned correctly, the perspectives of the applied camera and the external camera may differ. In this case, the quality of the resulting mixed image will be reduced.

[0005] For example, if an external camera captures a user holding a virtual object rendered based on the application's camera viewpoint, then the user should appear to be holding that virtual object in the blended image. If the external camera is not positioned correctly, the virtual object in the blended image may be off-center from the user's hand, potentially leading to an unpleasant visual experience.

[0006] Therefore, obtaining the correct position of the external camera is crucial. Summary of the Invention

[0007] Therefore, the present invention relates to a method, host, and computer-readable storage medium for providing visual content, which can be used to solve the above-mentioned technical problems.

[0008] Embodiments of the present invention provide a method for providing visual content suitable for a host computer. The method includes: providing a reference object at a first location to aim at the first point and determining a first reference line associated with the first point, wherein the first point is associated with an external camera; providing a reference object at a second location to aim at the first point and determining a second reference line associated with the first point; determining a camera position of the external camera based on the first and second reference lines; acquiring a specific image extracted by the external camera; and generating specific visual content by combining the specific image with a virtual scene based on the camera position.

[0009] Embodiments of the present invention provide a host computer including storage circuitry and a processor. The storage circuitry stores program code. A processor is coupled to the storage circuitry and accesses the program code to execute: providing a reference object at a first location to aim at a first point and thereby determine a first reference line associated with the first point, wherein the first point is associated with an external camera; providing a reference object at a second location to aim at the first point and thereby determine a second reference line associated with the first point; determining the camera position of the external camera based on the first and second reference lines; acquiring a specific image extracted by the external camera; and generating specific visual content by combining the specific image with a virtual scene based on the camera position.

[0010] Embodiments of the present invention provide a computer-readable storage medium recording an executable computer program loaded by a host computer to perform the following steps: providing a reference object at a first location to aim at a first point and thereby determine a first reference line associated with the first point, wherein the first point is associated with an external camera; providing a reference object at a second location to aim at the first point and thereby determine a second reference line associated with the first point; determining the camera position of the external camera based on the first and second reference lines; acquiring a specific image extracted by the external camera; and generating specific visual content by combining the specific image with a virtual scene based on the camera position. Attached Figure Description

[0011] The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0012] Figure 1 A schematic diagram of a host computer according to an embodiment of the present invention is shown.

[0013] Figure 2 A schematic diagram illustrating an MRC application scenario according to an embodiment of the present invention is shown.

[0014] Figure 3 A flowchart illustrating a method for providing visual content according to an embodiment of the present invention is shown.

[0015] Figures 4A to 4E A schematic diagram illustrating the determination of camera location according to an embodiment of the present invention is shown.

[0016] Figure 5 A schematic diagram illustrating the determination of camera position based on a first reference line and a second reference line according to an embodiment of the present invention is shown.

[0017] Figure 6 A schematic diagram illustrating the determination of the orientation of an external camera according to an embodiment of the present invention is shown.

[0018] Figure 7 A schematic diagram illustrating an application scenario according to an embodiment of the present invention is provided.

[0019] Explanation of icon numbers

[0020] 100: Host;

[0021] 102: Storage circuit;

[0022] 104: Processor;

[0023] 210: External camera;

[0024] 220: Tracking area;

[0025] 230: Green screen;

[0026] 299: User;

[0027] 400: Visual content;

[0028] 411: Reference Object;

[0029] 510: The first specific line segment;

[0030] 511: First location point;

[0031] 710: Specific image;

[0032] 720: Virtual scene;

[0033] 731, 732, 733: Specific visual content;

[0034] L1: First location;

[0035] L2: Second location;

[0036] P1: First point;

[0037] P2: Second point;

[0038] P3: Third point;

[0039] RL1: First reference line;

[0040] RL2: Second reference line;

[0041] RL3: Third reference line;

[0042] RL4: Fourth reference line;

[0043] RL5: Fifth reference line;

[0044] RL6: Sixth reference line;

[0045] S310, S320, S330, S340, S350: Steps. Detailed Implementation

[0046] Reference will now be made in detail to the presently preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Where possible, the same reference numerals are used in the drawings and embodiments to refer to the same or similar parts.

[0047] See Figure 1 This diagram illustrates a host computer according to an embodiment of the present invention. In various embodiments, the host computer 100 may be any electronic device, such as a smart device and / or a computer device. In some embodiments, the host computer 100 may be a head-mounted display (HMD) of a reality system (e.g., a VR system, an augmented reality (AR) system, a mixed reality (MR) system, etc.). In some embodiments, the host computer 100 may be a (standalone) HMD that provides visual content (e.g., VR content) for a user / wearer to view, but the invention is not limited thereto.

[0048] exist Figure 1 In this system, host 100 includes storage circuitry 102 and processor 104. Storage circuitry 102 is one or a combination of the following: static or mobile random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or any other similar device, and storage circuitry 102 records program code and / or multiple modules executable by processor 104.

[0049] The processor 104 is coupled to the storage circuit 102, and the processor 104 may be, for example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc.

[0050] In some embodiments, host 100 may be configured to perform tracking functions such as inside-out tracking and / or outside-in tracking to track the pose of host 100. In some embodiments, host 100 may be equipped with one or more cameras (e.g., a front-facing camera) for extracting images of the environment at host 100's location. In one embodiment, host 100 may track its pose based on the extracted images by performing, for example, simultaneous localization and mapping (SLAM). In some embodiments, the pose of host 100 may be characterized by one or more components of host 100 in one or more degrees of freedom (DOF), such as the so-called 6DOF of host 100, but the invention is not limited thereto.

[0051] In an embodiment of the present invention, the host 100 can be used in an MRC application scenario. See also Figure 2 It illustrates a schematic diagram of an MRC application scenario according to an embodiment of the present invention. Figure 2 In this embodiment, the host device 100 may be an HMD worn by the user 299. Additionally, an external camera 210 may be mounted outside the tracking area 220 to extract images of the user 299 and / or the host device 100 from a third-person perspective. In this embodiment, the tracking area 229 is an area that allows movement of the host device 100 and / or the user 299. Because the external camera 210 is neither in nor near the tracking area 200, it is difficult for the user 299 to determine the camera position of the external camera 210 using conventional methods, such as physically touching it with a handheld controller connected to the host device 100. Therefore, embodiments of the present invention provide a mechanism for determining the camera position of the external camera 210 even when it is far from the user 299 and / or the host device 100.

[0052] In one embodiment, the green screen 230 may be installed in the background of the user 299. In this case, once the external camera 210 captures an image of the user 299 and / or the host 100, the host 100 may perform some image preprocessing (e.g., background removal) on the image from the external camera 210. In one embodiment, the host 100 may combine the preprocessed image with a virtual scene (e.g., a VR scene) to create visual content for, for example, live streaming, but the invention is not limited thereto.

[0053] In embodiments of the present invention, processor 104 accesses modules and / or program code stored in storage circuit 102 to implement the method for providing visual content provided in the present invention, which will be further discussed below.

[0054] See Figure 3The diagram illustrates a flowchart of a method for providing visual content according to an embodiment of the present invention. This method can be provided by... Figure 1 The host 100 in the middle executes, and will be used in the following text. Figure 1 The components shown in the figure are used to describe Figure 3 Details of each step in the process. Furthermore, to better explain the concept of this invention, Figure 2 and Figures 4A to 4E This will be used as an example, where Figures 4A to 4E A schematic diagram illustrating the determination of camera location according to an embodiment of the present invention is shown.

[0055] In an embodiment of the present invention, Figure 3 Before the method begins, the processor 104 may provide instructions to instruct the user 299 to place the external camera 210 horizontally while facing the center of the tracking area 220, and may instruct the user 299 to stand at a first location L1 within the tracking area 220, but the invention is not limited thereto. In other embodiments, the external camera 210 may be placed in different ways according to the designer's requirements.

[0056] In one instance, the first location L1 may be the center of the tracking area 220. In other embodiments, the first location L1 may be another location within the tracking area 220, depending on the designer's requirements.

[0057] In an embodiment of the invention, the host 100 can operate in a transparency mode while acquiring the camera position of the external camera 210. That is, the host 100 can use, for example, a front-facing camera to extract an image of the environment in front of the user 299 and display the extracted image as visual content to the user 299. In this case, the user 299 can see the environment while wearing the host 100, which means that the user 299 can see the external camera 210 in the visual content provided by the host 100.

[0058] In some embodiments, the host 100 may operate in a see-through mode while acquiring the camera position of the external camera 210. That is, when wearing the host 100, the user can directly see the environment through, for example, the transparent display panel of the host 100, but the invention is not limited thereto.

[0059] exist Figure 4AIn this embodiment, it is assumed that processor 104 displays visual content 400 to user 299 within a specific field of view when operating in a transparency or perspective mode, and user 299 can see external camera 210 within the visual content 400. In this embodiment, the specific field of view corresponds to a specific eye image (e.g., a left-eye image or a right-eye image) provided by host 100. That is, only one eye of user 299 can see visual content 400. In this case, processor 104 may perform several steps to enable user 299 to determine the camera position of external camera 210. This will be discussed in detail below.

[0060] In step S310, the processor 104 provides a reference object 411 at the first location L1 to aim at the first point P1 and thereby determine a first reference line RL1 associated with the first point P1, wherein the first point P1 is associated with the external camera 210.

[0061] exist Figure 4A In response to determining that host 100 is at a first location L1 (e.g., the center of tracking area 220), processor 104 displays reference object 411 at a specific location within a specific field of view. In different embodiments, the specific location for displaying reference object 411 may be the center of the specific field of view or other desired location.

[0062] In an embodiment of the invention, the reference object 411 may be a reference ray emanating forward from the user's eye. In one embodiment, such as Figure 4A As illustrated, since the reference ray can be parallel to the user's line of sight, the reference ray appears as a point to the user. To facilitate the user's viewing of the reference object 411, the processor 104 may generate visual aids (e.g., a crosshair corresponding to the center of the reference object 411) for the user to view, but the invention is not limited thereto.

[0063] In one embodiment, the reference object 411 may be a reference ray emitted from a handheld controller connected to the host 100. In this case, the user can see the reference ray in the visual content 400, but the invention is not limited thereto.

[0064] In one embodiment, in order to determine the camera position of the external camera 210, it may be assumed that the first point P1 under consideration corresponds to the center of the lens of the external camera 210.

[0065] In one embodiment, when host 100 is at the first location L1, processor 104 may instruct user 299 to aim reference object 411 at the first point P1. In embodiments where reference object 411 corresponds to a reference ray emitted from a handheld controller, reference object 411 may move in response to movement of the handheld controller connected to host 100. Thus, user 299 may move the handheld controller around to attempt to align reference object 411 with the first point P1.

[0066] In an embodiment where reference object 411 corresponds to a reference ray emitted from the user's eye, the position of reference object 411 can be fixed within a specific field of view, and visual content 400 can change in response to movement of the user's head. Thus, user 299 can move his / her head to attempt to align reference object 411 with the first point P1, but the invention is not limited thereto.

[0067] Once the user 299 determines that the reference object 411 is aligned with the first point P1 (e.g., the center of the lens of the external camera 210), the user 299 may perform a confirmation operation, such as pressing a specific button on the handheld controller or the host 100, to notify the host 100.

[0068] exist Figure 4B In response to the detection of an acknowledgment operation during the determination of providing a reference object 411 at the first location L1, the processor 104 obtains the current representative position of the representative object and the current position corresponding to the reference object 411.

[0069] Next, the processor 104 determines a first reference line RL1, wherein the first reference line RL1 passes through the current representative position and the current position.

[0070] In one embodiment, if the reference object 411 moves in response to the movement of the host 100, then the representative object may be the host 100. In this case, the processor 104 may generate a line passing through the current representative position of the host 100 and corresponding to the current position of the reference object 411 as a first reference line RL1.

[0071] In another embodiment, if the reference object 411 moves in response to movement of the handheld controller, then the representative object may be the handheld controller. In this case, the processor 104 may generate a line passing through the current representative position of the handheld controller and corresponding to the current position of the reference object 411 as a first reference line RL1.

[0072] In step S320, the processor 104 provides a reference object 411 at the second location L2 and determines a second reference line RL2 associated with the first point P1.

[0073] In one embodiment, after determining the first reference line L1, the processor 104 may instruct the user 299 to move to another location within the tracking area 220. In another embodiment, the user 299 may select any location other than the first location L1 within the tracking area 220 as the second location L2.

[0074] like Figure 4CAs illustrated, in response to determining that host 100 is at a second location L2, processor 104 displays reference object 411 at a specific location within a specific field of view.

[0075] In one embodiment, since any location within the tracking area 220 can be considered as a second location L2, the processor 104 may display the reference object 411 during the user 299 mobile host 100, but the invention is not limited thereto.

[0076] In one embodiment, when host 100 is at the second location L2, processor 104 may instruct user 299 to aim at first point P1 with reference object 411. Therefore, user 299 may move his / her head and / or move the handheld controller around to attempt to align reference object 411 with first point P1.

[0077] Once the user 299 determines that the reference object 411 is aligned with the first point P1 (e.g., the center of the lens of the external camera 210), the user 299 may perform a confirmation operation, such as pressing a specific button on the handheld controller or the host 100, to notify the host 100.

[0078] exist Figure 4D In response to the detection of a confirmation operation during the determination of providing a reference object 411 at the second location L2, the processor 104 obtains the second representative position of the representative object and the current position corresponding to the reference object 411.

[0079] Next, the processor 104 determines a second reference line RL2, which passes through the second representative position and the current position.

[0080] In one embodiment, if the reference object 411 moves in response to the movement of the host 100, then the representative object may be the host 100. In this case, as Figure 4E As shown in the figure, the processor 104 can generate a line passing through the current representative position of the host 100 and corresponding to the current position of the reference object 411 as a second reference line RL2.

[0081] In another embodiment, if the reference object 411 moves in response to movement of the handheld controller, then the representative object may be the handheld controller. In this case, the processor 104 may generate a line passing through the current representative position of the handheld controller and corresponding to the current position of the reference object 411 as a second reference line RL2.

[0082] In one embodiment, if the angle between the first reference line RL1 and the second reference line RL2 is large, the accuracy of determining the camera position will be better.

[0083] Therefore, while the host 100 moves within the tracking area 220, the processor 104 acquires an instantaneous reference line associated with the first point P1 during the movement of the host 100. In one embodiment, the instantaneous reference line may be determined based on a principle similar to that used to determine the first reference line RL1, but without requiring confirmation from the user 299. That is, regardless of when the reference object 411 moves, the line passing through the current representative position of the representative object and the reference position corresponding to the reference object 411 will be used as the instantaneous reference line, but the invention is not limited thereto.

[0084] In this case, processor 104 determines whether the angle between the first reference line RL1 and the instantaneous reference line is less than an angle threshold (e.g., 15 degrees). In response to determining that the angle between the first reference line RL1 and the instantaneous reference line is less than the angle threshold, processor 104 may instruct user 299 to move further from the first location L1 to achieve a better position determination result.

[0085] On the other hand, in response to determining that the angle between the first reference line RL1 and the instantaneous reference line is not less than an angle threshold, the processor 104 may instruct the user 299 that the current location is suitable to be regarded as the second location L2, but the present invention is not limited thereto.

[0086] In step S330, the processor 104 determines the camera position of the external camera 210 based on the first reference line RL1 and the second reference line RL2.

[0087] In one embodiment, the processor 104 may determine the position of the intersection of the first reference line RL1 and the second reference line RL2 as the camera position of the external camera 210.

[0088] In other embodiments, there may not be an intersection of the first reference line RL1 and the second reference line RL2. In this case, Figure 5 The mechanism described in the text can be used to determine the camera position.

[0089] See Figure 5 It illustrates a schematic diagram of determining the camera position based on a first reference line and a second reference line according to an embodiment of the present invention. Figure 5 As shown, the first reference line RL1 and the second reference line RL2 can be oblique lines that do not intersect.

[0090] In one embodiment, the processor 104 may acquire a first specific line segment 510 connecting a first reference line RL1 and a second reference line RL2, wherein the first specific line segment 510 corresponds to the shortest distance between the first reference line RL1 and the second reference line RL2. In this case, the processor 104 may determine the position of a first location point 511 on the first specific line segment 510 (which may be, but is not limited to, the midpoint of the first specific line segment 510) as the camera position of the external camera 210.

[0091] Based on the above, even if the external camera 210 is far away from the user 299, the host 100 and / or the tracking area 220, embodiments of the present invention can obtain the precise camera position of the external camera 210 (e.g., the position of the center of the lens of the external camera 210).

[0092] In step S340, the processor 104 acquires a specific image extracted by the external camera 210. In one embodiment, the specific image may be an image captured by the external camera 210 of the host 100 and / or the user 299, and the processor 104 may receive the specific image from the external camera 210 via a connection therebetween, but the invention is not limited thereto.

[0093] In step S350, processor 104 generates specific visual content by combining a specific image with a virtual scene based on camera position. In one embodiment, processor 104 may first perform image preprocessing (e.g., background removal) on the specific image and then overlay the (preprocessed) specific image onto the virtual scene based on camera position to generate specific visual content. Details on combining a specific image with a virtual scene based on camera position to form specific visual content can be found in technical documents related to MRC technology.

[0094] In embodiments of the present invention, the camera position can be accurately and conveniently obtained, thus enabling specific visual content generated based on the camera position to provide a more satisfying visual experience.

[0095] Furthermore, embodiments of the present invention also provide a mechanism for acquiring the orientation of the external camera 210, which can be used to improve the quality of the resulting specific visual content. This will be discussed in detail below.

[0096] In one embodiment, the orientation of the external camera 210 can be characterized by the orientation of the lens of the external camera 210. In this case, the normal plane of the lens of the external camera 210 can be used to determine the orientation of the lens of the external camera 210.

[0097] In one embodiment, the external camera 210 may be implemented as a smart device (e.g., a smartphone) with a rear camera, and the lens in question may be the lens of the rear camera. In this case, the back panel of the smart device can be used to characterize the normal plane of the lens of the external camera 210. Therefore, the host 100 can use the concept for determining the camera position of the external camera 210 (which can be understood as the position of the first point P1) to determine the positions of other points (inventorily referred to as the second point P2 and the third point P3) on the normal plane (e.g., the back panel). After obtaining the positions of the other points on the back panel, the host 100 can determine the normal plane of the lens of the external camera 210 and obtain its orientation accordingly.

[0098] In other embodiments, since the external camera 210 may not have a suitable plane for characterizing the normal plane, the processor 104 may instruct the user 299 to place an object (e.g., a flat panel) near the external camera 210 to characterize the normal plane of the lens of the external camera 210, but the invention is not limited thereto.

[0099] See Figure 6 The diagram illustrates the determination of the orientation of an external camera according to an embodiment of the present invention.

[0100] exist Figure 6 In this context, it is assumed that external camera 210 is a smart device (e.g., a tablet computer) that uses a rear camera to extract images of user 299. In this case, the rear panel of the smart device can be used to characterize the normal plane of the lens of the rear camera of the smart device.

[0101] In one embodiment, when user 299 and / or host 100 is at the first location L1, processor 104 may obtain the first reference line RL1 based on the above teaching content.

[0102] In one embodiment, processor 104 may determine a third reference line RL3 associated with a second point P2 based on a reference object 411 provided at a first location L1, wherein the second point P2 is associated with an external camera 210. In another embodiment, processor 104 may determine the third reference line RL3 in a manner similar to that used to determine the first reference line RL1. For example, when user 299 and / or host 100 are at the first location L1, processor 104 may instruct user 299 to select a desired point on the back panel and / or solid object representing the normal plane of the rear camera as the second point P2 (e.g., one of the corners of the back panel). Next, processor 104 may prompt user 299 to use... Figure 3 The reference object 411 in the system aims at the second point P2 and performs a confirmation operation. After detecting that the user 299 has performed a confirmation operation, the processor 104 can generate a line passing through the current representative position of the host 100 and corresponding to the current position of the reference object 411 as a third reference line RL3.

[0103] In one embodiment, processor 104 may determine a fifth reference line RL5 associated with a third point P3 based on a reference object 411 provided at a first location L1, wherein the third point P3 is associated with an external camera 210. In another embodiment, processor 104 may determine the fifth reference line RL5 in a manner similar to that used to determine the first reference line RL1. For example, when user 299 and / or host 100 are at the first location L1, processor 104 may instruct user 299 to select another desired point on the back panel and / or solid object representing the normal plane of the rear camera as the third point P3 (e.g., another corner of the back panel). Next, processor 104 may request user 299 to use... Figure 3 The reference object 411 in the system aims at the third point P3 and performs a confirmation operation. After detecting that the user 299 has performed a confirmation operation, the processor 104 can generate a line passing through the current representative position of the host 100 and corresponding to the current position of the reference object 411 as the fifth reference line RL5.

[0104] In one embodiment, when user 299 and / or host 100 are at the first location L1, processor 104 may require user 299 to sequentially aim at a first point P1, a second point P2, and a third point P3 using reference object 411. Specifically, when user 299 and / or host 100 are at the first location L1, processor 104 may sequentially require user 299 to: (1) aim at the first point P1 using reference object 411 and perform a confirmation operation; (2) aim at the second point P2 using reference object 411 and perform a confirmation operation; and (3) aim at the third point P3 using reference object 411 and perform a confirmation operation. In this case, processor 104 may sequentially acquire a first reference line RL1, a third reference line RL3, and a fifth reference line RL5.

[0105] In one embodiment, when the processor 104 instructs the user 299 to sequentially aim the reference object 411 at the first point P1, the second point P2, and the third point P3, the processor 104 may further provide a corresponding sequence number for each of the first point P1, the second point P2, and the third point P3. For example, the first point P1 may be labeled as point number 1, the second point P2 may be labeled as point number 2, and the third point P3 may be labeled as point number 3, but the invention is not limited thereto.

[0106] In one embodiment, processor 104 may instruct user 299 to move to a second location L2. When user 299 and / or host 100 are at the second location L2, processor 104 may obtain a second reference line RL2 based on the above teachings.

[0107] In one embodiment, processor 104 may determine a fourth reference line RL4 associated with the second point P2 based on a reference object 411 provided at the second location L2. In this embodiment, processor 104 may determine the fourth reference line RL4 in a manner similar to that used to determine the first reference line RL1. For example, when user 299 and / or host 100 are at the second location L2, processor 104 may instruct user 299 to use... Figure 3 The reference object 411 in the system aims at the second point P2 and performs a confirmation operation. After detecting that the user 299 has performed a confirmation operation, the processor 104 can generate a line passing through the current representative position of the host 100 and corresponding to the current position of the reference object 411 as a fourth reference line RL4.

[0108] In one embodiment, processor 104 may determine a sixth reference line RL6 associated with a third point P3, which is associated with an external camera 210, based on a reference object 411 provided at a second location L2. In this embodiment, processor 104 may determine the sixth reference line RL6 in a manner similar to that used to determine the first reference line RL1. For example, when user 299 and / or host 100 are at the second location L2, processor 104 may instruct user 299 to use… Figure 3 The reference object 411 in the system aims at the third point P3 and performs a confirmation operation. After detecting that the user 299 has performed a confirmation operation, the processor 104 can generate a line passing through the current representative position of the host 100 and corresponding to the current position of the reference object 411 as the sixth reference line RL6.

[0109] In one embodiment, when user 299 and / or host 100 are at the second location L2, processor 104 may require user 299 to sequentially aim at a first point P1, a second point P2, and a third point P3 using reference object 411. Specifically, when user 299 and / or host 100 are at the second location L2, processor 104 may sequentially require user 299 to: (1) aim at the first point P1 using reference object 411 and perform a confirmation operation; (2) aim at the second point P2 using reference object 411 and perform a confirmation operation; and (3) aim at the third point P3 using reference object 411 and perform a confirmation operation. In this case, processor 104 may sequentially acquire a second reference line RL2, a fourth reference line RL4, and a sixth reference line RL6.

[0110] In one embodiment, processor 104 may determine the position of a first point P1 (i.e., the camera position) based on a first reference line RL1 and a second reference line RL2, as taught above. Similarly, processor 104 may determine the position of a second point P2 (e.g., one of the corners on the rear panel) based on a third reference line RL3 and a fourth reference line RL4, and determine the position of a third point P3 (e.g., another corner on the rear panel) based on a fifth reference line RL5 and a sixth reference line RL6. Details of determining the positions of the second point P2 and the third point P3 can be found in [reference]. Figure 5 The description.

[0111] After obtaining the positions of the first point P1, the second point P2, and the third point P3, the processor 104 can determine the normal plane of the lens of the external camera 210, and thus determine the orientation of the external camera 210.

[0112] In one embodiment, the processor 104 may generate specific visual content by combining specific images with a virtual scene based on the camera position and orientation of the external camera 210, and details can be found in technical documents related to MRC technology.

[0113] See Figure 7 The diagram illustrates an application scenario according to an embodiment of the present invention. Figure 7 In this context, it is assumed that the image captured by the external camera 210 and sent to the user 299 has been preprocessed into a specific image 710, and the specific image 710 will be combined with the virtual scene 720 to form specific visual content.

[0114] In an embodiment, the virtual scene 720 may be illustratively designed to have two lightsabers that should appear to be held by the hand of the user 299 in the generated specific visual content.

[0115] In one embodiment, if the camera position of the external camera 210 (e.g., the position of the application camera in a virtual scene) is inaccurate, then the distance and / or orientation between the user 299 and the lightsaber may be incorrect, so that in the generated specific visual content 731 and the generated specific visual content 733, the lightsaber may appear not to be held in the hand of the user 299.

[0116] However, since embodiments of the present invention can obtain the precise camera position of the external camera 210, the specific image 710 and the virtual scene 720 can be appropriately combined so that the lightsaber can appear to be held in the hand of the user 299 in the generated specific visual content 732.

[0117] The present invention further provides a computer-readable storage medium for performing a method for providing visual content. The computer-readable storage medium comprises a plurality of program instructions (e.g., setup program instructions and deployment program instructions) embodied herein. These program instructions can be loaded into and executed by host 100 to perform the method for providing visual content and the functions of host 100 described above.

[0118] In general, embodiments of the present invention provide a novel, accurate, and convenient way to determine the camera position and the orientation of an external camera. Therefore, the quality of specific visual content generated by combining extracted images with a virtual scene based on the camera position and / or the orientation of the external camera can be improved.

[0119] It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the invention without departing from its scope or spirit. In view of the foregoing, it is intended that this invention cover modifications and variations thereof, provided that such modifications and variations fall within the scope of the following claims and their equivalents.

Claims

1. A method for providing visual content, applicable to a host computer, characterized in that, include: A reference object is provided at a first location to aim at a first point and to determine a first reference line associated with the first point, wherein the first point is associated with an external camera; Instructions to move the host from the first location to another location; Acquire an instantaneous reference line related to the first point during the movement of the host; In response to determining that the angle between the first reference line and the instantaneous reference line is less than an angle threshold, it indicates that the distance has been moved further from the first location; In response to determining that the angle between the first reference line and the instantaneous reference line is not less than the angle threshold, it indicates that the current location is suitable to be regarded as a second location; The reference object is provided at the second location to aim at the first point and thereby determine a second reference line associated with the first point; The camera position of the external camera is determined based on the first reference line and the second reference line; Acquire a specific image extracted by the external camera; as well as Specific visual content is generated by combining the specific image with a virtual scene based on the camera position.

2. The method for providing visual content according to claim 1, further comprising: A third reference line is determined based on the reference object provided at the first location, which is associated with the second point and the external camera. A fifth reference line is determined based on the reference object provided at the first location, which is associated with the third point and the external camera. A fourth reference line related to the second point is determined based on the reference object provided at the second location; A sixth reference line related to the third point is determined based on the reference object provided at the second location; The orientation of the external camera is determined based on the first point, the second point, and the third point.

3. The method for providing visual content according to claim 2, wherein the first point, the second point, and the third point are located on the normal plane of the lens of the external camera.

4. The method for providing visual content according to claim 3, further comprising: This indicates that an object is placed near the external camera to characterize the normal plane of the lens of the external camera.

5. The method for providing visual content according to claim 2, wherein each of the first point, the second point, and the third point has a corresponding sequence number.

6. The method for providing visual content according to claim 2, wherein the step of generating the specific visual content by combining the specific image with the virtual scene based on the camera position includes: The specific visual content is generated by combining the specific image with the virtual scene based on the camera position and the orientation of the external camera.

7. The method for providing visual content according to claim 1, comprising: In response to determining that the host is at the first location, the reference object is displayed at a specific location within a specific field of view, wherein the specific field of view corresponds to a specific eye image provided by the host; In response to determining that the host is at the second location, the reference object is displayed at the specific location within the specific field of view.

8. The method for providing visual content according to claim 1, wherein the step of determining the first reference line associated with the first point comprises: In response to the detection of a confirmation operation during the period when the reference object is provided at the first location, the current representative position of the representative object and the current position corresponding to the reference object are obtained; Determine the first reference line, wherein the first reference line passes through the current representative position and the current position.

9. The method for providing visual content according to claim 8, further comprising: Instructions to aim at the first point using the reference object.

10. The method for providing visual content according to claim 8, wherein the representative object is the host or a handheld controller connected to the host.

11. The method for providing visual content according to claim 1, wherein the step of determining the camera position of the external camera based on the first reference line and the second reference line comprises: Obtain a first specific line segment connecting the first reference line and the second reference line, wherein the first specific line segment corresponds to the shortest distance between the first reference line and the second reference line; The position of the first location point on the first specific line segment is determined as the camera position of the external camera.

12. The method for providing visual content according to claim 1, wherein the specific image is an image captured by the external camera onto the host, and the step of generating the specific visual content by combining the specific image with the virtual scene based on the camera position comprises: Perform image preprocessing on the specific image; The specific image is overlaid onto the virtual scene based on the camera position.

13. The method for providing visual content according to claim 1, wherein the host is a head-mounted display and the host provides the reference object in a transparency mode.

14. The method for providing visual content according to claim 1, wherein the first point corresponds to the center of the lens of the external camera.

15. A host computer, characterized in that, include: Storage circuitry stores program code; as well as The processor, coupled to the storage circuitry, accesses the program code for execution: A reference object is provided at a first location to aim at a first point and to determine a first reference line associated with the first point, wherein the first point is associated with an external camera; Instructions to move the host from the first location to another location; Acquire an instantaneous reference line related to the first point during the movement of the host; In response to determining that the angle between the first reference line and the instantaneous reference line is less than an angle threshold, it indicates that the distance has been moved further from the first location; In response to determining that the angle between the first reference line and the instantaneous reference line is not less than the angle threshold, it indicates that the current location is suitable to be regarded as a second location; The reference object is provided at the second location to aim at the first point and thereby determine a second reference line associated with the first point; The camera position of the external camera is determined based on the first reference line and the second reference line; Acquire a specific image extracted by the external camera; as well as Specific visual content is generated by combining the specific image with a virtual scene based on the camera position.

16. The host computer of claim 15, wherein the processor further performs: A third reference line is determined based on the reference object provided at the first location, which is associated with the second point and the external camera. A fifth reference line is determined based on the reference object provided at the first location, which is associated with the third point and the external camera. A fourth reference line related to the second point is determined based on the reference object provided at the second location; A sixth reference line related to the third point is determined based on the reference object provided at the second location; The orientation of the external camera is determined based on the first point, the second point, and the third point.

17. The host according to claim 16, wherein each of the first point, the second point and the third point has a corresponding sequence number.

18. The host computer of claim 16, wherein the processor performs: The specific visual content is generated by combining the specific image with the virtual scene based on the camera position and the orientation of the external camera.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium records an executable computer program, which is loaded by a host computer to perform the following steps: A reference object is provided at a first location to aim at a first point and to determine a first reference line associated with the first point, wherein the first point is associated with an external camera; Instructions to move the host from the first location to another location; Acquire an instantaneous reference line related to the first point during the movement of the host; In response to determining that the angle between the first reference line and the instantaneous reference line is less than an angle threshold, it indicates that the distance has been moved further from the first location; In response to determining that the angle between the first reference line and the instantaneous reference line is not less than the angle threshold, it indicates that the current location is suitable to be regarded as a second location; The reference object is provided at the second location to aim at the first point and thereby determine a second reference line associated with the first point; The camera position of the external camera is determined based on the first reference line and the second reference line; Acquire a specific image extracted by the external camera; as well as Specific visual content is generated by combining the specific image with a virtual scene based on the camera position.

Images