Display method, display system, and non-transitory computer-readable storage medium

Abstract

The present disclosure provides a display method, a display system, and a non-transitory computer-readable storage medium. The display method is used to display a virtual object and includes: identifying multiple features of a main object from a video; identifying a first fixed object from the video; determining a first target point in the video based on the features of the main object; calculating a first positional relationship between the first fixed object and the first target point; determining an anchor point in a virtual environment; and setting a second positional relationship between the anchor point and a second target point to control a display device to display the virtual object at the second target point in the virtual environment, wherein the second positional relationship corresponds to the first positional relationship. The display system of the present disclosure automatically displays the virtual object at a position suitable for the user by analyzing the main object in the video. When the user is performing physical activities, he or she does not need to manually move or adjust the virtual object.

Description

Technical Field

[0001] The present disclosure relates to a display method and system, and more particularly to a display method and system for dynamically displaying virtual objects. Background Art

[0002] In augmented reality (AR) applications, multiple virtual contents are provided to users and overlaid on the real scene. To simulate real-world objects, virtual contents in AR generally remain in a fixed position relative to the real scene. For example, a virtual display screen can maintain its posture (for example, staying on the surface of a real table) when the user changes position or turns in different directions to simulate a real display device. In this case, when the user follows the video played on the virtual display screen to perform physical activities that require changing posture (for example, fitness), if the virtual display screen leaves the user's field of view, the user must, for example, operate a controller to pull the virtual display screen back into the user's field of view. When performing physical activities, it is not convenient for the user to move the virtual display screen in AR, which also leads to users being less likely to consider using AR products to replace mobile devices that can provide audio and video instructions (for example, smartphones or tablets). Summary of the Invention

[0003] One embodiment of the present disclosure is a display method. The display method is used to display a virtual object and includes: using at least one processor to identify multiple features of a primary object from a video; using the at least one processor to identify a first fixed object from the video; using the at least one processor to determine a first target point in the video based on the features of the primary object; using the at least one processor to calculate a first positional relationship between the first fixed object and the first target point; using the at least one processor to determine an anchor point in a virtual environment; and using the at least one processor to set a second positional relationship between the anchor point and a second target point to control a display device to display the virtual object at the second target point in the virtual environment, wherein the second positional relationship corresponds to the first positional relationship.

[0004] In some embodiments, the operation of determining the first target point includes: determining, by the at least one processor, a position of the main object in a viewing direction and a tilt angle and a size of the virtual object based on the features of the main object in the video, thereby establishing a contour.

[0005] In some embodiments, the operations of determining the position of the main object in the viewing direction and the tilt angle and size of the virtual object include: setting a reference line according to the features of the main object by the at least one processor; determining the tilt angle of the virtual object according to the reference line by the at least one processor; determining the position of the main object in the viewing direction based on a viewing distance by the at least one processor; and determining the size of the virtual object based on a field of view and the viewing distance by the at least one processor.

[0006] In some embodiments, if the features of the main object repeatedly move from a first position to a second position, the operation of setting the reference line includes: by the at least one processor, if the features of the main object are located at the first position, setting a first line based on the features of the main object; by the at least one processor, if the features of the main object are located at the first position, setting a second line based on the features of the main object, wherein the first line intersects with the second line to form an angle; by the at least one processor, searching for a third line between the first line and the second line and intersecting therewith, wherein the third line bisects the angle between the first line and the second line; and by the at least one processor, using the third line as the reference line.

[0007] In some embodiments, if the main object is a person and the features of the main object are a portion of the person above the shoulders, the reference line is set from a first point at the lower end of the person's neck to a second point above the person's head.

[0008] In some embodiments, the at least one processor is used to control a positioning assistance device to perform an interactive operation in a physical environment to generate positioning data, and the at least one processor is used to identify a second fixed object in the physical environment based on the positioning data, and to convert the position of the second fixed object into the anchor point in the virtual environment.

[0009] In some embodiments, the first fixed object is located on an attached object in the video.

[0010] In some embodiments, if the main object changes its posture, the display method further includes: determining, by the at least one processor, a third target point in the video based on the features of the main object; calculating, by the at least one processor, a third positional relationship between the first fixed object and the third target point; and setting, by the at least one processor, a fourth positional relationship between the anchor point and a fourth target point to control the display device to display the virtual object at the fourth target point in the virtual environment, wherein the fourth positional relationship corresponds to the third positional relationship.

[0011] In some embodiments, if the features of the main object repeatedly move from a first position to a second position and the display device is used to repeatedly move from a third position to a fourth position according to the movement of the features of the main object, the display method further includes: determining, by the at least one processor, the first target point in the video if the features of the main object are located at the first position; determining, by the at least one processor, a fifth target point in the video if the features of the main object are located at the second position; calculating, by the at least one processor, a fifth positional relationship between the first fixed object and the fifth target point; and calculating, by the at least one processor, a fifth positional relationship between the first fixed object and the fifth target point. , if the display device is located at the third position, controlling the display device to display the virtual object at the second target point in the virtual environment; by means of the at least one processor, if the display device is not located at the third position and the fourth position, controlling the display device to display the virtual object by following an upward movement trajectory of the display device and maintaining a preset distance from the user; and by means of the at least one processor, if the display device is located at the fourth position, setting a seventh positional relationship between the anchor point and a seventh target point to control the display device to display the virtual object at the seventh target point in the virtual environment, wherein the seventh positional relationship corresponds to the fifth positional relationship.

[0012] In some embodiments, the display method further includes: determining, by the at least one processor, a sixth target point in the virtual environment if the virtual object is displayed by following the upward movement trajectory of the display device; and controlling, by the at least one processor, the display device to display the virtual object at the sixth target point in the virtual environment if the display device moves from a position that is neither the third position nor the fourth position back to the third position.

[0013] Another embodiment of the present disclosure is a display system. The display system includes a display device and at least one processor. The display device is used to display a virtual object. The at least one processor is coupled to the display device and is used to: identify multiple features of a main object from a video; identify a first fixed object from the video; determine a first target point in the video based on the features of the main object; calculate a first positional relationship between the first fixed object and the first target point; determine an anchor point in a virtual environment; and set a second positional relationship between the anchor point and a second target point to control a display device to display the virtual object at the second target point in the virtual environment, wherein the second positional relationship corresponds to the first positional relationship.

[0014] In some embodiments, the at least one processor is configured to determine a position of the main object in a viewing direction and a tilt angle and a size of the virtual object according to the features of the main object in the video, thereby establishing a contour.

[0015] In some embodiments, the at least one processor is used to: set a reference line based on the features of the main object; determine the tilt angle of the virtual object based on the reference line; determine the position of the main object in the viewing direction based on a viewing distance; and determine the size of the virtual object based on a field of view and the viewing distance.

[0016] In some embodiments, if the features of the main object repeatedly move from a first position to a second position, the at least one processor is configured to: if the features of the main object are located at the first position, set a first line based on the features of the main object; if the features of the main object are located at the first position, set a second line based on the features of the main object, wherein the first line intersects the second line to form an angle; search for a third line between the first line and the second line and intersecting therewith, wherein the third line bisects the angle between the first line and the second line; and use the third line as the reference line.

[0017] In some embodiments, the display system further includes a positioning assistance device, wherein the positioning assistance device is used to perform an interactive operation in a physical environment to generate positioning data, wherein the at least one processor is used to identify a second fixed object in the physical environment based on the positioning data, and to convert the position of the second fixed object into the anchor point in the virtual environment.

[0018] In some embodiments, the first fixed object is located on an attached object in the video.

[0019] In some embodiments, if the main object changes its posture, the at least one processor is further used to: determine a third target point in the video based on the features of the main object; calculate a third positional relationship between the first fixed object and the third target point; and set a fourth positional relationship between the anchor point and a fourth target point to control the display device to display the virtual object at the fourth target point in the virtual environment, wherein the fourth positional relationship corresponds to the third positional relationship.

[0020] In some embodiments, if the features of the main object repeatedly move from a first position to a second position and the display device is used to repeatedly move from a third position to a fourth position according to the movement of the features of the main object, the at least one processor is further used to: determine the first target point in the video if the features of the main object are located at the first position; determine a fifth target point in the video if the features of the main object are located at the second position; calculate a fifth positional relationship between the first fixed object and the fifth target point; and calculate a fifth positional relationship between the first fixed object and the fifth target point if the display device is located at The third position controls the display device to display the virtual object at the second target point in the virtual environment; if the display device is not located at the third position and the fourth position, controls the display device to display the virtual object by following an upward movement trajectory of the display device and maintaining a preset distance from the user; and if the display device is located at the fourth position, sets a seventh position relationship between the anchor point and a seventh target point to control the display device to display the virtual object at the seventh target point in the virtual environment, wherein the seventh position relationship corresponds to the fifth position relationship.

[0021] In some embodiments, the at least one processor is further used to: determine a sixth target point in the virtual environment if the virtual object is displayed by following the upward movement trajectory of the display device; and control the display device to display the virtual object at the sixth target point in the virtual environment if the display device moves from a position that is neither the third position nor the fourth position back to the third position.

[0022] Another embodiment of the present disclosure is a non-transitory computer-readable storage medium having a computer program for executing a display method. The display method is used to display a virtual object and includes: using at least one processor to identify multiple features of a primary object from a video; using the at least one processor to identify a first fixed object from the video; using the at least one processor to determine a first target point in the video based on the features of the primary object; using the at least one processor to calculate a first positional relationship between the first fixed object and the first target point; using the at least one processor to determine an anchor point in a virtual environment; and using the at least one processor to set a second positional relationship between the anchor point and a second target point to control a display device to display the virtual object at the second target point in the virtual environment, wherein the second positional relationship corresponds to the first positional relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The present invention is a block diagram of a display system according to some embodiments of the present disclosure.

[0024] Figure 2The present invention is a flowchart of a display method according to some embodiments of the present disclosure.

[0025] Figure 3 The present invention is a flowchart illustrating an operation in a display method according to some embodiments of the present disclosure.

[0026] Figure 4 Schematic diagram of frames in a video analyzed by a display system according to some embodiments of the present disclosure.

[0027] Figures 5A-5B The present invention is a schematic diagram illustrating the operation of a display system according to some embodiments of the present disclosure.

[0028] Figures 6A-6B Schematic diagram of two frames in a video analyzed by a display system according to some embodiments of the present disclosure.

[0029] Figures 7A to 7D The present invention is a schematic diagram illustrating the operation of a display system according to some embodiments of the present disclosure.

[0030] Explanation of symbols:

[0031] 100: Display system

[0032] 110: Processor

[0033] 120: Display device

[0034] 130: Positioning auxiliary device

[0035] 200: Display method

[0036] AO: Accessory Object

[0037] FM: Exercise Mat

[0038] FOV: Field of View

[0039] FO1, FO2: Fixed objects

[0040] HD1, HD2, HD5, HD7: horizontal distance

[0041] MO: Main Object

[0042] Q:Angle

[0043] RL_1: First Line

[0044] RL_2: Second Line

[0045] RL_3: Third Line

[0046] SA: tilt angle

[0047] TO1,TO2,TO4,TO5,TO6,TO7: target points

[0048] UR: User

[0049] VO: Virtual Object

[0050] VD: Viewing distance

[0051] VD1, VD2, VD5, VD7: vertical distance

[0052] VF, VF1, VF2: screen

[0053] X: Position

[0054] S201~S206: Operation

[0055] S301~S306: Sub-operation DETAILED DESCRIPTION

[0056] The following is a detailed description of embodiments with reference to the accompanying drawings. However, the specific embodiments described are only used to explain the present invention and are not used to limit the present invention. The description of the structural operations is not used to limit the order of their execution. Any structure formed by re-combining the elements to produce a device with equivalent functions is within the scope of the present disclosure.

[0057] As used herein, “coupled” or “connected” may refer to direct physical or electrical contact or indirect physical or electrical contact between two or more elements, or may refer to mutual operation or movement between two or more elements.

[0058] See also Figure 1 , Figure 1 FIG1 is a block diagram of a display system 100 according to some embodiments of the present disclosure. The display system 100 includes a processor 110, a display device 120, and a positioning assistance device 130. The processor 110 is coupled to the display device 120 and the positioning assistance device 130. In some embodiments, the display device 120 is a wearable device, such as a head-mounted display. The processor 110 is used to analyze a video to control the display device 120 to display a virtual object VO (described later). Figure 5A and 5B As shown in FIG, the virtual object VO is used to play the video analyzed by the processor 110 in a virtual environment, wherein the processor 110 determines the display position of the virtual object VO based on the analysis result of the video. The virtual object VO can be a two-dimensional object, such as a virtual display screen. In this way, the user UR (as described later) Figure 5A and 5B As shown in FIG. 1 , the user can view the virtual object VO through the display device 120 to follow the instructions of the video.

[0059] In some embodiments, the processor 110 may be implemented by one or more central processing units (CPUs), application specific integrated circuits (ASICs), microprocessors, system-on-chips (SoCs), or other suitable processing units. It should be understood that the processor 110 may be a server independent of the display device 120 and may transmit the video and the analysis results of the video to the display device 120 via a network. The display device 120 may include other processors (not shown). Figure 1 ), and the other processors are used to generate a virtual object VO according to the analysis result. In addition, the processor 110 can also be integrated into the display device 120, that is, the display device 120 can be an integrated head-mounted display.

[0060] In some embodiments, the positioning assistance device 130 is placed in a physical environment and is used to emit invisible light (i.e., perform an interactive operation in a physical environment). In some embodiments, the positioning assistance device 130 can be implemented by one or more base stations used to form a tracking system. One or more receivers can be set on the display device 120 to receive or intercept invisible light, thereby generating positioning data. The processor 110 or the display device 120 is used to calculate an initial position of the display device 120 in the physical environment based on the positioning data, and to convert the initial position of the display device 120 into a reference point in the virtual environment to determine the position of the virtual object VO in the virtual environment. The above-mentioned embodiment will be further described in the following paragraphs.

[0061] In other embodiments, the positioning assistance device 130 is configured to capture at least one photo in a physical environment (i.e., to perform an interactive operation in a physical environment). Specifically, the positioning assistance device 130 may include one or more cameras. If a marker (e.g., a Quick Response Matrix (QR code)) is located in the physical environment, the at least one photo generated by the positioning assistance device 130 may include an image of the marker. In response to receiving the at least one photo generated by the positioning assistance device 130, the processor 110 or the display device 120 may use the at least one photo as positioning data and calculate the position of the marker in the physical environment by, for example, calculating the size and position of the marker image in the at least one photo. In this way, the position of the marker can be converted into a reference point in the virtual environment to determine the position of the virtual object VO in the virtual environment. This embodiment will be further described in the following sections. If there is no marker in the physical environment, the processor 110 may determine a reference point in the virtual environment (e.g., a reference point) by performing image recognition, such as comparing the video analyzed by the processor 110 with the at least one photo captured by the positioning assistance device 130. Figure 5A The above-mentioned embodiment will be further described in the following paragraphs.

[0062] See also Figure 2 , Figure 2 FIG. 2 is a flow chart of a display method 200 according to some embodiments of the present disclosure. Figure 1 The processor 110 in the embodiment may execute the display method 200, but the present disclosure is not limited thereto. Figure 2 As shown, the display method 200 includes operations S201 to S206. Operations S201 to S206 will be referred to later. Figure 4 Detailed description.

[0063] Before executing the display method 200, the processor 110 receives the video to be analyzed in one of a variety of ways. In some embodiments, the video is stored in a memory (not shown) of the display system 100, allowing the processor 110 to access the video through the memory. In other embodiments, the video is streamed over a network, allowing the processor 110 to receive the video from the network.

[0064] See also Figure 4 , Figure 4 FIG. 1 is a schematic diagram illustrating a frame VF in a video to be analyzed by the processor 110 according to some embodiments of the present disclosure. Figure 4 As shown, the video shows a main object MO with an action (or posture) on an accessory object AO. For example, a character in the video (i.e., the main object MO) is performing physical activities (e.g., fitness or yoga) on an exercise mat (i.e., the accessory object AO).

[0065] In operation S201, processor 110 identifies multiple features of a primary object MO from the video. In some embodiments, the multiple features of primary object MO correspond to a portion of a person's body above the shoulders. That is, the portion may include the person's head and neck. It should be understood that processor 110 may use 3D object recognition technology to identify the multiple features of primary object MO.

[0066] In operation S202, the processor 110 identifies a first fixed object FO1 from the video. Figure 4 As shown, the first fixed object FO1 can be a mark on the accessory object AO, or can be preset to the center point of the left short side of the accessory object AO. However, the present disclosure is not limited to this. The first fixed object FO1 can be another point on the accessory object AO (e.g., a corner) or other suitable point in the video (e.g., the center of a character's eyes).

[0067] In operation S203, the processor 110 determines a first target point TO1 in the video according to a plurality of features of the main object MO (eg, Figure 4As shown). A first positional relationship between the first fixed object FO1 and the first target point TO1 can be used to determine the position of the virtual object VO in the virtual environment. The above-mentioned implementation will be further described in the following paragraphs. Figure 3 , Figure 3 FIG. 1 is a flow chart of operation S203. Figure 3 As shown, operation S203 includes sub-operations S301 to S306.

[0068] In sub-operation S301, the processor 110 sets a reference line according to multiple features of the main object MO. In some embodiments, the main object MO in the video maintains a posture for a period of time. If the main object MO maintains such a posture, the main object MO is positioned in the same direction as the main object MO. Figure 4 For the posture shown, the processor 110 sets a first line RL_1 as a reference line, where the first line RL_1 is set from a first point at the lower end of the character's neck to a second point above the character's head. In other words, the first line RL_1 can actually be a connecting line between the back of the character's head and the shoulders.

[0069] In some other embodiments, the main object MO in the video repeatedly changes its posture from a first posture (eg, Figure 4 The main object MO is in a crunch position) and changes to a second position (for example, the main object MO lies on the accessory object AO). The change in the posture of the main object MO will drive the corresponding changes in multiple features of the main object MO. For example, multiple features of the main object MO repeatedly move from a first position to a second position. In this case, if the main object MO is in the first posture, the processor 110 sets the first line RL_1 (such as Figure 4 If the main object MO is in the second posture, the processor 110 sets a second line RL_2 (as shown in FIG. 1 ) according to a plurality of features located at a second position corresponding to the second posture of the main object MO. Figure 4 As shown). It should be understood that the first line RL_1 and the second line RL_2 intersect to form an angle Q. After setting the first line RL_1 and the second line RL_2, the processor 110 searches for a third line RL_3 that is between the first line RL_1 and the second line RL_2 and intersects therewith. Figure 4 As shown, the third line RL_3 bisects the angle Q between the first line RL_1 and the second line RL_2. Accordingly, the processor 110 uses the third line RL_3 as a reference line.

[0070] In sub-operation S302, the processor 110 determines the reference line to be set in sub-operation S301. Figures 5A-5BIt is worth noting that the virtual object VO does not really exist in the video analyzed by the processor 110. Figure 4 The virtual object VO shown in FIG is for convenience of explanation only. In some embodiments, the virtual object VO is set by the processor 110 to be parallel to the reference line. Accordingly, the tilt angle SA of the virtual object VO is determined to set the virtual object VO parallel to the reference line. For example, if the first line RL_1 is set as the reference line, the virtual object VO will be parallel to the reference line. Figure 4 A vertical dashed line is shown with an inclined angle SA, thereby being parallel to the reference line (ie, the first line RL_1 ).

[0071] In sub-operation S303, in order to determine the position of the first target point TO1, the processor 110 first uses a viewing distance VD. Figure 4 In the embodiment of FIG, the processor 110 determines a position X in the viewing direction of the person (ie, the main object MO), and the position X is separated from the center of the person's eyes by a viewing distance VD. In this example, the viewing distance VD may be the average arm length of a human.

[0072] In other embodiments, the processor 110 determines the position X in the viewing direction of the person (i.e., the main object MO) by separating the position X from a reference line (i.e., the first line RL_1) by a viewing distance VD. In this example, the viewing distance VD may be the sum of the average human arm length and the average human head diameter.

[0073] In sub-operation S304, the processor 110 determines a size of the virtual object VO based on a field of view (FOV) and a viewing distance (VD). It should be understood that after determining the tilt angle SA, the processor 110 may determine the size of the virtual object VO by placing the virtual object VO within the field of view (FOV) at the tilt angle SA, separating the virtual object VO from the end of the field of view (FOV) by the viewing distance (VD), and further extending the virtual object VO to the edge of the field of view (FOV). In some embodiments, the viewing distance (VD) and the field of view (FOV) are preset to provide a comfortable viewing experience for the user UR. In other embodiments, the average human arm length, average human head diameter, and human field of view (FOV) may be pre-stored in the memory of the display system 100.

[0074] In sub-operation S305, the processor 110 further creates a profile according to the viewing distance VD and the tilt angle SA and size of the virtual object VO to determine the position of the first target point TO1. Figure 4 The outline is shown in the video for the sake of illustration only, but the outline can be Figure 4The edges of the virtual object VO are represented to simplify the graphics. In some embodiments, the outline is a rectangle. That is, the outline includes four sides (e.g., top side, bottom side, left side, right side), and each side is perpendicular to the two sides connected to it. It should be understood that Figure 4 Only one of the four sides is shown (eg, the left side).

[0075] In sub-operation S306, the processor 110 sets a first target point TO1 on the contour. For better understanding, Figure 4 The first target point TO1 is depicted at the lower end of the left side of the outline. In some embodiments, the first target point TO1 is located at the center of the lower side of the outline, but the present disclosure is not limited thereto.

[0076] The first target point TO1 may also be set at any appropriate point on the contour.

[0077] In operation S204, after determining the first fixed object FO1 and the first target point TO1, the processor 110 calculates a first positional relationship between the first fixed object FO1 and the first target point TO1. Figure 4 As shown, in some embodiments, the processor 110 calculates a length of a vertical distance VD1 and a length of a horizontal distance HD1 to obtain a first positional relationship between the first fixed object FO1 and the first target point TO1 .

[0078] Operation S205 refers to Figure 5A Provide explanation. Figure 5A Describe the user UR follows Figure 4 Following the instructions in the video, perform one of the physical activities on an exercise mat FM in the physical environment. The virtual objects VO provided to the user UR in the virtual environment are only shown for the convenience of explanation. Figure 5A In the physical environment. In practice, the virtual object VO is only presented to the user UR by the display device 120, and other people in the physical environment cannot directly see the virtual object VO. In operation S205, after the user UR has mounted the display device 120, the processor 110 identifies a second fixed object FO2 in the physical environment. The position of the second fixed object FO2 is converted by the processor 110 into an anchor point in the virtual environment to subsequently determine the position of the virtual object VO.

[0079] In some embodiments, before executing operation S205, a marker for interacting with the positioning assisting device 130 is set at the center of the left short side of the exercise mat FM (i.e., the position corresponding to the position of the first fixed object FO1, but the present disclosure is not wired thereto). As described above, the processor 110 obtains the position of the marker by controlling the positioning assisting device 130 to interact with the marker. For example, the processor 110 controls the positioning assisting device 130 to take a photo with the image of the marker. Accordingly, the processor 110 uses the marker as the second fixed object FO2 and converts the second fixed object FO2 into an anchor point in the virtual environment. The processor 110 determines that the anchor point is located at a position corresponding to the second fixed object FO2. Figure 4 The position corresponding to the position of the first fixed object FO1 in the video.

[0080] In other embodiments, no markers are provided in the physical environment, and the processor 110 compares the photos taken by the positioning auxiliary device 130 with the Figure 4 The second fixed object FO2 is identified by image recognition based on the video of the user UR. In other embodiments, no markers are provided in the physical environment, and the user UR's initial posture (e.g., standing) is the same as the main object MO. In this case, the processor 110 controls the positioning assisting device 130 to interact with the receiver on the display device 120 to obtain the initial position of the display device 120 mounted on the user UR and sets the initial position of the display device 120 as the second fixed object FO2.

[0081] In operation S206, the processor 110 controls the display device 120 to display the virtual object VO at the second target point TO2 in the virtual environment by setting a second position relationship corresponding to the first position relationship between the anchor point and the second target point TO2. Figure 5A As shown, the second positional relationship includes a vertical distance VD2 and a horizontal distance HD2 in the virtual environment. In this embodiment, the vertical distance VD2 corresponds to (e.g., is the same as) the vertical distance VD1, and the horizontal distance HD2 corresponds to (e.g., is the same as) the horizontal distance HD1. By adding the vertical distance VD2 and the horizontal distance HD2 to the coordinates of the anchor point, the processor 110 can determine the second target point TO2 with reference to the anchor point in the virtual environment. Accordingly, the processor 110 controls the display device 120 to display the virtual object VO at an inclined angle SA based on the second target point TO2. For example, the center of the lower side of the virtual object VO will overlap with the second target point TO2. Furthermore, the distance between the virtual object VO and the center of the user's eyes is essentially the viewing distance VD. It should be understood that the position of the second target point TO2 in the virtual environment corresponds to the position of the first target point TO1 in the video.

[0082] See also Figure 5B , Figure 5B The user UR is described as performing another physical activity on the exercise mat FM in the physical environment. The virtual object VO provided to the user UR in the virtual environment is only shown for the convenience of explanation. Figure 5B If the main object MO in the video changes its posture from Figure 4 The posture shown changes to Figure 5B The processor 110 can learn another posture corresponding to the posture of the user UR by identifying multiple features of the main object MO. Figure 4 The main object MO changes its posture. Accordingly, the processor 110 may Figure 4 When the main object MO changes its posture, operations S203 to S204 and S206 are executed again to determine a third target point (not shown in the figure, but can be understood as the new first target point TO1) in the video based on multiple features of the main object MO. The processor 110 calculates a third positional relationship between the first fixed object FO1 and the third target point (i.e., the new first target point TO1). Figure 5B As shown, the processor 110 controls the display device 120 to display the virtual object VO at the fourth target point TO4 in the virtual environment by setting a fourth positional relationship corresponding to the third positional relationship between the second fixed object FO2 and the fourth target point TO4. The descriptions of determining the third target point (i.e., the new first target point TO1), calculating the third positional relationship, and controlling the display device 120 to display the virtual object VO at the fourth target point TO4 are the same or similar to the descriptions of operations S203-S204 and S206, respectively, and therefore are omitted here.

[0083] In the aforementioned embodiment, when the user UR is performing the same physical activity as the primary object MO in the video, the display system 100 is configured to display the virtual object VO at an appropriate location (e.g., in front of the user UR) by identifying multiple features of the primary object MO. However, if the user UR is, for example, resting and therefore has a different posture than the primary object MO, the virtual object VO generated by the aforementioned multiple operations may not be displayed at the appropriate location.

[0084] See also Figures 6A-6B picture, Figures 6A-6B FIG. 1 illustrates a plurality of frames in a video analyzed by the processor 110, wherein the main object MO repeatedly changes its posture from a first sub-posture (e.g., Figure 6A ) changes to a second sub-posture (as shown Figure 6B The first and second sub-postures belong to the same posture (e.g. sit-ups). Figure 6AAs shown, the processor 110 determines the first fixed object FO1 and the first target point TO1 in a first frame VF1 corresponding to the first sub-gesture in the video, thereby calculating the first positional relationship between the first fixed object FO1 and the first target point TO1 (i.e., the length of the vertical distance VD1 and the length of the horizontal distance HD1). Figure 6B As shown, the processor 110 determines the first fixed object FO1 and a fifth target point TO5 in a second frame VF2 corresponding to the second sub-gesture in the video, thereby calculating a fifth positional relationship between the first fixed object FO1 and the fifth target point TO5 (i.e., the length of a vertical distance VD5 and the length of a horizontal distance HD5). In other words, the processor 110 determines the two display positions of the virtual object VO based on the two sub-gestures of the main object MO by performing operations S203-S204 and S206 on the two sub-gestures in the video.

[0085] See also Figures 7A to 7D , Figures 7A to 7D Describe the user UR wearing the display device 120 according to Figures 6A-6B It should be understood that if the user UR Figures 6A-6B The main object MO in the display device 120 will automatically Figure 7A The position shown (ie, the third position) moves to Figure 7D Another position (i.e., the fourth position) shown in FIG. Figure 7A As shown, the user UR maintains the same first sub-posture as the main object MO in the first frame VF1. Figure 7A The processor 110 controls the display device 120 to display the virtual object VO at the second target point TO2 in the virtual environment by setting a second positional relationship (i.e., the length of the vertical distance VD2 and the length of the horizontal distance HD2) between the second fixed object FO2 and the second target point TO2 that corresponds to the first positional relationship. In this example, since the virtual object VO is positioned according to the initial stage of the posture (e.g., the first sub-posture), the user UR at rest is more likely to be in the virtual object VO than in the virtual object VO. Figure 5A In the embodiment, the virtual object VO can be viewed more easily.

[0086] like Figure 7B As shown, the user UR changes his posture to a posture different from the first sub-posture and the second sub-posture, so that the display device 120 is not located at Figure 7A and 7D If the display device 120 is not located Figure 7A and 7DThe processor 110 controls the display device 120 to display the virtual object VO by following the upward movement trajectory of the user UR's head (or the upward movement trajectory of the display device 120) and maintaining a preset distance from the center of the user UR's eyes. The processor 110 can use the positioning auxiliary device 130 to detect the movement trajectory of the user UR's head or the display device 120. In some embodiments, the preset distance can be the aforementioned viewing distance VD (e.g., Figure 7B ), or it may be the vertical distance from the center of the user UR's eyes to the virtual object VO. In this way, if the user UR is unable to perform the second sub-gesture like the main object MO in the second frame VF2, the virtual object VO can still be displayed in an appropriate position for the user UR to see.

[0087] In some embodiments, when the virtual object VO is displayed at a predetermined distance from the display device 120, the processor 110 further determines a sixth target point TO6 in the virtual environment. For example, the processor 110 may control the positioning assisting device 130 to interact with a marker or a receiver located on the display device 120 to calculate the position of the display device 120. The processor 110 then calculates the position of the sixth target point TO6 based on the position of the display device 120, the predetermined distance, and the size of the virtual object VO. Based on this, a sixth positional relationship between the sixth target point TO6 and the second fixed object FO2 may be calculated.

[0088] like Figure 7C As shown, when user UR Figure 7B When the posture returns to the first sub-posture to try to make the second sub-posture next time, the display device 120 Figure 7B Move back to the position Figure 7A To allow the user UR to comfortably view the virtual object VO when the user UR still only performs the posture shown in FIG. 7B next time, the processor 110 sets a sixth positional relationship between the sixth target point TO6 and the second fixed object FO2, thereby controlling the display device 120 to display (or maintain) the virtual object VO at the sixth target point TO6 in the virtual environment.

[0089] The user UR attempts to change his posture from the first sub-posture to Figure 7D The second sub-gesture of . Figure 7B Before the user UR makes the gesture, the processor 110 controls the display device 120 to display the virtual object VO at the sixth target point TO6 in the virtual environment. Figure 7BAfter the user UR makes the gesture, the processor 110 controls the display device 120 to display the virtual object VO by following the upward movement trajectory of the user UR's head (or the display device 120) and maintaining a preset distance from the center of the user UR's eyes. The above embodiment can be understood as the virtual object VO being substantially pushed diagonally upward by the user. Figure 7D As shown, the user UR makes the same second sub-gesture as the main object MO in the second frame VF2, so that the display device 120 is positioned at a position larger than Figures 7A to 7C If the user UR makes the second sub-gesture (or the display device 120 is at a higher position Figure 7D The processor 110 controls the display device 120 to display the virtual object VO at the seventh target point TO7 in the virtual environment by setting a seventh position relationship (i.e., a length of a vertical distance VD7 and a length of a horizontal distance HD7) between the second fixed object FO2 and a seventh target point TO7 corresponding to the fifth position relationship (i.e., a length of a vertical distance VD5 and a length of a horizontal distance HD5).

[0090] In some embodiments, the posture performed by the user UR exceeds the standard of physical activity. For example, an angle (not shown) between the upper body of the user UR and the exercise mat FM is greater than Figure 6B The second sub-pose corresponds to another angle between the upper body of the main object MO and the accessory object AO. In this case, because the processor 110 has previously determined that a point of the virtual object VO overlaps with the seventh target point TO7 corresponding to the fifth target point TO5, the virtual object VO is still displayed at the seventh target point TO7, notifying the user UR that their pose meets the standard.

[0091] In summary, the display system 100 of the present disclosure automatically displays the virtual object VO at a position suitable for the user UR by analyzing the main object MO in the video. In this way, when the user UR is performing physical activities, he or she does not need to manually move or adjust the virtual object VO.

[0092] The methods of the present disclosure may be in the form of a program. The program may be included in a physical medium, such as a floppy disk, a CD, a hard disk, or any other non-transitory computer-readable storage medium, wherein when the program is loaded and executed by a computer, the computer becomes an apparatus for implementing the method. The program may also be transmitted via some transmission medium, such as a wire or cable, via optical fiber, or via any other transmission mode, wherein when the program is received, loaded, and executed by a computer, the computer becomes an apparatus for implementing the method. When implemented on a general-purpose processor, the program combines with the processor to provide a unique device that operates similarly to an application-specific logic circuit.

[0093] Although the present disclosure has been disclosed in the form of an embodiment as described above, it is not intended to limit the present disclosure. A person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be determined by the scope of the attached patent application.

Claims

1. A display method, characterized in that: Used to display a virtual display screen in a virtual environment, and comprising: Recognizing, by at least one processor, a plurality of features of a person from a video to be played by the virtual display screen in the virtual environment; identifying, by the at least one processor, a first fixed object on an attached object from the video; Determining, by the at least one processor, a first target point in the video according to the features of the person; calculating, by the at least one processor, a first positional relationship between the first fixed object and the first target point, wherein the first positional relationship is used to determine a position of the virtual display screen in the virtual environment; Determining, by the at least one processor, an anchor point in the virtual environment, the at least one processor being configured to identify a second fixed object in the physical environment and convert a position of the second fixed object to the anchor point in the virtual environment; and By means of the at least one processor, a second positional relationship is set between the anchor point and a second target point to control a display device to display the virtual display screen at the second target point in the virtual environment, wherein the second positional relationship corresponds to the first positional relationship, the position of the anchor point corresponds to the position of the first fixed object, and the position of the second target point corresponds to the position of the first target point.

2. The display method according to claim 1, wherein: The operations for determining the first target point include: The at least one processor determines a position of the character in a viewing direction and an inclination angle and a size of the virtual display screen according to the features of the character in the video, thereby establishing a contour.

3. The display method according to claim 2, wherein: The operation of determining the position of the character in the viewing direction and the tilt angle and size of the virtual display screen includes: Setting a reference line according to the features of the character by the at least one processor; Determining, by the at least one processor, the tilt angle of the virtual display screen according to the reference line; determining, by the at least one processor, the position of the character in the viewing direction based on a viewing distance; and The size of the virtual display screen is determined by the at least one processor based on a field of view and the viewing distance.

4. The display method according to claim 3, wherein: If the features of the person repeatedly move from a first position to a second position, the operation of setting the reference line includes: By means of the at least one processor, if the features of the character are located at the first position, setting a first line according to the features of the character; By means of the at least one processor, if the features of the person are located at the first position, setting a second line according to the features of the person, wherein the first line intersects the second line to form an angle; searching, by the at least one processor, for a third line between and intersecting the first line and the second line, wherein the third line bisects the angle between the first line and the second line; and The at least one processor uses the third line as the reference line.

5. The display method according to claim 3, wherein: If the features of the character are a portion above the character's shoulders, the reference line is set from a first point at the lower end of the character's neck to a second point above the character's head.

6. The display method according to claim 1, wherein: The at least one processor is used to control a positioning auxiliary device to perform an interactive operation in a physical environment to generate positioning data, and the at least one processor is used to identify a second fixed object in the physical environment based on the positioning data, and to convert the position of the second fixed object into the anchor point in the virtual environment.

7. The display method according to claim 1, wherein: If the character changes its posture, the display method further includes: Determining, by the at least one processor, a third target point in the video according to the features of the person; calculating, by the at least one processor, a third positional relationship between the first fixed object and the third target point; and The at least one processor sets a fourth positional relationship between the anchor point and a fourth target point to control the display device to display the virtual display screen at the fourth target point in the virtual environment, wherein the fourth positional relationship corresponds to the third positional relationship.

8. The display method according to claim 1, wherein: If the features of the character repeatedly move from a first position to a second position and the display device is configured to repeatedly move from a third position to a fourth position according to the movement of the features of the character, the display method further includes: determining, by the at least one processor, the first target point in the video if the features of the person are located at the first position; determining, by the at least one processor, a fifth target point in the video if the features of the person are located at the second position; calculating, by the at least one processor, a fifth positional relationship between the first fixed object and the fifth target point; controlling, by the at least one processor, the display device to display the virtual display screen at the second target point in the virtual environment if the display device is located at the third position; If the display device is not located at the third position or the fourth position, controlling the display device to display the virtual display screen by following an upward movement trajectory of the display device and maintaining a predetermined distance from the user by the at least one processor; and By means of the at least one processor, if the display device is located at the fourth position, a seventh position relationship is set between the anchor point and a seventh target point to control the display device to display the virtual display screen at the seventh target point in the virtual environment, wherein the seventh position relationship corresponds to the fifth position relationship.

9. The display method according to claim 8, wherein: Also includes: determining, by the at least one processor, a sixth target point in the virtual environment if the virtual display screen is displayed by following the upward movement trajectory of the display device; as well as By means of the at least one processor, if the display device moves from a position other than the third position and the fourth position back to the third position, the display device is controlled to display the virtual display screen at the sixth target point in the virtual environment.

10. A display system, characterized in that: include: a display device for displaying a virtual display screen in a virtual environment; as well as At least one processor is coupled to the display device and is configured to: identifying a plurality of features of a person from a video to be played by the virtual display screen in the virtual environment; identifying a first fixed object on an attached object from the video; determining a first target point in the video according to the characteristics of the character; calculating a first positional relationship between the first fixed object and the first target point, wherein the first positional relationship is used to determine the position of the virtual display screen in the virtual environment; determining an anchor point in the virtual environment, identifying a second fixed object in the physical environment, and converting a position of the second fixed object to the anchor point in the virtual environment; as well as A second positional relationship is set between the anchor point and a second target point to control the display device to display the virtual display screen at the second target point in the virtual environment, wherein the second positional relationship corresponds to the first positional relationship, the position of the anchor point corresponds to the position of the first fixed object, and the position of the second target point corresponds to the position of the first target point.

11. The display system according to claim 10, wherein: The at least one processor is used to determine a position of the character in a viewing direction and a tilt angle and a size of the virtual display screen according to the features of the character in the video, thereby establishing a contour.

12. The display system according to claim 11, wherein: The at least one processor is configured to: Setting a reference line according to the features of the character; Determining the tilt angle of the virtual display screen according to the reference line; determining the position of the character in the viewing direction based on a viewing distance; and The size of the virtual display screen is determined based on a field of view and the viewing distance.

13. The display system according to claim 12, wherein: If the features of the person repeatedly move from a first position to a second position, the at least one processor is configured to: If the features of the character are located at the first position, setting a first line according to the features of the character; If the features of the person are located at the first position, setting a second line according to the features of the person, wherein the first line intersects the second line to form an angle; searching for a third line between and intersecting the first line and the second line, wherein the third line bisects the angle between the first line and the second line; and The third line is used as the reference line.

14. The display system according to claim 10, wherein: It further includes a positioning assistance device, wherein the positioning assistance device is used to perform an interactive operation in a physical environment to generate positioning data, wherein the at least one processor is used to identify a second fixed object in the physical environment based on the positioning data, and to convert the position of the second fixed object into the anchor point in the virtual environment.

15. The display system according to claim 10, wherein: If the character changes its posture, the at least one processor is further configured to: determining a third target point in the video according to the characteristics of the character; calculating a third positional relationship between the first fixed object and the third target point; and A fourth positional relationship is set between the anchor point and a fourth target point to control the display device to display the virtual display screen at the fourth target point in the virtual environment, wherein the fourth positional relationship corresponds to the third positional relationship.

16. The display system according to claim 10, wherein: If the features of the character repeatedly move from a first position to a second position and the display device is configured to repeatedly move from a third position to a fourth position according to the movement of the features of the character, the at least one processor is further configured to: If the features of the person are located at the first position, determining the first target point in the video; If the features of the person are located at the second position, determining a fifth target point in the video; calculating a fifth positional relationship between the first fixed object and the fifth target point; If the display device is located at the third position, controlling the display device to display the virtual display screen at the second target point in the virtual environment; If the display device is not located at the third position or the fourth position, controlling the display device to display the virtual display screen by following an upward movement trajectory of the display device and maintaining a predetermined distance from the user; as well as If the display device is located at the fourth position, a seventh position relationship is set between the anchor point and a seventh target point to control the display device to display the virtual display screen at the seventh target point in the virtual environment, wherein the seventh position relationship corresponds to the fifth position relationship.

17. The display system according to claim 16, wherein: The at least one processor is further configured to: If the virtual display screen is displayed by following the upward movement trajectory of the display device, determining a sixth target point in the virtual environment; as well as If the display device moves back to the third position from a position other than the third position and the fourth position, the display device is controlled to display the virtual display screen at the sixth target point in the virtual environment.

18. A non-transitory computer-readable storage medium, characterized in that: A computer program is provided for executing a display method, wherein the display method is used to display a virtual display screen in a virtual environment and includes: Recognizing, by at least one processor, a plurality of features of a person from a video to be played by the virtual display screen in the virtual environment; identifying, by the at least one processor, a first fixed object on an attached object from the video; Determining, by the at least one processor, a first target point in the video according to the features of the person; calculating, by the at least one processor, a first positional relationship between the first fixed object and the first target point, wherein the first positional relationship is used to determine a position of the virtual display screen in the virtual environment; Determining, by the at least one processor, an anchor point in the virtual environment, the at least one processor being configured to identify a second fixed object in the physical environment and convert a position of the second fixed object to the anchor point in the virtual environment; and By means of the at least one processor, a second positional relationship is set between the anchor point and a second target point to control a display device to display the virtual display screen at the second target point in the virtual environment, wherein the second positional relationship corresponds to the first positional relationship, the position of the anchor point corresponds to the position of the first fixed object, and the position of the second target point corresponds to the position of the first target point.

Images