Multi-screen display audio playing method and device, head-mounted display device and storage medium

Abstract

The application discloses a multi-screen display audio playing method and device, a head-mounted display device and a computer readable storage medium. The method comprises the following steps: when the head-mounted display device performs multi-screen display, a physical target point of the lens is determined, and a corresponding target ray is determined according to the physical target point; a target virtual screen intersecting with the target ray displayed in the head-mounted display device is determined; target audio of the target virtual screen is played, and interference audio of a non-target virtual screen displayed in the head-mounted display device is shielded. The multi-screen display audio playing method in the application is applied to the head-mounted display device, so that when the head-mounted display device performs multi-screen display, the simultaneous playing audio can not interfere with each other, and the viewing effect of the user and the user experience of the head-mounted display device are improved.

Description

Technical Field

[0001] This invention relates to the field of smart head-mounted display technology, and in particular to a multi-screen display audio playback method, apparatus, head-mounted display device, and computer-readable storage medium. Background Technology

[0002] In recent years, smart head-mounted display devices, including AR (Augmented Reality) glasses and VR (Virtual Reality) glasses, have gained increasing popularity and adoption among users. Many smart head-mounted display devices now support multi-screen display functionality, meaning users can view multiple simultaneously displayed virtual screens on the same device. Users can interact with these virtual screens by turning their heads or turning their bodies while wearing the device, thus determining which screen's content to view.

[0003] With current technology, users can create multiple virtual screens on smart head-mounted display devices, but they can only choose one to watch. When multiple virtual screens are displayed and playing simultaneously, users often hear audio from multiple screens, causing audio interference and severely affecting the viewing experience, resulting in a poor user experience. Summary of the Invention

[0004] The main objective of this invention is to provide a multi-screen audio playback method, apparatus, head-mounted display device, and computer-readable storage medium, aiming to solve the technical problem that multiple audio streams interfere with each other when a head-mounted display device displays multiple screens, resulting in a poor viewing experience for the user.

[0005] To achieve the above objectives, the present invention provides a multi-screen display audio playback method, which is applied to a head-mounted display device, the head-mounted display device including lenses; the method includes the following steps:

[0006] When the head-mounted display device performs multi-screen display, the physical target point of the lens is determined, and the corresponding target ray is determined based on the physical target point;

[0007] Identify the target virtual screen displayed in the head-mounted display device that intersects with the target ray;

[0008] Play the target audio of the target virtual screen and block out the interfering audio of the non-target virtual screen displayed on the head-mounted display device.

[0009] Optionally, the step of determining the corresponding target ray based on the physical target point includes:

[0010] Using the physical target point as the endpoint, the target ray is determined perpendicular to the mirror surface of the lens and according to the line of sight of the user wearing the head-mounted display device.

[0011] Optionally, the step of determining the target virtual screen displayed in the head-mounted display device that intersects with the target ray includes:

[0012] Determine the planar position information of each virtual screen displayed in the head-mounted display device; wherein the planar position information represents a set of information consisting of any corner point of the virtual screen and the long side vector and the wide side vector with the corner point as the endpoint;

[0013] Based on the planar position information and the target ray, determine the target virtual screen that intersects with the target ray.

[0014] Optionally, the step of determining the target virtual screen intersecting with the target ray based on the planar position information and the target ray includes:

[0015] Based on the unit plane normal vector corresponding to the plane position information, the ray unit vector of the corner point and the target ray, and the object target point, obtain each unconfirmed intersection point when the target ray intersects with each of the virtual screens;

[0016] Based on the line segment formed by the intersection of the corner point and the point to be confirmed, determine the long side projection of the line segment along the corresponding long side vector and the wide side projection of the corresponding wide side vector.

[0017] The actual intersection point among the various intersection points to be confirmed is determined based on the first length value of the long side projection and the second length value of the wide side projection.

[0018] The virtual screen corresponding to the actual intersection point is determined to be the target virtual screen that intersects with the target ray.

[0019] Optionally, the step of determining the actual intersection point among the various intersection points to be confirmed based on the first length value of the projection of the longer side and the second length value of the projection of the wider side includes:

[0020] Determine whether the first length value of the projection of the long side is less than or equal to the third length value corresponding to the long side vector;

[0021] If the first length value is less than or equal to the third length value, then determine whether the second length value of the wide side projection is less than or equal to the fourth length value corresponding to the wide side vector;

[0022] If the second length value is less than or equal to the fourth length value, then the intersection point to be confirmed corresponding to both the long side projection and the wide side projection is determined to be the actual intersection point among the various intersection points to be confirmed.

[0023] Optionally, before the step of obtaining the points to be confirmed when the target ray intersects with each of the virtual screens based on the unit plane normal vector corresponding to the planar position information, the ray unit vector of the corner point and the target ray, and the object target point, the method further includes:

[0024] Determine whether the angle formed between the unit plane normal vector corresponding to the plane position information and the ray unit vector of the target ray is zero;

[0025] If it is not zero, then determine whether the angle is an obtuse angle;

[0026] If it is an obtuse angle, then the step of obtaining the points to be confirmed when the target ray intersects with each of the virtual screens is performed based on the unit plane normal vector corresponding to the plane position information, the corner point and the ray unit vector of the target ray, and the object target point.

[0027] Optionally, after the step of playing the target audio of the target virtual screen and blocking interfering audio from non-target virtual screens displayed in the head-mounted display device, the method further includes:

[0028] Obtain the playback duration of the target audio on the target virtual screen;

[0029] If the playback duration exceeds the preset duration, then each of the non-target virtual screens is controlled to enter a standby state.

[0030] Furthermore, to achieve the above objectives, the present invention also provides a multi-screen display audio playback device, the multi-screen display audio playback device comprising:

[0031] The ray generation module is used to determine the physical target point of the lens when the head-mounted display device performs multi-screen display, and to determine the corresponding target ray based on the physical target point;

[0032] A gaze-capturing module is used to determine the target virtual screen displayed in the head-mounted display device that intersects with the target ray;

[0033] The audio management module is used to play the target audio of the target virtual screen and block the interference audio of the non-target virtual screen displayed in the head-mounted display device.

[0034] In addition, to achieve the above objectives, the present invention also provides a head-mounted display device, including a processor, a storage unit, and a multi-screen display audio playback program stored in the storage unit and executable by the processor, wherein when the multi-screen display audio playback program is executed by the processor, it implements the steps of the multi-screen display audio playback method as described above.

[0035] The present invention also provides a computer-readable storage medium storing a multi-screen display audio playback program, wherein when the multi-screen display audio playback program is executed by a processor, it implements the steps of the multi-screen display audio playback method described above.

[0036] The multi-screen audio playback method in this invention determines the physical target point of the lens when the head-mounted display device is displaying multiple screens. Based on this physical target point, a corresponding target ray is determined, thus obtaining a target ray that aligns with the simulated user's line of sight and the user's line of sight. By determining the target virtual screen displayed on the head-mounted display device that intersects with the target ray, the virtual screen that the user is actually focused on during multi-screen use is identified. Furthermore, by playing the target audio of the target virtual screen and blocking interfering audio from non-target virtual screens displayed on the head-mounted display device, the head-mounted display device only plays the target audio of the target virtual screen that the user is actually focused on during multi-screen display, while blocking other audio. This ensures that the audio played simultaneously does not interfere with each other, meets the user's actual viewing needs, and improves the user's viewing experience and the overall user experience of the head-mounted display device. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the terminal structure of the hardware operating environment of the head-mounted display device involved in the embodiments of the present invention;

[0038] Figure 2 This is a flowchart illustrating the first embodiment of the multi-screen audio playback method of the present invention;

[0039] Figure 3 This is a detailed flowchart of step S20 in an embodiment of the multi-screen display audio playback method of the present invention;

[0040] Figure 4 This is a detailed flowchart illustrating step S22 of an embodiment of the multi-screen display audio playback method of the present invention;

[0041] Figure 5 This is a flowchart illustrating the process before step S220 in the first embodiment of the multi-screen display audio playback method of the present invention;

[0042] Figure 6This is a flowchart illustrating the process after step S30 in the first embodiment of the multi-screen display audio playback method of the present invention;

[0043] Figure 7 This is a schematic diagram of the multi-screen display involved in the multi-screen audio playback method of the present invention;

[0044] Figure 8 This is a schematic diagram of the frame structure of the multi-screen display audio playback device of the present invention.

[0045] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0046] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0047] This invention provides a head-mounted display device. The head-mounted display device can be a Mixed Reality (MR) device (e.g., MR glasses or MR helmet), Augmented Reality (AR) device (e.g., AR glasses or AR helmet), Virtual Reality (VR) device (e.g., VR glasses or VR helmet), Extended Reality (XR) device, or some combination thereof, etc., and is not limited thereto.

[0048] like Figure 1 As shown, Figure 1 This is a schematic diagram of the hardware operating environment of the head-mounted display device involved in the embodiments of the present invention.

[0049] like Figure 1 As shown, the head-mounted display device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a storage unit 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display or an input unit such as a control panel; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The storage unit 1005 may be a high-speed RAM storage unit or a stable storage unit (non-volatile memory), such as a disk storage unit. The storage unit 1005 may also optionally be a storage device independent of the aforementioned processor 1001. As a computer storage medium, the storage unit 1005 may include a multi-screen audio playback program.

[0050] Those skilled in the art will understand that Figure 1 The hardware structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0051] Continue to refer to Figure 1 , Figure 1 The storage unit 1005, which is a computer-readable storage medium, may include an operating system, a user interface module, a network communication module, and a multi-screen display audio playback program.

[0052] exist Figure 1 In this module, the network communication module is mainly used to connect to the server and communicate with it for data; while the processor 1001 can call the multi-screen display audio playback program stored in the storage unit 1005 and perform the following operations:

[0053] When the head-mounted display device performs multi-screen display, the physical target point of the lens is determined, and the corresponding target ray is determined based on the physical target point;

[0054] Identify the target virtual screen displayed in the head-mounted display device that intersects with the target ray;

[0055] Play the target audio of the target virtual screen and block out the interfering audio of the non-target virtual screen displayed on the head-mounted display device.

[0056] Furthermore, the processor 1001 can call the multi-screen display audio playback program stored in the memory 1005 and also perform the following operations:

[0057] Using the physical target point as the endpoint, the target ray is determined perpendicular to the mirror surface of the lens and according to the line of sight of the user wearing the head-mounted display device.

[0058] Furthermore, the processor 1001 can call the multi-screen display audio playback program stored in the memory 1005 and also perform the following operations:

[0059] Determine the planar position information of each virtual screen displayed in the head-mounted display device; wherein the planar position information represents a set of information consisting of any corner point of the virtual screen and the long side vector and the wide side vector with the corner point as the endpoint;

[0060] Based on the planar position information and the target ray, determine the target virtual screen that intersects with the target ray.

[0061] Furthermore, the processor 1001 can call the multi-screen display audio playback program stored in the memory 1005 and also perform the following operations:

[0062] Based on the unit plane normal vector corresponding to the plane position information, the ray unit vector of the corner point and the target ray, and the object target point, obtain each unconfirmed intersection point when the target ray intersects with each of the virtual screens;

[0063] Based on the line segment formed by the intersection of the corner point and the point to be confirmed, determine the long side projection of the line segment along the corresponding long side vector and the wide side projection of the corresponding wide side vector.

[0064] The actual intersection point among the various intersection points to be confirmed is determined based on the first length value of the long side projection and the second length value of the wide side projection.

[0065] The virtual screen corresponding to the actual intersection point is determined to be the target virtual screen that intersects with the target ray.

[0066] Furthermore, the processor 1001 can call the multi-screen display audio playback program stored in the memory 1005 and also perform the following operations:

[0067] Determine whether the first length value of the projection of the long side is less than or equal to the third length value corresponding to the long side vector;

[0068] If the first length value is less than or equal to the third length value, then determine whether the second length value of the wide side projection is less than or equal to the fourth length value corresponding to the wide side vector;

[0069] If the second length value is less than or equal to the fourth length value, then the intersection point to be confirmed corresponding to both the long side projection and the wide side projection is determined to be the actual intersection point among the various intersection points to be confirmed.

[0070] Furthermore, the processor 1001 can call the multi-screen display audio playback program stored in the memory 1005 and also perform the following operations:

[0071] Determine whether the angle formed between the unit plane normal vector corresponding to the plane position information and the ray unit vector of the target ray is zero;

[0072] If it is not zero, then determine whether the angle is an obtuse angle;

[0073] If it is an obtuse angle, then the step of obtaining the points to be confirmed when the target ray intersects with each of the virtual screens is performed based on the unit plane normal vector corresponding to the plane position information, the corner point and the ray unit vector of the target ray, and the object target point.

[0074] Furthermore, the processor 1001 can call the multi-screen display audio playback program stored in the memory 1005 and also perform the following operations:

[0075] Obtain the playback duration of the target audio on the target virtual screen;

[0076] If the playback duration exceeds the preset duration, then each of the non-target virtual screens is controlled to enter a standby state.

[0077] Based on the hardware structure of the controller described above, various embodiments of the multi-screen display audio playback method of the present invention are proposed.

[0078] To facilitate understanding of the various embodiments of the present invention, the embodiments of the technical solution of the present invention are briefly summarized in terms of the overall solution:

[0079] This invention primarily simulates the user's actual pupillary line of sight using a virtual target ray. By identifying the target virtual screen among the various virtual screens displayed on the head-mounted display device that intersects with the target ray, it determines the virtual screen the user is currently focusing on. Consequently, it only plays the audio provided by the target virtual screen the user is focusing on, preventing simultaneous playback of audio from various virtual screens from causing auditory interference to the user, thereby improving the user's viewing experience.

[0080] This invention provides a method for multi-screen audio playback.

[0081] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of the multi-screen display audio playback method of the present invention. In the first embodiment, the multi-screen display audio playback method is applied to a head-mounted display device, and the method includes the following steps:

[0082] Step S10: When the head-mounted display device performs multi-screen display, determine the physical target point of the lens, and determine the corresponding target ray based on the physical target point;

[0083] In this embodiment, the head-mounted display device, as mentioned above, can be an AR device, VR device, MR device, XR device, or other intelligent head-mounted display device, without limitation. The head-mounted display device is equipped with eyeglass lenses and has the function of enabling users to watch movies through the eyeglass lenses and related projection technology. Here, user watching movies refers to the user's ability to meet their needs for visual entertainment, work, and life through the head-mounted display device, including watching movies, TV programs, live broadcasts, games, shopping, etc.

[0084] When the head-mounted display device performs multi-screen display, the physical target point of the glasses lens of the head-mounted display device can be periodically determined according to a preset acquisition cycle. The acquisition cycle can be set according to actual needs and is not limited here. The physical target point can correspond to the position of the user's pupil, that is, the physical target point is directly in front of the pupil position. The physical target point can be the physical center point of the lens surface. It should be noted that any point within the preset range of the strictly defined physical center point can be used as the physical center point in this embodiment. For example, the preset range can be obtained by taking the strictly defined physical center point as the center and according to a preset radius. The preset radius can be (0-10mm) and is not limited here. Considering that the user's pupil has a certain downward tendency when focusing on objects, that is, when the user's eyes are looking at an object, the pupil's line of sight is often not directly on the object, but at a certain position below the object. Therefore, the position of the physical target point can also be directly below the physical center point of the lens surface at a preset distance. The preset distance can be set according to actual needs, such as 0.5mm, 1mm, etc., and is not limited here.

[0085] After determining the physical target point, the physical target point is used as the endpoint of the target ray to be generated, thereby simulating the user's actual pupil line of sight to obtain a virtual target ray that is basically consistent with the user's pupil line of sight.

[0086] Step S20: Determine the target virtual screen displayed in the head-mounted display device that intersects with the target ray;

[0087] In head-mounted display devices, one or more virtual screens can be created based on user needs and the device's actual performance. Please refer to [reference needed]. Figure 7 , Figure 7 This is a schematic diagram illustrating the multi-screen display involved in the multi-screen audio playback method of the present invention. For example... Figure 7 As shown, the head-mounted display device creates four virtual screens, including screen 1, screen 2, screen 3, and screen 4, which surround the user. Of course, this... Figure 6 This is just an example to aid understanding. In actual use cases, it is not limited to four virtual screens, nor is it limited to the location of each virtual screen.

[0088] When a user watches a movie while wearing a head-mounted display device, what the user sees are the various display areas provided by the head-mounted display device, i.e., various virtual screens. Since multiple virtual screens are playing simultaneously, and the user's attention is limited, the user only needs to focus on the content playing on one virtual screen at a time. Therefore, it is necessary to determine the virtual screen that the user is actually focusing on, i.e., the target virtual screen. In this embodiment, this can be determined by identifying the target virtual screen among the various virtual screens displayed in the head-mounted display device that intersects with the target ray, which is actually the target virtual screen that intersects with the user's pupil line of sight. This determines the virtual screen that the user is actually focusing on, and other virtual screens that the user is not focusing on are considered non-target virtual screens.

[0089] Step S30: Play the target audio of the target virtual screen and block the interference audio of the non-target virtual screen displayed in the head-mounted display device.

[0090] After identifying the target virtual screen and non-target virtual screens among the various virtual screens in the head-mounted display device, only the target audio provided by the target virtual screen can be played while the interfering audio from the non-target virtual screens is blocked. Thus, through the above-mentioned audio management, the user's actual viewing needs are met, and the viewing experience is not disturbed by multiple audio streams playing simultaneously.

[0091] To be more specific, for interference audio from non-target virtual screens, the output of interference audio from non-target virtual screens can be directly stopped, or the output of interference audio can be left uninterrupted, but the interference audio can be stopped by not transmitting it to the speaker, thus stopping the playback of interference audio. For target audio from the target virtual screen, it is transmitted to the speaker for normal playback.

[0092] The multi-screen audio playback method in this invention determines the physical target point of the lens when the head-mounted display device is displaying multiple screens. Based on this physical target point, a corresponding target ray is determined, thus obtaining a target ray that aligns with the simulated user's line of sight and the user's line of sight. By determining the target virtual screen displayed on the head-mounted display device that intersects with the target ray, the virtual screen that the user is actually focused on during multi-screen use is identified. Furthermore, by playing the target audio of the target virtual screen and blocking interfering audio from non-target virtual screens displayed on the head-mounted display device, the head-mounted display device only plays the target audio of the target virtual screen that the user is actually focused on during multi-screen display, while blocking other audio. This ensures that the audio played simultaneously does not interfere with each other, meets the user's actual viewing needs, and improves the user's viewing experience and the overall user experience of the head-mounted display device.

[0093] Based on the first embodiment described above, in one embodiment, step S10, the step of determining the corresponding target ray based on the physical target point, includes:

[0094] Step a: Using the physical target point as the endpoint, perpendicular to the mirror surface of the lens, and based on the line of sight of the user wearing the head-mounted display device, determine the corresponding target ray.

[0095] In this embodiment, the physical target point is used as the endpoint of the target ray. This target ray is perpendicular to the lens surface, and its direction is consistent with the user's line of sight. That is, with the user as the standard, the direction of the target ray is outward from the user. In this way, a target ray that is basically consistent with the user's actual pupillary line of sight is obtained. By reflecting the user's pupillary line of sight through the target ray, it is possible to determine the user's focus when watching movies using the head-mounted display device.

[0096] Based on the above embodiments, please refer to Figure 3 In one embodiment, step S20 includes:

[0097] Step S21: Determine the planar position information of each virtual screen displayed in the head-mounted display device; wherein, the planar position information represents an information set composed of any corner point of the virtual screen and the long side vector and the wide side vector with the corner point as the endpoint;

[0098] In this embodiment, the planar position information of each virtual screen displayed in the head-mounted display device refers to the position of each virtual screen in the virtual space provided by the head-mounted display device. Since the virtual screen is planar, it is planar position information, which can be referred to as... Figure 6 You can see various virtual screens surrounding the user, positioned in different planar locations.

[0099] Specifically, to accurately represent the location of each virtual screen, we can define a corner point P0 (a rectangular corner point) of the virtual screen and the vectors corresponding to the two sides intersecting that corner point: the long side vector S1 and the wide side vector S2. That is, the long side vector corresponding to the length of the virtual screen and the wide side vector corresponding to its width. Both the long side vector S1 and the wide side vector S2 have the corner point P0 as their endpoint (starting point). Each virtual screen can be represented by a corner point and its corresponding long and wide side vectors, thus defining its display area. By determining the coordinates of the corner point, we can also determine the planar position information of each virtual screen in the virtual space.

[0100] Step S22: Based on the planar position information and the target ray, determine the target virtual screen that intersects with the target ray.

[0101] If a target ray intersects with one of the virtual screens, then there must be a point on the target ray that coincides with the planar position information of the intersecting virtual screen. In other words, this point on the target ray is within the planar position information of the target virtual screen, meaning that the target ray and the target virtual screen have an intersection point.

[0102] The target virtual screen intersecting with the target ray can be determined by determining the planar position information of which virtual screen the intersection point of the target ray falls into, and then only the audio of the target virtual screen is played.

[0103] Based on the above embodiments, please refer to Figure 4 In one embodiment, step S22 includes:

[0104] Step S220: Based on the unit plane normal vector corresponding to the plane position information, the ray unit vector of the corner point and the target ray, and the object target point, obtain the intersection points to be confirmed when the target ray intersects with each of the virtual screens respectively;

[0105] The unit plane normal vector corresponding to the planar position information is the unit plane normal vector N perpendicular to its virtual screen surface. This vector N is equal to the unit vector of the cross product between the long side vector S1 and the wide side vector S2. The ray unit vector D of the target ray represents the unit vector along the target ray direction. The target point of the object can be represented as R0, which is also the endpoint R0 of the target ray. Thus, the target ray can be represented as R0 + t * D, where t represents the vector length of the target ray.

[0106] Since it is not yet known which virtual screen the target ray intersects with, we can assume that the target ray intersects with each virtual screen individually. Taking one virtual screen as an example, if the target ray intersects with that virtual screen, the point of intersection to be confirmed (at this time, it cannot be determined whether the point of intersection to be confirmed is the actual point of intersection) is the intersection point P. Since the intersection point P is on the target ray, P can be expressed as P = R0 + a * D, where a is the vector length from the endpoint R0 to the intersection point P. Since it is assumed that the target ray intersects with the virtual screen, the vector P0P, that is, the vector from the corner point P0 to the intersection point P, must be perpendicular to the normal vector N of the unit plane, which can be expressed as P0P·N = 0. After mathematical transformation, we can get a = ((P0-R0)·N) / (D·N). Therefore, we first determine the difference between the corner point P0 and the endpoint R0, then determine the product of the difference and the unit plane normal vector N, divide the product by the vector product between the ray unit vector D and the unit plane normal vector N to obtain a, and then obtain the various intersection points to be confirmed when the target ray intersects with each of the virtual screens according to the above calculation method.

[0107] Step S221: Based on the line segment formed by the intersection of the corner point and the point to be confirmed, determine the long side projection of the line segment along the corresponding long side vector and the wide side projection of the corresponding wide side vector.

[0108] The line segment formed by the intersection of the corner point and the point to be confirmed is also the vector P0P. In order to determine whether the intersection point P to be confirmed is located within the corresponding virtual screen, the long side projection Q1 of P0P is captured along the long side vector S1 and the wide side projection Q2 of P0P is captured along S2, so as to obtain the length values ​​of the long side projection Q1 and the wide side projection Q2 respectively.

[0109] Step S222: Determine the actual intersection point among the various intersection points to be confirmed based on the first length value of the long side projection and the second length value of the wide side projection;

[0110] The determination of the target virtual screen can be achieved by judging whether the first length value of the long side projection of each virtual screen meets a first preset condition and whether the second length value of the corresponding wide side projection meets a second preset condition. If both the first and second length values ​​of a virtual screen simultaneously meet the first and second preset conditions, the intersection point to be confirmed for that virtual screen can be determined as the actual intersection point with the target ray, and that virtual screen is the target virtual screen. If neither the first nor the second preset condition is met simultaneously, including meeting one condition or not meeting any conditions, then the corresponding virtual screen is a non-target virtual screen. Of course, if the target virtual screen has already been determined, the other virtual screens are naturally non-target virtual screens, and there is no need to judge the non-target virtual screens according to the above conditions, thereby saving the computing resources of the head-mounted display device and improving the efficiency of determining the target virtual screen. It should be noted that the first preset condition can be that the first length value of the long side projection is less than or equal to the third length value corresponding to the long side vector, and the second preset condition can be that the second length value of the wide side projection is less than or equal to the fourth length value corresponding to the wide side vector. The first and second preset conditions can also be other preset conditions, which are not limited here.

[0111] Step S223: Determine that the virtual screen corresponding to the actual intersection point is the target virtual screen that intersects with the target ray.

[0112] Once the actual intersection point is determined, the virtual screen where the actual intersection point is located is also determined. This virtual screen is the target virtual screen that intersects with the target ray. This embodiment applies simple mathematical and geometric principles to finding and determining the target virtual screen, which can accurately and efficiently identify the target virtual screen that the user is focusing on, thereby enabling timely audio playback management and conveniently improving the user's viewing experience.

[0113] Based on the above embodiments, in one embodiment, step S222 includes:

[0114] Step b: Determine whether the first length value of the projection of the long side is less than or equal to the third length value corresponding to the long side vector;

[0115] Step c: If the first length value is less than or equal to the third length value, then determine whether the second length value of the wide side projection is less than or equal to the fourth length value corresponding to the wide side vector;

[0116] Step d: If the second length value is less than or equal to the fourth length value, then the intersection point to be confirmed corresponding to both the long side projection and the wide side projection is determined to be the actual intersection point among the various intersection points to be confirmed.

[0117] First, the first length value (Q1) of the long side projection can be compared with the third length value (S1) corresponding to the long side vector S1. When the first length value is less than or equal to the third length, that is, 0 <= length (Q1) <= length (S1), it can be further determined whether the second length value (Q2) of the wide side projection is less than or equal to the fourth length value (S2) corresponding to the wide side vector. When the second length value is less than or equal to the fourth length value, that is, 0 <= length (Q2) <= length (S2), it can be determined that the intersection point P to be confirmed is in the planar position information of the virtual screen corresponding to both the long side projection and the wide side projection. Thus, the intersection point to be confirmed corresponding to the virtual screen is accurately and reliably determined to be the actual intersection point among the various intersection points to be confirmed, and the virtual screen is the target virtual screen.

[0118] Furthermore, if the first length value is greater than the third length value, the corresponding virtual screen is determined to be a non-target virtual screen and can be marked. This non-target virtual screen does not need to undergo further determination. If the second length value is greater than the fourth length value, the corresponding virtual screen is determined to be a non-target virtual screen and can be marked. If, during the above determination process for each virtual screen, one virtual screen is determined to be the target virtual screen, the other virtual screens are naturally non-target virtual screens. The determination process for the other virtual screens can be terminated, thereby saving computing resources for the head-mounted display device.

[0119] Based on the above embodiments, please refer to Figure 5 In one embodiment, prior to step S220, the method further includes:

[0120] Step S210: Determine whether the angle formed between the unit plane normal vector corresponding to the plane position information and the ray unit vector of the target ray is zero;

[0121] Step S211: If the value is not zero, determine whether the angle is an obtuse angle;

[0122] Step S212: If the angle is obtuse, then execute the step of obtaining the points to be confirmed when the target ray intersects with each of the virtual screens based on the unit plane normal vector corresponding to the plane position information, the corner point and the ray unit vector of the target ray, and the object target point.

[0123] Before obtaining all the intersection points to be confirmed, some virtual screens that are unlikely to intersect with the target ray can be eliminated, thereby improving the efficiency of determining the target virtual screen and saving computing resources of the head-mounted display device.

[0124] First, it can be determined whether the angle formed between the unit plane normal vector N corresponding to the plane position information and the ray unit vector D of the target ray is zero, that is, the case where the unit plane normal vector N and the ray unit vector D are parallel. If the angle is zero, then the corresponding virtual screen can be excluded and marked as a non-target virtual screen, and the following judgment process will not be carried out.

[0125] If the angle is not zero, it can be further determined whether the angle is obtuse. Specifically, it can be determined whether the vector product D·N < 0 between the unit plane normal vector N and the ray unit vector D satisfies this condition, thus determining whether the angle is obtuse. If D·N > 0, that is, the angle is acute, it means that the corresponding virtual screen does not intersect with the target ray, and it can be marked as a non-target virtual screen. In the case of an obtuse angle, in order to accurately determine the target virtual screen, the steps of obtaining the intersecting points of the target ray with each virtual screen if they intersect, based on the unit plane normal vector corresponding to the plane position information, the corner point and the ray unit vector of the target ray, and the object target point, can be performed on other virtual screens that have not been marked as non-target virtual screens. This allows for accurate and efficient determination of the target virtual screen that the user is currently focusing on.

[0126] Based on the above embodiments, please refer to Figure 6 In one embodiment, after step S30, the method further includes:

[0127] Step S40: Obtain the playback duration of the target audio on the target virtual screen;

[0128] Step S50: If the playback duration is longer than the preset duration, then control each of the non-target virtual screens to enter a standby state.

[0129] In this embodiment, during the playback of the target audio on the target virtual screen, the playback duration of the target audio can be recorded simultaneously. This playback duration is compared with a preset duration (which can be set according to actual needs). When the playback duration is greater than or equal to the preset duration, it indicates that the user has been watching the video on the target virtual screen for a relatively long time. To save energy and improve the battery life of the head-mounted display device, other non-target virtual screens can be controlled to enter a standby state. In this standby state, a screen saver can be displayed, or only a prompt text, such as "Standby," can be displayed. After other non-target virtual screens enter the standby state, if the user wants to use another non-target virtual screen, when the user turns their head or turns so that the target ray intersects with the corresponding non-target virtual screen, the non-target virtual screen becomes the target virtual screen, thus exiting the standby state and entering the normal display state. In this embodiment, even if each of the non-target virtual screens enters the standby state, when the user needs to use the corresponding non-target virtual screen, the system can still quickly respond to the user's relevant operations based on the standby state, ensuring the user's multi-screen display experience and truly allowing them to feel the convenience and fun of multi-screen display.

[0130] In addition, refer to Figure 7 , Figure 7 This is a schematic diagram of the frame structure of the multi-screen audio playback device of the present invention. The present invention also proposes a multi-screen audio playback device, which includes:

[0131] The ray generation module A10 is used to determine the physical target point of the lens when the head-mounted display device performs multi-screen display, and to determine the corresponding target ray based on the physical target point;

[0132] The gaze-capture module A20 is used to determine the target virtual screen displayed in the head-mounted display device that intersects with the target ray;

[0133] The audio management module A30 is used to play the target audio of the target virtual screen and block the interference audio of the non-target virtual screen displayed in the head-mounted display device.

[0134] Optionally, the ray generation module A10 is further configured to:

[0135] Using the physical target point as the endpoint, the target ray is determined perpendicular to the mirror surface of the lens and according to the line of sight of the user wearing the head-mounted display device.

[0136] Optionally, the gaze-capturing module A20 is further configured to:

[0137] Determine the planar position information of each virtual screen displayed in the head-mounted display device; wherein the planar position information represents a set of information consisting of any corner point of the virtual screen and the long side vector and the wide side vector with the corner point as the endpoint;

[0138] Based on the planar position information and the target ray, determine the target virtual screen that intersects with the target ray.

[0139] Optionally, the gaze-capturing module A20 is further configured to:

[0140] Based on the unit plane normal vector corresponding to the plane position information, the ray unit vector of the corner point and the target ray, and the object target point, obtain each unconfirmed intersection point when the target ray intersects with each of the virtual screens;

[0141] Based on the line segment formed by the intersection of the corner point and the point to be confirmed, determine the long side projection of the line segment along the corresponding long side vector and the wide side projection of the corresponding wide side vector.

[0142] The actual intersection point among the various intersection points to be confirmed is determined based on the first length value of the long side projection and the second length value of the wide side projection.

[0143] The virtual screen corresponding to the actual intersection point is determined to be the target virtual screen that intersects with the target ray.

[0144] Optionally, the gaze-capturing module A20 is further configured to:

[0145] Determine whether the first length value of the projection of the long side is less than or equal to the third length value corresponding to the long side vector;

[0146] If the first length value is less than or equal to the third length value, then determine whether the second length value of the wide side projection is less than or equal to the fourth length value corresponding to the wide side vector;

[0147] If the second length value is less than or equal to the fourth length value, then the intersection point to be confirmed corresponding to both the long side projection and the wide side projection is determined to be the actual intersection point among the various intersection points to be confirmed.

[0148] Optionally, the gaze-capturing module A20 is further configured to:

[0149] Determine whether the angle formed between the unit plane normal vector corresponding to the plane position information and the ray unit vector of the target ray is zero;

[0150] If it is not zero, then determine whether the angle is an obtuse angle;

[0151] If it is an obtuse angle, then the step of obtaining the points to be confirmed when the target ray intersects with each of the virtual screens is performed based on the unit plane normal vector corresponding to the plane position information, the corner point and the ray unit vector of the target ray, and the object target point.

[0152] Optionally, the audio management module A30 is further configured to:

[0153] Obtain the playback duration of the target audio on the target virtual screen;

[0154] If the playback duration exceeds the preset duration, then each of the non-target virtual screens is controlled to enter a standby state.

[0155] The specific implementation of the multi-screen display audio playback device of the present invention is basically the same as the embodiments of the multi-screen display audio playback method described above, and will not be repeated here.

[0156] Furthermore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium of the present invention stores a multi-screen display audio playback program, wherein, when executed by a processor, the multi-screen display audio playback program implements the steps of the multi-screen display audio playback method described above.

[0157] The method implemented when the multi-screen audio playback program is executed can be referred to in various embodiments of the multi-screen audio playback method of the present invention, and will not be repeated here.

[0158] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage unit that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage unit produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0163] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0164] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for playing audio on multiple screens, characterized in that, The multi-screen display audio playback method is applied to a head-mounted display device, the head-mounted display device including lenses; the method includes the following steps: When the head-mounted display device performs multi-screen display, the physical target point of the lens is determined, and the corresponding target ray is determined based on the physical target point. The head-mounted display device displays multiple virtual screens surrounding the user. Determine the planar position information of each virtual screen displayed in the head-mounted display device; wherein the planar position information represents a set of information consisting of any corner point of the virtual screen and the long side vector and the wide side vector with the corner point as the endpoint; Determine whether the angle formed between the unit plane normal vector corresponding to the plane position information and the ray unit vector of the target ray is zero; If it is not zero, then determine whether the angle is an obtuse angle; If it is an obtuse angle, then based on the planar position information and the target ray, determine the target virtual screen that intersects with the target ray; Play the target audio of the target virtual screen and block out the interfering audio of the non-target virtual screen displayed on the head-mounted display device.

2. The multi-screen display audio playback method as described in claim 1, characterized in that, The step of determining the corresponding target ray based on the physical target point includes: Using the physical target point as the endpoint, the target ray is determined perpendicular to the mirror surface of the lens and according to the line of sight of the user wearing the head-mounted display device.

3. The multi-screen audio playback method as described in claim 1, characterized in that, The step of determining the target virtual screen intersecting with the target ray based on the planar position information and the target ray includes: Based on the unit plane normal vector corresponding to the plane position information, the ray unit vector of the corner point and the target ray, and the physical target point, obtain the various unconfirmed intersection points when the target ray intersects with each of the virtual screens; Based on the line segment formed by the intersection of the corner point and the point to be confirmed, determine the long side projection of the line segment along the corresponding long side vector and the wide side projection of the corresponding wide side vector. The actual intersection point among the various intersection points to be confirmed is determined based on the first length value of the long side projection and the second length value of the wide side projection. The virtual screen corresponding to the actual intersection point is determined to be the target virtual screen that intersects with the target ray.

4. The multi-screen display audio playback method as described in claim 3, characterized in that, The step of determining the actual intersection point among the various intersection points to be confirmed based on the first length value of the long side projection and the second length value of the wide side projection includes: Determine whether the first length value of the projection of the long side is less than or equal to the third length value corresponding to the long side vector; If the first length value is less than or equal to the third length value, then determine whether the second length value of the wide side projection is less than or equal to the fourth length value corresponding to the wide side vector; If the second length value is less than or equal to the fourth length value, then the intersection point to be confirmed corresponding to both the long side projection and the wide side projection is determined to be the actual intersection point among the various intersection points to be confirmed.

5. The multi-screen audio playback method as described in claim 1, characterized in that, After the steps of playing the target audio of the target virtual screen and blocking interfering audio from non-target virtual screens displayed in the head-mounted display device, the method further includes: Obtain the playback duration of the target audio on the target virtual screen; If the playback duration exceeds the preset duration, then each of the non-target virtual screens is controlled to enter a standby state.

6. A multi-screen audio playback device, characterized in that, The multi-screen audio playback device includes: The ray generation module is used to determine the physical target point of the lens when the head-mounted display device performs multi-screen display, and to determine the corresponding target ray based on the physical target point; A gaze-capture module is used to determine the planar position information of each virtual screen displayed in the head-mounted display device; wherein, the planar position information represents an information set composed of any corner point of the virtual screen and the long side vector and the wide side vector with the corner point as the endpoint; it determines whether the angle formed between the unit plane normal vector corresponding to the planar position information and the ray unit vector of the target ray is zero; if it is not zero, it determines whether the angle is an obtuse angle; if it is an obtuse angle, it determines the target virtual screen intersecting with the target ray based on the planar position information and the target ray; The audio management module is used to play the target audio of the target virtual screen and block the interference audio of the non-target virtual screen displayed in the head-mounted display device.

7. A head-mounted display device, characterized in that, The head-mounted display device includes a processor, a storage unit, and a multi-screen display audio playback program stored on the storage unit and executable by the processor, wherein when the multi-screen display audio playback program is executed by the processor, it implements the steps of the multi-screen display audio playback method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-screen display audio playback program, wherein when the multi-screen display audio playback program is executed by a processor, it implements the steps of the multi-screen display audio playback method as described in any one of claims 1 to 5.

Images