Method and device for determining interaction position, equipment and medium

Abstract

The invention discloses an interaction position determination method and device, equipment and a medium, and relates to the technical field of head-mounted equipment. The method is applied to the head-mounted display device and comprises the steps that a first image is displayed, and the first image is synthesized by at least one second image received by the head-mounted display device; according to the initial pose and the target pose of the control device, a first virtual identifier of the control device in the display space of the head-mounted display device is determined, and the control device is used for controlling the first image; according to the first virtual identifier and a third image, determining a first coordinate of an intersection point of the first virtual identifier and the third image and a first included angle between the first virtual identifier and the third image, the third image being located in a display space of the head-mounted display device and being arranged at a preset distance from the first image; and determining a second coordinate of an intersection point of the first virtual identifier and the first image according to the first coordinate, the preset distance and the first included angle. According to the method, the corresponding interaction position of the control equipment on the first image can be determined.

Description

Technical Field

[0001] The present application relates to the technical field of head-mounted devices, and more specifically, to a method for determining an interaction position, an apparatus for determining an interaction position, a head-mounted display device, and a computer-readable storage medium. Background Art

[0002] At present, at least one terminal device can stream at least one image data to the head-mounted display device, so that the head-mounted display device can display the image data of the terminal device. For example, the image data of the foreground and / or background running of multiple smartphones can be streamed to the head-mounted display device, so that the head-mounted display device can display it.

[0003] When displaying the images streamed by the terminal device on the head-mounted display device, if the aforementioned streamed images can be controlled by the control device of the head-mounted display device, the interaction between the head-mounted display device and the terminal device can be achieved.

[0004] In order to realize the interaction between the head mounted display device and the terminal device, it is first necessary to determine the interactive position of the control device of the head mounted display device corresponding to the aforementioned streaming image. Therefore, how to determine the interactive position of the control device of the head mounted display device corresponding to the aforementioned streaming image becomes a technical problem to be solved urgently. Summary of the invention

[0005] One object of the present application is to provide a new technical solution for determining interaction location.

[0006] According to a first aspect of the present application, a method for determining an interaction position is provided, which is applied to a head mounted display device, comprising:

[0007] Displaying a first image, the first image being synthesized from at least one second image received by the head mounted display device;

[0008] Determining a first virtual identifier of the control device in the display space of the head mounted display device according to an initial position and a target position of the control device, wherein the control device is used to control the first image;

[0009] Determine, according to the first virtual marker and the third image, a first coordinate of an intersection of the first virtual marker and the third image and a first angle between the first virtual marker and the third image, wherein the third image is located in a display space of the head mounted display device and is arranged at a preset distance from the first image;

[0010] A second coordinate of an intersection point between the first virtual marker and the first image is determined according to the first coordinate, the preset distance, and the first angle.

[0011] Optionally, determining a second coordinate of an intersection of the first virtual marker and the first image according to the first coordinate, the preset distance, and the first angle includes:

[0012] Determine, according to the preset distance and the first angle, a coordinate offset of a second coordinate of an intersection point of the first virtual marker and the first image compared to the first coordinate;

[0013] A second coordinate of an intersection point between the first virtual marker and the first image is determined according to the coordinate offset and the first coordinate.

[0014] Optionally, before determining the first virtual identifier of the control device in the head mounted display device space according to the initial posture and the target posture of the control device, the method further includes:

[0015] Obtaining an initial position and posture of the control device;

[0016] receiving an interaction instruction for the first image sent by a control device;

[0017] In response to the interaction instruction, a target posture of the control device is determined according to inertial data of the control device.

[0018] Optionally, the method further comprises:

[0019] The target second image pointed to by the first virtual marker is determined according to the second coordinates and the display range of the at least one second image.

[0020] Optionally, the method further comprises:

[0021] According to the first virtual marker and the second coordinates, a second virtual marker including the first virtual marker and having an end point coordinate as the second coordinates is displayed.

[0022] Optionally, before determining the first coordinates of the intersection of the first virtual marker and the third image and determining the first angle between the first virtual marker and the third image according to the first virtual marker and the third image, the method further includes:

[0023] The third image is constructed according to the size of the first image and the preset interval.

[0024] According to a second aspect of the present application, a device for determining an interaction position is provided, which is applied to a head mounted display device, comprising:

[0025] A display module, configured to display a first image, wherein the first image is synthesized from at least one second image received by the head mounted display device;

[0026] A first determination module, configured to determine a first virtual identifier of the control device in the display space of the head mounted display device according to an initial position and a target position of the control device, wherein the control device is configured to control the first image;

[0027] A second determining module, configured to determine, according to the first virtual marker and the third image, a first coordinate of an intersection of the first virtual marker and the third image and a first angle between the first virtual marker and the third image, wherein the third image is located in a display space of the head mounted display device and is arranged at a preset interval with respect to the first image;

[0028] The third determination module is used to determine the second coordinates of the intersection of the first virtual marker and the first image according to the first coordinates, the preset distance and the first angle.

[0029] Optionally, the third determining module includes:

[0030] A first determining unit, configured to determine, according to the preset interval and the first angle, a coordinate offset of a second coordinate of an intersection point between the first virtual marker and the first image compared to the first coordinate;

[0031] The second determining unit is used to determine the second coordinates of the intersection of the first virtual marker and the first image according to the coordinate offset and the first coordinates.

[0032] According to a third aspect of the present application, a head mounted display device is provided, the head mounted display device comprising a device for determining an interaction position as described in any one of the second aspects;

[0033] Alternatively, it comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the interactive position determination method as described in any one of the first aspects.

[0034] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for determining an interaction position according to any one of the first aspects is implemented.

[0035] The embodiment of the present application provides a method for determining an interactive position, which is applied to a head-mounted display device, including: displaying a first image, the first image being synthesized by at least one second image received by the head-mounted display device; determining a first virtual identifier of the control device in the display space of the head-mounted display device according to the initial position and target position of the control device, the control device being used to control the first image; determining a first coordinate of an intersection of the first virtual identifier and the third image and a first angle between the first virtual identifier and the third image according to the first virtual identifier and the third image, the third image being located in the display space of the head-mounted display device and being arranged at a preset interval with the first image; determining a second coordinate of an intersection of the first virtual identifier and the first image according to the first coordinate, the preset interval and the first angle. The method can determine the interactive position of the control device corresponding to the first image. That is, in the case where the second image is an image obtained by streaming, the method can determine the interactive position of the control device corresponding to the streamed image.

[0036] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0038] Figure 1 It is a block diagram of the hardware configuration of a head mounted display device according to a method for determining an interaction position provided in an embodiment of the present application.

[0039] Figure 2 This is a flowchart of a method for determining an interaction position provided according to an embodiment of the present application.

[0040] Figure 3 is a schematic diagram of a first image and a third image provided according to an embodiment of the present application;

[0041] Figure 4a It is a schematic diagram of an X-axis offset calculation principle provided according to an embodiment of the present application;

[0042] Figure 4b It is a schematic diagram of a Y-axis offset calculation principle provided according to an embodiment of the present application;

[0043] Figure 5 is a structural schematic diagram of a device for determining an interaction position provided according to an embodiment of the present application;

[0044] Figure 6 It is a schematic diagram of the hardware structure of a head-mounted display device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0046] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0047] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0048] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0049] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0050] Figure 1 It is a block diagram of the hardware configuration of a head mounted display device according to a method for determining an interaction position provided in an embodiment of the present application.

[0051] The head mounted display device 1000 may be any one of an AR head mounted display device, an MR head mounted display device, a VR head mounted display device, and an XR head mounted display device. Also, the head mounted display device 1000 may be implemented in the form of smart glasses or a smart helmet.

[0052] The head mounted display device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 may be a central processing unit CPU, a microprocessor MCU, and the like. The memory 1200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, and the like. The interface device 1300 may include, for example, a USB interface, a headphone interface, and the like. The communication device 1400 may, for example, be capable of wired or wireless communication. The display device 1500 may be, for example, a liquid crystal display screen, a touch display screen, and the like. The input device 1600 may include, for example, a touch screen, a keyboard, and the like. The user may input / output voice information through the speaker 1700 and the microphone 1800.

[0053] Despite Figure 1In the figure, multiple devices are shown for the head-mounted display device 1000, but the present application may only involve some of the devices, for example, the head-mounted display device 1000 only involves the memory 1200 and the processor 1100.

[0054] In the embodiment of the present application, the memory 1200 of the head mounted display device 1000 is used to store instructions, and the instructions are used to control the processor 1100 to execute the method for determining the interaction position provided in the embodiment of the present application.

[0055] In the above description, the technician can design instructions according to the scheme disclosed in this application. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0056] Figure 2 1 is a flow chart of a method for determining an interaction position according to an embodiment of the present application. The method for determining an interaction position provided in an embodiment of the present application is applied to a head mounted display device, and includes the following S2100-S2400:

[0057] S2100: Display a first image.

[0058] The first image is synthesized by at least one second image received by the head mounted display device.

[0059] In one embodiment of the present application, at least one streaming device is communicatively connected to a head-mounted display device. For any streaming device, at least one image is sent to the head-mounted display device. Based on this, the head-mounted display device receives at least one image. It should be noted that in the embodiment of the present application, the image received by the head-mounted display device is recorded as the second image. The streaming device may be a terminal device, such as a smart phone, a tablet computer, etc. Of course, the streaming device may also be a server, such as a cloud server, etc. In the case where the streaming device is a smart phone, the streaming image may be a foreground running image and / or a background running image of the smart phone.

[0060] And, the head mounted display device displays the corresponding second image in display windows located at different positions in its display area.

[0061] In the embodiment of the present application, the second image displayed by the display window is merged with the second image, and the second image with the same depth value after merging is recorded as the first image. Based on this, it can be known that the second image in the first image is equivalent to being located in the same plane as the first image.

[0062] It should be noted that the depth value of the second image in the first image is known and fixed.

[0063] In one embodiment of the present application, the display space of the head mounted display device may be specifically represented as a three-dimensional coordinate system of the scene displayed by the head mounted display device. The three-dimensional coordinate system may be exemplarily as follows: Figure 3 The depth value shown is composed of an X-axis, a Y-axis, a Z-axis and an origin O. On this basis, the depth value can be specifically reflected as a coordinate value on the Z-axis in the display space of the head-mounted display device.

[0064] In one example, the first image M2, the second image Q1, the second image Q2, and the second image Q3 may be as follows: Figure 3 shown.

[0065] S2200: Determine a first virtual identifier of the control device in a display space of a head mounted display device according to an initial position and a target position of the control device.

[0066] The control device is used to control the first image.

[0067] In the embodiment of the present application, the control device controlling the first image specifically refers to the control device controlling a second image in the first image.

[0068] In one example, the control device may be exemplarily a handle compatible with the head-mounted display device.

[0069] The target pose is the pose when the control device sends an interaction command to the head mounted display device. The initial pose is the pose of the control device in the initial state.

[0070] And, the first virtual identifier is used to describe a path where the interaction position on the first image is pointed to by the control device.

[0071] In an example, the first virtual marker may be a straight line, a ray, or a curve, etc., which is not limited in this application.

[0072] It is understandable that the method for determining the interaction position provided in the embodiment of the present application further includes the step of obtaining the initial posture and target posture of the control device before the above S2200. This step can be specifically implemented by the following S2210-S2212:

[0073] S2210: Obtain an initial position and posture of the control device.

[0074] In one embodiment of the present application, an inertial sensor is provided on the control device, and the inertial sensor is used to collect inertial data of the control device in an initial state, so as to obtain the initial position and posture of the control device.

[0075] S2211. Receive an interaction instruction sent by a control device for at least one first image.

[0076] In the embodiment of the present application, the user uses the control device to select a position on the first image as the interaction position, and then triggers a button on the control device to send an interaction instruction to the head mounted display device.

[0077] S2212. In response to the interaction instruction, determine a target posture of the control device according to the inertial data of the control device.

[0078] In the embodiment of the present application, when the above interaction instruction is received, the interaction instruction is responded to and the inertial data collected by the inertial sensor at this time is recorded. Further, based on the inertial data, the target position and posture of the control device is obtained.

[0079] In one embodiment of the present application, the specific implementation of the above S2200 may be: according to the initial posture and target posture of the control device, determine the virtual identification of the control device in its own space. Based on the conversion relationship between the space of the control device and the display space of the head-mounted display device, convert the virtual first of the control device in its own space into a virtual identification in the display space of the head-mounted display device. In the embodiment of the present application, the virtual identification obtained by converting to the display space of the head-mounted display device is recorded as the first virtual identification.

[0080] S2300: Determine, according to the first virtual marker and the third image, first coordinates of an intersection of the first virtual marker and the third image and determine a first angle between the first virtual marker and the third image.

[0081] The third image is constructed to determine the second coordinates of the intersection of the first virtual marker and the first image. The third image is located in the display space of the head mounted display device and is arranged at a preset distance from the first image.

[0082] In one embodiment of the present application, the specific manner in which the third image is located in the display space of the head mounted display device and is arranged at a preset interval with the first image may be: the third image is located in the display space of the head mounted display device and is arranged in parallel with the first image at a preset interval. Of course, other manners are also possible, and the present application does not limit this.

[0083] In an embodiment of the present application, the third image is located in the space where the first virtual identifier rendered by the head mounted display device is located, and its data comes from the local head mounted display device. The first image is the space where the second image sent by the streaming device is located, and its data source is the streaming device. Based on this, according to the relationship between the collision of rays and objects, the first virtual identifier can only be rendered on the third image, but not on the first image, that is, the first virtual identifier can only generate an intersection with the third image as a collision point, but cannot generate an intersection with the first image as a collision point. Among them, the intersection generated as a collision point between the first virtual identifier and the third image can be as follows Figure 3In P1(x1, y1), the intersection point generated by the first virtual mark and the first image as the collision point can be as follows Figure 3 As shown in P2(x2, y2).

[0084] In actual control, the intersection of the first virtual marker and the first image, that is, the interactive position of the first virtual marker and the first image, is meaningful.

[0085] In the embodiment of the present application, determining the interaction position between the first virtual marker and the first image is to determine the second coordinates of the intersection of the first virtual marker and the first image.

[0086] In one embodiment of the present application, the method for determining the interaction position provided in the embodiment of the present application further includes the step of acquiring a third image before the above S2300. Based on this, the method for determining the interaction position provided in the embodiment of the present application further includes the following S2310 before the above S2300:

[0087] S2310: Construct a third image according to the size of the first image and the preset spacing.

[0088] The size of the first image is known in advance.

[0089] In one embodiment of the present application, the third image may be determined as an image having the same size as the first image and being parallel to the plane where the image in the first image is located at a preset distance.

[0090] In one example, the third image may be Figure 3 As shown in M1 in , the distance between M1 and M2 is H.

[0091] In one embodiment of the present application, the specific implementation of the above S2300 may be: determining the plane equation corresponding to the third image according to the third image; determining the first coordinates of the intersection of the first virtual marker and the third image by calculating the intersection of the ray and the plane.

[0092] Furthermore, the angle between the first virtual marker and the third image, ie, the first angle, may be determined according to a formula for calculating the angle between a line and a plane.

[0093] S2400: Determine a second coordinate of an intersection of the first virtual marker and the first image according to the first coordinate, the preset distance, and the first angle.

[0094] In the embodiment of the present application, the first image is parallel to the third image, so the first virtual marker will intersect with the first image after extension. In the embodiment of the present application, the coordinates of the intersection formed by the first virtual marker after extension and the first image are marked as the second coordinates.

[0095] In one embodiment of the present application, the above S2400 may be specifically implemented by the following S2410 and S2411:

[0096] S2410: Determine, according to a preset distance and a first angle, a coordinate offset of a second coordinate of an intersection point between the first virtual marker and the first image compared to the first coordinate.

[0097] In the embodiment of the present application, Figure 3 Taking the three-dimensional coordinates of the head-mounted display device shown as an example, the coordinate offset of the second coordinate of the intersection of the first virtual marker and the first image compared to the first coordinate includes an offset on the X-axis and an offset on the Y-axis.

[0098] It can be understood that, since the third image is parallel to the first image, the angle between the first virtual marker and the third image is the angle between the first virtual marker and the first image.

[0099] Figure 4a The calculation principle of the X-axis offset is shown in FIG. Figure 4a The angle b in is the angle between the first virtual marker and the first image projected on the YOZ plane, that is, the angle between the first angle projected on the YOZ plane. H is the preset distance. dx is the offset of the intersection of the first virtual marker and the first image compared to the first coordinate on the X-axis.

[0100] For details, see Figure 4a , dx = H*tan(b).

[0101] Based on the same principle, Figure 4b The calculation principle of the Y-axis offset is shown in FIG. Figure 4b The angle a in is the angle between the first virtual marker and the first image, projected on the XOZ plane, that is, the angle of the first angle projected on the XOZ plane. dy is the offset of the first coordinate on the Y axis compared to the intersection of the first virtual marker and the first image.

[0102] For details, see Figure 4a , dy = H*tan(a).

[0103] S2411. Determine a second coordinate of an intersection of the first virtual marker and the first image according to the coordinate offset and the first coordinate.

[0104] In the embodiment of the present application, the coordinate of the intersection of the first virtual marker and the first image on the y-axis is specifically y2=y1+dy, and the coordinate on the x-axis is specifically x2=x1+dx, that is, the second coordinate is (x1+dx, y2=y1+dy).

[0105] The embodiment of the present application provides a method for determining an interactive position, which is applied to a head-mounted display device, including: displaying a first image, the first image being synthesized by at least one second image received by the head-mounted display device; determining a first virtual identifier of the control device in the display space of the head-mounted display device according to the initial position and target position of the control device, the control device being used to control the first image; determining a first coordinate of an intersection of the first virtual identifier and the third image and a first angle between the first virtual identifier and the third image according to the first virtual identifier and the third image, the third image being located in the display space of the head-mounted display device and being arranged at a preset interval with the first image; determining a second coordinate of an intersection of the first virtual identifier and the first image according to the first coordinate, the preset interval and the first angle. The method can determine the interactive position of the control device corresponding to the first image. That is, in the case where the second image is an image obtained by streaming, the method can determine the interactive position of the control device corresponding to the streamed image.

[0106] In one embodiment of the present application, the method for determining an interaction position provided in the embodiment of the present application further includes the following S2500:

[0107] S2500: Determine a target second image pointed to by the first virtual marker according to the second coordinates and a display range of at least one second image.

[0108] In the embodiment of the present application, the second image is displayed in the display window of the head mounted display device, and the positional relationship between the display window of the head mounted display device and the display space of the head mounted display device is known. Therefore, the display range of the second image corresponding to the display space of the head mounted display device is known.

[0109] In an embodiment of the present application, the first image includes at least one second image. There are two situations in which the first virtual identifier points to one of the second images, and points to an image other than the second image in the first image. Based on this, the position pointed to by the first virtual identifier can be determined according to the display range of each second image. Specifically, when the second coordinate is within the display range of one of the second images, it is determined that the first virtual identifier points to the second image, and the second image is used as the target second image, that is, as the second image that the current user wants to control through the control device. Based on this, according to the correspondence between the second image and the streaming device, the target streaming device corresponding to the target second image can be determined. In this way, a reference can be provided for the user to control the target streaming device.

[0110] In one embodiment of the present application, the method for determining the interaction position provided in the embodiment of the present application further includes the following S2600 after the above S2400:

[0111] S2600: Display, according to the first virtual identifier and the second coordinates, a second virtual identifier including the first virtual identifier and having an end point coordinate as the second coordinates.

[0112] In the embodiment of the present application, the first virtual identifier and the second virtual identifier have the same expression form. For example, when the first virtual identifier is a virtual ray, the second virtual identifier is also a virtual ray.

[0113] Through the above S2600, the first virtual identifier is displayed as a virtual identifier extended to the first image, that is, the second virtual identifier. In this way, interaction visualization can be achieved, that is, the user can visually know which point on the first image the control device controlled by the user interacts with.

[0114] It is understandable that, when the depth values ​​of the merged second images are different, for any second image, the image with the same depth value as the second image can be directly regarded as the first image, and the method for determining the interaction position provided in any of the above embodiments can be executed. In this way, the intersection of the first virtual marker and the first image can still be obtained, that is, the interaction position of the control device corresponding to the first image can be determined.

[0115] The embodiment of the present application also provides a device 500 for determining an interaction position, which is applied to a head mounted display device, such as Figure 5 As shown, including:

[0116] A display module 510, configured to display a first image, wherein the first image is synthesized from at least one second image received by the head mounted display device;

[0117] A first determination module 520, configured to determine a first virtual identifier of the control device in the display space of the head mounted display device according to an initial position and a target position of the control device, wherein the control device is configured to control the first image;

[0118] A second determining module 530 is used to determine, according to the first virtual marker and the third image, a first coordinate of an intersection of the first virtual marker and the third image and a first angle between the first virtual marker and the third image, wherein the third image is located in the display space of the head mounted display device and is arranged at a preset distance from the first image;

[0119] The third determination module 540 is configured to determine a second coordinate of an intersection of the first virtual marker and the first image according to the first coordinate, the preset distance, and the first angle.

[0120] In an embodiment of the present application, a device for determining an interactive position is provided, which is applied to a head-mounted display device, and includes: a display module for displaying a first image, the first image being synthesized by at least one second image received by the head-mounted display device; a first determination module for determining a first virtual identifier of the control device in the display space of the head-mounted display device according to the initial position and target position of the control device, the control device being used to control the first image; a second determination module for determining a first coordinate of an intersection of the first virtual identifier and the third image and a first angle between the first virtual identifier and the third image according to the first virtual identifier and the third image, the third image being located in the display space of the head-mounted display device and being arranged at a preset interval with the first image; a third determination module for determining a second coordinate of an intersection of the first virtual identifier and the first image according to the first coordinate, the preset interval and the first angle. The device can determine the interactive position of the control device corresponding to the first image. That is, in the case where the second image is an image obtained by streaming, the device can determine the interactive position of the control device corresponding to the streamed image.

[0121] In one embodiment of the present application, the third determination module 540 includes:

[0122] A first determining unit, configured to determine, according to the preset interval and the first angle, a coordinate offset of a second coordinate of an intersection point between the first virtual marker and the first image compared to the first coordinate;

[0123] The second determining unit is used to determine the second coordinates of the intersection of the first virtual marker and the first image according to the coordinate offset and the first coordinates.

[0124] In one embodiment of the present application, the apparatus 500 for determining an interaction position provided in the embodiment of the present application further includes:

[0125] An acquisition module, used to acquire an initial position and posture of the control device;

[0126] receiving an interaction instruction for the first image sent by a control device;

[0127] In response to the interaction instruction, a target posture of the control device is determined according to inertial data of the control device.

[0128] In one embodiment of the present application, the apparatus 500 for determining an interaction position provided in the embodiment of the present application further includes:

[0129] The fourth determining module is used to determine the target second image pointed to by the first virtual marker according to the second coordinates and the display range of the at least one second image.

[0130] In one embodiment of the present application, the display module 510 is also used for:

[0131] According to the first virtual marker and the second coordinates, a second virtual marker including the first virtual marker and having an end point coordinate as the second coordinates is displayed.

[0132] In one embodiment of the present application, the apparatus 500 for determining an interaction position provided in the embodiment of the present application further includes:

[0133] A construction module is used to construct the third image according to the size of the first image and a preset interval.

[0134] The embodiment of the present application also provides a head mounted display device 600, comprising the device 500 for determining an interaction position provided by any of the above device embodiments.

[0135] Or, if Figure 6 As shown, the head mounted display device 600 includes a memory 610 and a processor 620, wherein the memory 610 is used to store computer instructions, and the processor 620 is used to call the computer instructions from the memory 610 to execute the method for determining the interaction position as described in any one of the above method embodiments.

[0136] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for determining an interaction position according to any one of the above method embodiments is implemented.

[0137] The present application may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0138] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0139] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0140] The computer program instructions for performing the operation of the present application can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of a computer-readable program instruction to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute a computer-readable program instruction, thereby realizing various aspects of the present application.

[0141] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0142] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0143] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0144] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0145] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present application is defined by the appended claims.

Claims

1. A method for determining an interaction position, characterized in that: Applied to head mounted display devices, including: Displaying a first image, the first image being synthesized from at least one second image received by the head mounted display device; Determining a first virtual identifier of the control device in the display space of the head mounted display device according to an initial position and a target position of the control device, wherein the control device is used to control the first image; Determine, according to the first virtual marker and the third image, a first coordinate of an intersection of the first virtual marker and the third image and a first angle between the first virtual marker and the third image, wherein the third image is located in a display space of the head mounted display device and is arranged at a preset distance from the first image; A second coordinate of an intersection point between the first virtual marker and the first image is determined according to the first coordinate, the preset distance, and the first angle.

2. The method according to claim 1, characterized in that The determining, according to the first coordinates, the preset distance, and the first angle, a second coordinate of an intersection of the first virtual marker and the first image includes: Determine, according to the preset distance and the first angle, a coordinate offset of a second coordinate of an intersection point of the first virtual marker and the first image compared to the first coordinate; A second coordinate of an intersection point between the first virtual marker and the first image is determined according to the coordinate offset and the first coordinate.

3. The method according to claim 1, characterized in that The method further includes, before determining a first virtual identifier of the control device in the head mounted display device space according to the initial posture and the target posture of the control device: Obtaining an initial position and posture of the control device; receiving an interaction instruction for the first image sent by a control device; In response to the interaction instruction, a target posture of the control device is determined according to inertial data of the control device.

4. The method according to claim 1, characterized in that: The method further comprises: The target second image pointed to by the first virtual marker is determined according to the second coordinates and the display range of the at least one second image.

5. The method according to claim 1, characterized in that The method further comprises: According to the first virtual marker and the second coordinates, a second virtual marker including the first virtual marker and having an end point coordinate as the second coordinates is displayed.

6. The method according to claim 1, characterized in that Before determining the first coordinates of the intersection of the first virtual marker and the third image and determining the first angle between the first virtual marker and the third image according to the first virtual marker and the third image, the method further includes: The third image is constructed according to the size of the first image and the preset interval.

7. A device for determining an interaction position, characterized in that: Applied to head mounted display devices, including: A display module, configured to display a first image, wherein the first image is synthesized from at least one second image received by the head mounted display device; A first determination module, configured to determine a first virtual identifier of the control device in the display space of the head mounted display device according to an initial position and a target position of the control device, wherein the control device is configured to control the first image; A second determining module, configured to determine, according to the first virtual marker and the third image, a first coordinate of an intersection of the first virtual marker and the third image and a first angle between the first virtual marker and the third image, wherein the third image is located in a display space of the head mounted display device and is arranged at a preset interval with respect to the first image; The third determination module is used to determine the second coordinates of the intersection of the first virtual marker and the first image according to the first coordinates, the preset distance and the first angle.

8. The device according to claim 7, characterized in that The third determination module comprises: A first determining unit, configured to determine, according to the preset interval and the first angle, a coordinate offset of a second coordinate of an intersection point between the first virtual marker and the first image compared to the first coordinate; The second determining unit is used to determine a second coordinate of an intersection point of the first virtual marker and the first image according to the coordinate offset and the first coordinate.

9. A head mounted display device, characterized in that: The head mounted display device comprises the apparatus as claimed in claim 7 or 8; Alternatively, it comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the interactive position determination method according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for determining an interaction position according to any one of claims 1-6 is implemented.

Images