Interaction method and device in three-dimensional space, storage medium and electronic device

Abstract

Embodiments of the present application provide a kind of interactive method in three-dimensional space, device, storage medium and electronic device, wherein the method comprises: display target augmented reality three-dimensional picture on target augmented reality device, and display target virtual interactive object in virtual three-dimensional space in target augmented reality three-dimensional picture, reality three-dimensional space is rasterized, virtual three-dimensional space is rasterized, there is mapping between the two;Obtain the target moving track of target object in reality three-dimensional space;Determine the reality cell set that target moving track passes through;Determine the first virtual cell set that passes through corresponding with the reality cell set that passes through;According to the position relationship between first virtual cell set and second virtual cell set, determine whether target moving track triggers target virtual interactive object target interaction operation. By the present application, the technical problem that the application range of interactive method in related art is narrow is solved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of human-computer interaction technology, and more specifically, to an interaction method, device, storage medium and electronic device in three-dimensional space. Background Technology

[0002] In recent years, with the widespread application and popularization of computer technology, human-computer interaction technology has gradually become a research hotspot. For example, augmented reality (AR) and virtual reality (VR) technologies have also developed rapidly. Currently, interoperability between AR and VR scenes is mainly used in game scenarios. Relatively mature systems include Microsoft's Kinect, Nintendo's Wii motion controller, and Ring Fit Adventure. The common feature of these devices is that they are based on the position of a 2D plane to confirm interoperability between VR or AR scenes. Systems like Kinect are only suitable for interoperability between two-dimensional virtual space and real space. Such an interaction method greatly limits its application scenarios.

[0003] There is currently no effective solution to the technical problem of the narrow application scope of the interaction methods in related technologies. Summary of the Invention

[0004] This invention provides an interaction method, apparatus, storage medium, and electronic device in three-dimensional space, to at least solve the technical problem of the narrow application scope of interaction methods in related technologies.

[0005] According to an embodiment of the present invention, an interaction method in three-dimensional space is provided, comprising: displaying a target augmented reality three-dimensional image on a target augmented reality device, and displaying a target virtual interactive object in a virtual three-dimensional space within the target augmented reality three-dimensional image, wherein the target augmented reality three-dimensional image is a three-dimensional image formed by superimposing a real three-dimensional image captured in real three-dimensional space with a virtual three-dimensional image in the virtual three-dimensional space, the real three-dimensional space is rasterized into a set of real cells, the virtual three-dimensional space is rasterized into a set of virtual cells, and there is a mapping relationship between the set of real cells and the set of virtual cells; when a target object is identified in the real three-dimensional space, acquiring the target movement trajectory of the target object in the real three-dimensional space; and displaying a target virtual interactive object in the set of real cells. The system determines the set of real-world cells traversed by the target movement trajectory; based on the mapping relationship between the set of real-world cells and the set of virtual cells, it determines the first set of virtual cells traversed corresponding to the set of real-world cells traversed; based on the positional relationship between the first set of virtual cells and the second set of virtual cells, it determines whether the target movement trajectory triggers a target interaction operation of the target virtual interactive object, wherein the second set of virtual cells includes the virtual cells occupied by the target virtual interactive object in the set of virtual cells; if the target movement trajectory triggers the target interaction operation of the target virtual interactive object, it displays the screen corresponding to the execution of the target interaction operation on the target augmented reality device.

[0006] In an exemplary embodiment, acquiring the target movement trajectory of the target object in the real three-dimensional space includes: acquiring ranging information obtained by the target acquisition module measuring the distance of the target object at each time moment in a set of time moments; determining the position of the target object in the real three-dimensional space at each time moment based on the ranging information acquired at each time moment; and fitting the position of the target object in the real three-dimensional space at each time moment into the target movement trajectory.

[0007] In an exemplary embodiment, determining the position of the target object in the real three-dimensional space at each time step based on the ranging information acquired at each time step includes: for the ranging information acquired at each time step, performing the following operations, wherein each time step is the current time step: when the target acquisition module is a ranging module and the ranging information acquired at the current time step represents the current distance between the target object and a preset marker point in the real three-dimensional space, determining the position of the target object in the real three-dimensional space at the current time step based on the position of the preset marker point in the real three-dimensional space and the current distance represented by the ranging information; when the target acquisition module is a ranging module and the ranging information acquired at the current time step represents the current distance between the target object and a preset marker point in the real three-dimensional space ... The block includes multiple ranging modules. The ranging information obtained at the current moment includes multiple ranging information obtained by the multiple ranging modules measuring the target object at the current moment, and each of the multiple ranging information represents the current distance between the target object and the preset marker point in the real three-dimensional space. Based on the position of the preset marker point in the real three-dimensional space and the multiple current distances represented by the multiple ranging information, multiple positions of the target object in the real three-dimensional space at the current moment are determined. Based on the multiple positions of the target object in the real three-dimensional space at the current moment, the position of the target object in the real three-dimensional space at the current moment is determined.

[0008] In an exemplary embodiment, determining the position of the target object in the real three-dimensional space at the current moment based on the plurality of positions of the target object in the real three-dimensional space at the current moment includes: averaging the three-dimensional coordinates of the plurality of positions in the real three-dimensional space to obtain a first target three-dimensional coordinate, and determining the position represented by the first target three-dimensional coordinate in the real three-dimensional space as the position of the target object in the real three-dimensional space at the current moment; or performing a weighted average of the three-dimensional coordinates of the plurality of positions in the real three-dimensional space to obtain a second target three-dimensional coordinate, and determining the position represented by the second target three-dimensional coordinate in the real three-dimensional space as the position of the target object in the real three-dimensional space at the current moment.

[0009] In an exemplary embodiment, before displaying the target augmented reality 3D image on the target augmented reality device, the method further includes: acquiring hardware resource information of a target acquisition module associated with the target augmented reality device and processing capability information of the target augmented reality device, wherein the target acquisition module is used to measure the distance of the target object to obtain distance information, and the distance information is used to determine the target movement trajectory of the target object in the real 3D space; and based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device, rasterizing the real 3D space to obtain the set of real cells, and rasterizing the virtual 3D space to obtain a set of virtual cells.

[0010] In an exemplary embodiment, the step of rasterizing the real three-dimensional space to obtain the set of real cells based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device, and rasterizing the virtual three-dimensional space to obtain a set of virtual cells, includes: determining the value of a target processing capability parameter based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device; rasterizing the real three-dimensional space according to a first cell size corresponding to the value of the target processing capability parameter to obtain the set of real cells, and rasterizing the virtual three-dimensional space to obtain a set of virtual cells, wherein the value of the target processing capability parameter is negatively correlated with the first cell size; or, rasterizing the real three-dimensional space according to a second cell size corresponding to the value range of the target processing capability parameter to obtain the set of real cells, and rasterizing the virtual three-dimensional space to obtain a set of virtual cells, wherein the boundary value of the value range of the target processing capability parameter is negatively correlated with the second cell size.

[0011] In an exemplary embodiment, determining the set of real cells traversed by the target movement trajectory within the set of real cells includes: searching for the cell containing the position of the target movement trajectory within the set of real cells to obtain the set of real cells.

[0012] In an exemplary embodiment, determining whether the target movement trajectory triggers a target interaction operation of the target virtual interactive object based on the positional relationship between the first virtual cell set and the second virtual cell set includes: when a target response range is set for the target virtual interactive object in the virtual three-dimensional space, and the target response range includes a third virtual cell set in the virtual three-dimensional space, detecting whether virtual cells in the first virtual cell set are located inside the third virtual cell set, wherein the second virtual cell set is located inside the third virtual cell set in the virtual three-dimensional space; when a subset of virtual cells in the first virtual cell set is detected to be located inside the third virtual cell set, determining whether the target movement trajectory triggers a target interaction operation of the target virtual interactive object based on the positional relationship between the subset of virtual cells in the first virtual cell set and the second virtual cell set; or when no target response range is set for the target virtual interactive object in the virtual three-dimensional space, determining whether the target movement trajectory triggers a target interaction operation of the target virtual interactive object based on the positional relationship between the first virtual cell set and the second virtual cell set.

[0013] In an exemplary embodiment, determining whether the target movement trajectory triggers a target interaction operation of the target virtual interaction object based on the positional relationship between the subset of virtual cells in the first virtual cell set and the second virtual cell set includes: determining whether the subset of virtual cells in the first virtual cell set is located in the second virtual cell set; if it is determined that there are virtual cells in the subset of virtual cells located in the second virtual cell set, determining that the target movement trajectory triggers the target interaction operation of the target virtual interaction object; or determining the number of virtual cells in the first virtual cell set located in the second virtual cell set; if the number of virtual cells in the subset of virtual cells located in the second virtual cell set is greater than or equal to a predetermined number threshold, determining that the target movement trajectory triggers the target interaction operation of the target virtual interaction object; or determining whether the subset of virtual cells in the first virtual cell set is located in the second virtual cell set; if it is determined that there are virtual cells in the subset of virtual cells located in the second virtual cell set, and the duration of stay of the virtual cells in the subset of virtual cells in the second virtual cell set is greater than or equal to a predetermined duration threshold, determining that the target movement trajectory triggers the target interaction operation of the target virtual interaction object.

[0014] In one exemplary embodiment, the method further includes: searching for interaction logs between the target object and the target virtual interaction object in the log information of the target augmented reality device; determining abnormal interaction operations between the target object and the target virtual interaction object based on the interaction logs between the target object and the target virtual interaction object; and adjusting the triggering conditions of the target interaction operation based on the abnormal interaction operation.

[0015] According to another embodiment of the present invention, an interactive device in three-dimensional space is provided, comprising: a first display module, configured to display a target augmented reality three-dimensional image on a target augmented reality device, and display a target virtual interactive object in a virtual three-dimensional space within the target augmented reality three-dimensional image, wherein the target augmented reality three-dimensional image is a three-dimensional image formed by superimposing a real three-dimensional image captured in real three-dimensional space with a virtual three-dimensional image in the virtual three-dimensional space, the real three-dimensional space is rasterized into a set of real cells, the virtual three-dimensional space is rasterized into a set of virtual cells, and there is a mapping relationship between the set of real cells and the set of virtual cells; a first acquisition module, configured to acquire the target movement trajectory of the target object in the real three-dimensional space when the target object is identified in the real three-dimensional space; and a first determination module, configured to determine the target movement trajectory of the target object in the real three-dimensional space when the target object is identified in the real three-dimensional space; and a first determination module, configured to determine the target movement trajectory of the target object in the real three-dimensional space when the target object is identified in the real three-dimensional space. The system comprises: a first determining module, which determines the set of real cells traversed by the target movement trajectory within the set of real cells; a second determining module, which determines, based on the mapping relationship between the set of real cells and the set of virtual cells, a first set of virtual cells corresponding to the set of real cells traversed; a third determining module, which determines, based on the positional relationship between the first set of virtual cells and the second set of virtual cells, whether the target movement trajectory triggers a target interaction operation of the target virtual interaction object, wherein the second set of virtual cells includes the virtual cells occupied by the target virtual interaction object in the set of virtual cells; and a second display module, which, when the target movement trajectory triggers the target interaction operation of the target virtual interaction object, displays the screen corresponding to the execution of the target interaction operation on the target augmented reality device.

[0016] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0017] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0018] In this embodiment of the invention, the target object's movement trajectory in real three-dimensional space is obtained, and then the set of real-world cells traversed by the target movement trajectory is determined. Based on a pre-established mapping relationship between real and virtual three-dimensional spaces, a first set of virtual cells in the virtual three-dimensional space corresponding to the set of real-world cells is determined. Then, based on the positional relationship between the first and second virtual cell sets, it is determined whether the target movement trajectory triggers a target virtual interactive object. The second virtual cell set includes the virtual cells occupied by the target virtual interactive object. When the target movement trajectory triggers a target interactive operation of the target virtual interactive object, the corresponding screen is displayed on the target augmented reality device. That is, by operating the target object in real three-dimensional space, the target interactive operation of the target virtual interactive object in virtual three-dimensional space is triggered, and the screen corresponding to the target interactive operation is displayed on the target augmented reality device. This embodiment realizes interactive operations in three-dimensional space, solving the technical problem of narrow application scope of interactive methods in related technologies and achieving the effect of broadening the application scope of interactive methods. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with the description thereof, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0020] Figure 1 This is a block diagram of the mobile terminal hardware structure of the interaction method in three-dimensional space according to an embodiment of the present invention.

[0021] Figure 2 This is a flowchart of an interaction method in three-dimensional space according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram illustrating the relationship between real space and virtual space according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram illustrating the response range of a target virtual interactive object according to an embodiment of the present invention;

[0024] Figure 5 This is an overall flowchart of a three-dimensional space interaction method according to a specific embodiment of the present invention;

[0025] Figure 6This is an overall flowchart of a three-dimensional space interaction method according to another specific embodiment of the present invention;

[0026] Figure 7 This is a flowchart of the initialization module according to a specific embodiment of the present invention;

[0027] Figure 8 This is a flowchart of the distance measurement object locking module according to a specific embodiment of the present invention;

[0028] Figure 9 This is a flowchart of the three-dimensional coordinate system corresponding module according to a specific embodiment of the present invention;

[0029] Figure 10 This is a working example diagram of a confirmation feedback module according to a specific embodiment of the present invention;

[0030] Figure 11 This is a working example diagram of the confirmation feedback module according to another specific embodiment of the present invention;

[0031] Figure 12 This is a structural block diagram of an interactive device in three-dimensional space according to an embodiment of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1This is a block diagram of the mobile terminal hardware structure of the interaction method in three-dimensional space according to an embodiment of the present invention, such as... Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the three-dimensional interaction method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0037] This embodiment provides an interaction method in three-dimensional space. Figure 2 This is a flowchart of an interaction method in three-dimensional space according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0038] Step S202: Display a target augmented reality 3D image on the target augmented reality device, and display a target virtual interactive object in the virtual 3D space of the target augmented reality 3D image. The target augmented reality 3D image is a 3D image formed by superimposing a real 3D image captured in real 3D space with a virtual 3D image in the virtual 3D space. The real 3D space is rasterized into a set of real cells, and the virtual 3D space is rasterized into a set of virtual cells. There is a mapping relationship between the set of real cells and the set of virtual cells.

[0039] Step S204: When a target object is identified in the real three-dimensional space, the target object's movement trajectory in the real three-dimensional space is obtained;

[0040] Step S206: Determine the set of real cells through which the target movement trajectory passes within the set of real cells;

[0041] Step S208: Based on the mapping relationship between the set of real cells and the set of virtual cells, determine the first set of virtual cells that has been passed and corresponds to the set of real cells passed in the set of virtual cells.

[0042] Step S210: Based on the positional relationship between the first set of virtual cells and the second set of virtual cells, determine whether the target movement trajectory triggers the target interaction operation of the target virtual interaction object, wherein the second set of virtual cells includes the virtual cells occupied by the target virtual interaction object in the set of virtual cells;

[0043] Step S212: When the target movement trajectory triggers the target interaction operation of the target virtual interaction object, the screen corresponding to the execution of the target interaction operation is displayed on the target augmented reality device.

[0044] Through the above steps, the target object's movement trajectory in real 3D space is obtained, and the set of real-world cells traversed by the trajectory is determined. Then, based on a pre-established mapping relationship between real and virtual 3D space, a first set of virtual cells in the virtual 3D space corresponding to the set of real-world cells is determined. Based on the positional relationship between the first and second sets of virtual cells, it is determined whether the target movement trajectory triggers a target virtual interactive object. The second set of virtual cells includes the virtual cells occupied by the target virtual interactive object. When the target movement trajectory triggers a target interactive operation of the target virtual interactive object, the corresponding screen is displayed on the target augmented reality device. In other words, by operating the target object in real 3D space, the target interactive operation of the target virtual interactive object in virtual 3D space is triggered, and the screen corresponding to the target interactive operation is displayed on the target augmented reality device. This embodiment realizes interactive operations in 3D space, solving the technical problem of narrow application scope of interactive methods in related technologies and achieving the effect of broadening the application scope of interactive methods.

[0045] The entity executing the above steps can be an interactive system, such as a human-computer interaction system, a 3D interactive system, an AR interactive system (or device), or a VR interactive system (or device). The entity executing the above steps can also be other processing devices or processing units with similar processing capabilities, but is not limited to these. The following explanation uses an AR interactive system performing the above operations as an example (this is merely an illustrative example; in actual operation, other devices or modules can also perform the above operations):

[0046] In the above embodiments, a target augmented reality 3D image is displayed on the target augmented reality device. For example, the target augmented reality 3D image is a 3D image formed by superimposing a real 3D image captured in real 3D space with a virtual 3D image in virtual 3D space. A target virtual interactive object is displayed in the virtual 3D space within the target augmented reality 3D image. For example, in a game scene, the target virtual interactive object can be a confirmation button, an item, or other objects used for interaction with the user in a virtual interface. In practical applications, a mapping relationship between real 3D space and virtual 3D space can be pre-established. Optionally, real 3D space is rasterized into a set of real cells, and virtual 3D space is rasterized into a set of virtual cells, thus establishing a mapping relationship between the set of real cells and the set of virtual cells. When a target object is identified in real 3D space, the target object's movement trajectory in real 3D space is obtained. For example, the target object can be an optical card, a cursor card, or a combination of an optical card and a cursor card, a rangefinder card, an optical label, or an Ultra Wideband (UWB) device. Band (UWB) tags, millimeter-wave relay beacons, or users wearing certain cards or tags, such as optical cards or tags on their fingers, are all considered. In practical applications, AR interactive systems need to be configured with a data acquisition module to collect spatial data of target objects in real-world 3D space, such as angles, distances, or other data, to determine the coordinates of the target object in real-world 3D space. The acquisition module can include one or more ranging modules, such as UWB modules, millimeter-wave ranging modules, and optical ranging modules, or other acquisition modules. Different ranging modules or combinations thereof can be selected based on different application scenarios. Then, the set of real cells traversed by the target's movement trajectory can be determined from a set of real cells in real-world 3D space. Finally, based on the mapping between a set of real cells and a set of virtual cells... The relationship is as follows: First, within a set of virtual cells, the first set of virtual cells corresponding to the set of real cells traversed is determined. This first set of virtual cells is the set of real cells in the real 3D space mapped to the corresponding set of cells in the virtual 3D space. Alternatively, it can be understood as the movement trajectory of a target object in the real 3D space mapped to its movement trajectory in the virtual 3D space. For example, the movement trajectory of a cursor in the virtual 3D space; the virtual cells traversed by this movement trajectory are the aforementioned first set of virtual cells. Then, based on the positional relationship between the first and second sets of virtual cells, it is determined whether the target movement trajectory triggers the target interaction operation of the target virtual interactive object. The second set of virtual cells includes the virtual cells occupied by the target virtual interactive object within the set of virtual cells. For example, the target virtual interactive object may occupy only one virtual cell or multiple virtual cells.When a target's movement trajectory triggers a target interaction operation on a target virtual interactive object—for example, the operation on the target virtual interactive object could be a single click, double click, or other method—the corresponding screen for performing the target interaction operation is displayed on the target augmented reality device. For instance, upon determining that a target interaction operation has been triggered on the target virtual interactive object, the screen corresponding to the execution of the target interaction operation is displayed on the target augmented reality device. This responds to the target movement trajectory of the target object in real three-dimensional space, realizing interactive operations in three-dimensional space. This solves the technical problem of the narrow application scope of interaction methods in related technologies and achieves the effect of broadening the application scope of interaction methods.

[0047] Figure 3 This is a schematic diagram illustrating the relationship between real space and virtual space according to an embodiment of the present invention, such as... Figure 3 As shown, Figure 3 The left side corresponds to a raster image of the real three-dimensional space. Figure 3 The image on the right corresponds to a raster image of the virtual 3D space. It should be noted that in practical applications, the size of the smallest real-world cell after rasterization in the real 3D space can be the same as or different from the size of the smallest virtual cell after rasterization in the virtual 3D space. Figure 3 In the left-middle image, A (the upper surface in the left image) corresponds to the aforementioned target object. Figure 3 In the right-hand diagram, B (the upper surface) corresponds to the target object in real-world 3D space mapped to the object in virtual 3D space. For example, it can be represented by a cursor, or by a cartoon character or figure. Figure 3 In the left image, the target object moves from cell A to cell C, that is, from cell A to cell C. This corresponds to the target movement trajectory of the target object in the above embodiment. This target movement trajectory, after being mapped onto the virtual 3D space, corresponds to... Figure 3 In the right-hand diagram, the target moves from B to C', where C' represents the aforementioned target virtual interactive object. When the target's movement trajectory (e.g., A→C) triggers a target interactive operation on the target virtual interactive object, such as a single click, double click, or other method, the corresponding screen for performing the target interactive operation is displayed on the target augmented reality device. For example, it could be a response after confirming the target virtual interactive object. In practical applications, sound, light, or touch can also be used to prompt the operator simultaneously.

[0048] In an optional embodiment, obtaining the target object's movement trajectory in the real three-dimensional space includes: acquiring ranging information obtained by the target acquisition module measuring the target object at each moment in a set of time intervals; determining the position of the target object in the real three-dimensional space at each moment based on the ranging information acquired at each moment; and fitting the position of the target object in the real three-dimensional space at each moment into the target movement trajectory. For example, the target acquisition module can measure the target object at multiple moments within a period of time to obtain a set of ranging information. The ranging information can be obtained based on multiple frames of images included in the video data acquired by the target acquisition module. Optionally, the ranging information of the target object can be acquired periodically within a period of time according to a predetermined cycle, and then the position of the target object in the real three-dimensional space at each moment can be determined. Figure 3 The target acquisition module uses real-world cells in a 3D space. Within a given time period, the target may traverse multiple cells. By fitting the positions of these multiple cells to a curve, the target's movement trajectory can be obtained. In this embodiment, the target acquisition module can be a single ranging module or multiple ranging modules. This embodiment achieves the goal of determining the target's movement trajectory in real-world 3D space.

[0049] In an optional embodiment, determining the position of the target object in the real three-dimensional space at each time step based on the ranging information acquired at each time step includes: for the ranging information acquired at each time step, performing the following operations, wherein each time step is the current time step: when the target acquisition module is a ranging module and the ranging information acquired at the current time step represents the current distance between the target object and a preset marker point in the real three-dimensional space, determining the position of the target object in the real three-dimensional space at the current time step based on the position of the preset marker point in the real three-dimensional space and the current distance represented by the ranging information; when the target acquisition module is a ranging module and the ranging information acquired at the current time step represents the current distance between the target object and a preset marker point in the real three-dimensional space ... The block includes multiple ranging modules. The ranging information obtained at the current moment includes multiple ranging information obtained by the multiple ranging modules measuring the target object at the current moment, and each of the multiple ranging information represents the current distance between the target object and the preset marker point in the real three-dimensional space. Based on the position of the preset marker point in the real three-dimensional space and the multiple current distances represented by the multiple ranging information, multiple positions of the target object in the real three-dimensional space at the current moment are determined. Based on the multiple positions of the target object in the real three-dimensional space at the current moment, the position of the target object in the real three-dimensional space at the current moment is determined. In this embodiment, when the target acquisition module is a ranging module and the ranging information acquired at the current moment represents the current distance between the target object and the preset marker point, the preset marker point can be the origin of the coordinate system in the real three-dimensional space or a reference point in the real three-dimensional space. The position of the target object in the real three-dimensional space at the current moment can be determined based on the position of the preset marker point in the real three-dimensional space and the aforementioned current distance. When the target acquisition module consists of multiple ranging modules, and the ranging information acquired at the current moment includes multiple ranging information obtained by multiple ranging modules measuring the target object, and each ranging information represents the distance between the target object and the preset marker point. When determining the current distance, the preset marker point can be the origin of the coordinate system in real 3D space or a reference point in real 3D space. Based on the position of the preset marker point in real 3D space and the multiple current distances represented by each of the aforementioned distance measurement information, multiple positions of the target object in real 3D space at the current moment can be determined. Then, based on the multiple positions of the target object in real 3D space, the final position of the target object in real 3D space at the current moment can be determined. For example, in practical applications, the final position of the target object can be determined by averaging multiple positions, or by using a weighted average algorithm for multiple positions. Through this embodiment, the purpose of determining the position of the target object in real 3D space at each moment is achieved.

[0050] In an optional embodiment, determining the position of the target object in the real three-dimensional space at the current moment based on the plurality of positions of the target object in the real three-dimensional space at the current moment includes: averaging the three-dimensional coordinates of the plurality of positions in the real three-dimensional space to obtain a first target three-dimensional coordinate, and determining the position represented by the first target three-dimensional coordinate in the real three-dimensional space as the position of the target object in the real three-dimensional space at the current moment; or performing a weighted average of the three-dimensional coordinates of the plurality of positions in the real three-dimensional space to obtain a second target three-dimensional coordinate, and determining the position represented by the second target three-dimensional coordinate in the real three-dimensional space as the position of the target object in the real three-dimensional space at the current moment. In this embodiment, the position of the target object in the real three-dimensional space at the current moment can be determined by averaging multiple positions in the real three-dimensional space; alternatively, the position can be determined by using a weighted average algorithm on multiple positions in the real three-dimensional space. For example, when the target acquisition module includes multiple ranging modules, if the optical ranging module is not applicable in certain application scenarios, the weighting coefficient of the optical ranging module can be set to a very small value or zero. Similarly, if the millimeter-wave ranging module is not applicable to certain application scenarios, the weighting coefficient of that module can be reduced or set to zero. This allows for flexible integration of ranging data from different ranging modules for different application scenarios, thereby improving the reliability of the ranging data. Through this embodiment, by using multiple ranging modules to measure the distance to the target object and performing comprehensive calculations (such as averaging or weighted average calculations) on the measured distance information to determine the position of the target object, the accuracy of determining the position of the target object can be improved, thereby reducing spatial system errors.

[0051] In an optional embodiment, before displaying the target augmented reality 3D image on the target augmented reality device, the method further includes: acquiring hardware resource information of a target acquisition module associated with the target augmented reality device and processing capability information of the target augmented reality device, wherein the target acquisition module is used to measure the distance of the target object to obtain distance information, and the distance information is used to determine the target movement trajectory of the target object in the real 3D space; based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device, the real 3D space is rasterized to obtain a set of real cells, and the virtual 3D space is rasterized to obtain a set of virtual cells. In this embodiment, the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device can be obtained in advance. Then, based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device, the real three-dimensional space and the virtual three-dimensional space are rasterized. For example, in practical applications, the target acquisition module and / or the target augmented reality device can be initialized by software. For example, the system processing module CPU, GPU, etc. can be initialized, and then a hardware benchmarking program can be started to evaluate the hardware computing power, 3D display capability, and current network operation capability of the target acquisition module and / or the target augmented reality device. Image processing capability can also be evaluated. The network operation capability, computing power, image processing capability, and 3D display capability are integrated to calculate a general score, that is, the hardware capability of the system is integrated, and then the appropriate raster is determined to be used to rasterize the real three-dimensional space and the virtual three-dimensional space. This embodiment achieves the goal of determining the corresponding rasterization standard by combining the hardware capabilities of the system. This allows the rasterization to be matched with the processing capabilities of the target acquisition module and the target enhancement display device, thereby improving the efficiency and accuracy of the system operation.

[0052] In an optional embodiment, the step of rasterizing the real three-dimensional space to obtain the set of real cells and rasterizing the virtual three-dimensional space to obtain a set of virtual cells based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device includes: determining the value of a target processing capability parameter based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device; rasterizing the real three-dimensional space according to a first cell size corresponding to the value of the target processing capability parameter to obtain the set of real cells and rasterizing the virtual three-dimensional space to obtain a set of virtual cells, wherein the value of the target processing capability parameter is negatively correlated with the first cell size; or, rasterizing the real three-dimensional space according to a second cell size corresponding to the value range of the target processing capability parameter to obtain the set of real cells and rasterizing the virtual three-dimensional space to obtain a set of virtual cells, wherein the boundary value of the value range of the target processing capability parameter is negatively correlated with the second cell size. In this embodiment, based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device, the value of the target processing capability parameter is determined. For example, a general score is calculated by integrating the current network operation capability, computing power, image processing capability, and 3D display capability. Then, a corresponding first cell size is set according to this value, and the real three-dimensional space and the virtual three-dimensional space are rasterized to obtain a set of real cells and a set of virtual cells. The value of the target processing capability parameter is negatively correlated with the first cell size, that is, the larger the value of the target processing capability parameter, the smaller the first cell size can be as needed to achieve the purpose of more refined rasterization. Alternatively, a corresponding second cell size is set according to the value range of the target processing capability parameter, and then the real three-dimensional space and the virtual three-dimensional space are rasterized. Similarly, the boundary value of the value range is negatively correlated with the second cell size, that is, the larger the boundary value of the value range, the smaller the second cell size can be as needed to achieve the purpose of more refined rasterization. This embodiment achieves the goal of determining the unit grid size in rasterization based on the value of the target processing capability parameter.

[0053] In an optional embodiment, determining the set of real cells traversed by the target movement trajectory within the set of real cells includes: searching for the cell containing the position of the target movement trajectory within the set of real cells to obtain the set of real cells. In this embodiment, by searching for the cell containing the position of the target movement trajectory within a set of real cells, the set of real cells is obtained, i.e., the set of real cells traversed by the target movement trajectory is determined. Then, based on the mapping relationship between real 3D space and virtual 3D space, the first set of virtual cells traversed by the movement trajectory of the cursor (or other object corresponding to the target object in real space, such as a cartoon character or cartoon image) in virtual 3D space can be determined. Through this embodiment, the purpose of determining the set of real cells traversed by the target movement trajectory in real 3D space is achieved, thereby further enabling the determination of the movement trajectory of the cursor in virtual 3D space.

[0054] In an optional embodiment, determining whether the target movement trajectory triggers a target interaction operation of the target virtual interactive object based on the positional relationship between the first virtual cell set and the second virtual cell set includes: when a target response range is set for the target virtual interactive object in the virtual three-dimensional space, and the target response range includes a third virtual cell set in the virtual three-dimensional space, detecting whether virtual cells in the first virtual cell set are located inside the third virtual cell set, wherein the second virtual cell set is located inside the third virtual cell set in the virtual three-dimensional space; when a subset of virtual cells in the first virtual cell set is detected to be located inside the third virtual cell set, determining whether the target movement trajectory triggers a target interaction operation of the target virtual interactive object based on the positional relationship between the subset of virtual cells in the first virtual cell set and the second virtual cell set; or when no target response range is set for the target virtual interactive object in the virtual three-dimensional space, determining whether the target movement trajectory triggers a target interaction operation of the target virtual interactive object based on the positional relationship between the first virtual cell set and the second virtual cell set. In this embodiment, when a target response range is set for the target virtual interactive object in the virtual 3D space, and the target response range includes a third set of virtual cells in the virtual 3D space (for example, the third set of virtual cells consists of multiple virtual cells in the virtual 3D space whose distance from the target virtual interactive object is a target threshold (e.g., 1 virtual cell, 5 virtual cells, or other values)), it is then detected whether the virtual cells in the first set of virtual cells are located inside the third set of virtual cells. If it is detected that a subset of virtual cells (such as one or more virtual cells) included in the first set of virtual cells is located inside the third set of virtual cells, the results are determined based on the virtual cell subset and the second set of virtual cells. The positional relationship between the sets determines whether the target movement trajectory triggers the target interaction operation of the target virtual interactive object. In this embodiment, a response area is set for the target virtual interactive object (such as an icon) in the virtual three-dimensional space. The operation of the target object, such as a confirmation action or key response, is only monitored when the cursor moves to the response area. This can avoid the problem of possible misoperation in actual operation and improve the interaction efficiency. Optionally, if no target response range is set for the target virtual interactive object (such as an icon) in the virtual three-dimensional space, the positional relationship between the first set of virtual cells and the second set of virtual cells can be used to determine whether the target movement trajectory triggers the target interaction operation of the target virtual interactive object. Figure 4 This is a schematic diagram illustrating the response range of a target virtual interactive object according to an embodiment of the present invention. Figure 4 In the diagram, B represents the cursor that moves with the target object in the real three-dimensional space, C' represents the aforementioned virtual interactive object, and the shaded virtual cell area in the figure corresponds to the aforementioned area. Figure 4 The response area is defined as one virtual cell away from the target virtual interactive object. It should be noted that... Figure 4 This is merely an example. In practical applications, other response ranges can be set according to the needs of different application scenarios. For example, the response area can be set to a distance of 5 or 10 virtual cells from the target virtual interactive object. Through this embodiment, the purpose of determining whether the target movement trajectory triggers the target interactive operation of the target virtual interactive object is achieved based on the positional relationship between the first set of virtual cells and the second set of virtual cells.

[0055] In an optional embodiment, determining whether the target movement trajectory triggers the target interaction operation of the target virtual interaction object based on the positional relationship between the virtual cell subset in the first virtual cell set and the second virtual cell set includes: determining whether the virtual cell subset in the first virtual cell set is located in the second virtual cell set; if it is determined that there are virtual cells in the virtual cell subset located in the second virtual cell set, determining that the target movement trajectory triggers the target interaction operation of the target virtual interaction object; or determining the number of virtual cells in the first virtual cell set located in the second virtual cell set; if the number of virtual cells in the virtual cell subset located in the second virtual cell set is greater than or equal to a predetermined number threshold, determining that the target movement trajectory triggers the target interaction operation of the target virtual interaction object; or determining whether the virtual cell subset in the first virtual cell set is located in the second virtual cell set; if it is determined that there are virtual cells in the virtual cell subset located in the second virtual cell set, and the dwell time of the virtual cells in the virtual cell subset in the second virtual cell set is greater than or equal to a predetermined duration threshold, determining that the target movement trajectory triggers the target interaction operation of the target virtual interaction object.In this embodiment, it can be determined whether the target movement trajectory has triggered the target interaction operation of the target virtual interaction object by judging whether the aforementioned subset of virtual cells is located in the second set of virtual cells. For example, when it is determined that there are virtual cells in the subset of virtual cells located in the second set of virtual cells, it is determined that the target movement trajectory has triggered the target interaction operation of the target virtual interaction object. Optionally, it can also be determined whether the target movement trajectory has triggered the target interaction operation of the target virtual interaction object by determining the number of virtual cells in the subset of virtual cells located in the second set of virtual cells. For example, when it is determined that the number of virtual cells in the subset of virtual cells located in the second set of virtual cells is greater than or equal to a predetermined number threshold (e.g., 2, 3, or other numbers), then... By determining that the target movement trajectory triggers the target interaction operation of the target virtual interactive object, this embodiment avoids potential misoperation issues in actual operation, thereby improving interaction efficiency. Optionally, it can also be determined that the target movement trajectory has triggered the target interaction operation of the target virtual interactive object when it is determined that there is a virtual cell in the second virtual cell set within the subset of virtual cells in the first virtual cell set, and the duration of the virtual cell in the subset of virtual cells in the second virtual cell set is greater than or equal to a predetermined duration threshold. For example, when the cursor in the virtual 3D space moves to the second virtual cell set where the target virtual interactive object is located, and the duration of the cursor in the second virtual cell set is greater than or equal to a predetermined duration threshold (such as 1 second, 0.5 seconds, or other durations), then it is determined that the target movement trajectory has triggered the target interaction operation of the target virtual interactive object. Through this embodiment, different methods can be set to determine whether the target movement trajectory triggers the target interaction operation of the target virtual interactive object, achieving the goal of flexibly setting different interaction operation methods according to different application scenarios, thus broadening the application scope of the interaction method.

[0056] In an optional embodiment, the method further includes: searching the interaction logs between the target object and the target virtual interactive object in the log information of the target augmented reality device; determining abnormal interaction operations between the target object and the target virtual interactive object based on the interaction logs; and adjusting the triggering conditions of the target interaction operation based on the abnormal interaction operation. In this embodiment, by searching the interaction logs between the target object and the target virtual interactive object, the abnormal interaction operation is determined, and then the triggering conditions of the target interaction operation are adjusted based on the abnormal interaction operation. For example, in practical applications, erroneous operations can be recorded to modify the mapping equation between real three-dimensional space and virtual three-dimensional space, thereby achieving a hierarchical optimization effect for complex control.

[0057] Obviously, the embodiments described above are only some embodiments of the present invention, and not all embodiments. The present invention will be specifically described below with reference to specific embodiments.

[0058] Figure 5 This is an overall flowchart of a three-dimensional space interaction method according to a specific embodiment of the present invention, such as... Figure 5 As shown, the process includes the following steps:

[0059] S502, initialization is performed using the initialization module, that is, the hardware included in the interactive system is initialized and started. For example, the hardware may include a target acquisition module and a target augmented reality device, etc.

[0060] S504 uses the ranging object locking module to lock the ranging object (corresponding to the target object in the aforementioned real three-dimensional space). The ranging standard point is used as the tracking point for three-dimensional space perception, that is, the position of the mark is locked, and the movement trajectory of the mark is measured. When the target acquisition module includes multiple ranging modules, the average error processing of multiple sets of ranging data is performed, that is, the composite ranging data is integrated to achieve a better fit with reality.

[0061] S506 uses a three-dimensional coordinate system correspondence module to perform calculations on composite ranging data, average the three-dimensional coordinate system error, fit the control motion curve, and interact with the virtual three-dimensional display interface.

[0062] The S508, through its movement unit setting module, sets movement restriction areas according to system capabilities. This determines which operation method is more suitable for each situation and can be adjusted based on the user's specific actions. For example, one driving principle exists for icon confirmation, while another exists for design and drawing operations. The movement trajectory and user habits differ depending on the operating mode. For instance, taking icon confirmation as an example, to prevent accidental operation when the cursor approaches the icon (target), a response range (or response area) is drawn in three-dimensional space. Confirmation or button response is only monitored when the cursor enters the response area, thus improving confirmation efficiency. Because operation takes place in a three-dimensional environment, user scenarios are easily identifiable, making it much easier to refine control methods according to different usage scenarios than mouse-like operations, resulting in a more intelligent and improved user experience. Unlike linear settings in mouse operations, the movement unit setting here is an intelligent setting that includes adaptive action recognition, designed to provide the best interactive experience for different application scenarios.

[0063] S510 further captures the action of the target object through the motion or ranging sensor capture module. For example, taking chart confirmation as an example, when the movement of the target object (such as a finger) is mapped to the virtual three-dimensional space and the cursor has touched the virtual interactive object (or the above-mentioned response area), the action of the target object is further captured to determine whether to respond to the confirmation action, which can improve the efficiency of confirmation.

[0064] S512 controls the final confirmation of the movement through the confirmation feedback module. This process can be performed on a handheld device or as a separate gesture operation.

[0065] Optionally, for some application scenarios where the accuracy requirement for confirming whether an interactive operation has been triggered is not very high, it can be done according to... Figure 6 The process shown is implemented as follows: Figure 6 This is an overall flowchart of a three-dimensional spatial interaction method according to another specific embodiment of the present invention. Taking a system including a UWB module, millimeter wave, and optical composite ranging as an example, the system includes an initialization module, a ranging object locking module, a three-dimensional coordinate system correspondence module, a movement unit setting module, and a confirmation feedback module. This process is similar to... Figure 5 Compared to the previous process, steps in S510 are omitted. For example, in practical applications, when the movement of the target object (such as a finger) is mapped to the virtual 3D space showing that the cursor has touched the virtual interactive object (or the aforementioned response area), it is considered that the target interactive operation has been triggered, while the other steps are respectively... Figure 5 The steps are the same as in the middle, specifically including the following steps:

[0066] S602, initialization is performed using the initialization module, that is, the hardware included in the interactive system is initialized and started. For example, the hardware may include a target acquisition module and a target augmented reality device, etc.

[0067] S604 uses the ranging object locking module to lock the ranging object (corresponding to the target object in the aforementioned real three-dimensional space). The ranging standard point is used as the tracking point for three-dimensional space perception, that is, the position of the mark is locked, and the movement trajectory of the mark is measured. When the target acquisition module includes multiple ranging modules, the average error processing of multiple sets of ranging data is performed, that is, the composite ranging data is integrated to achieve a better fit with reality.

[0068] S606 uses a three-dimensional coordinate system correspondence module to perform calculations on composite ranging data, average the three-dimensional coordinate system error, fit the control motion curve, and interact with the virtual three-dimensional display interface.

[0069] The S608, through its movement unit setting module, sets movement restriction areas according to system capabilities. This determines which operation method is more suitable for each situation and can be adjusted based on the user's specific actions. For example, one driving principle exists for icon confirmation, while another exists for design and drawing operations. The movement trajectory and user habits differ depending on the operating mode. For instance, taking icon confirmation as an example, to prevent accidental operation when the cursor approaches the icon (target), a response range (or response area) is drawn in three-dimensional space. Confirmation or button response is only monitored when the cursor enters the response area, thus improving confirmation efficiency. Because operation takes place in a three-dimensional environment, user scenarios are easily identifiable, making it much easier to refine control methods according to different usage scenarios than mouse-like operations, resulting in a more intelligent and improved user experience. Unlike linear settings in mouse operations, the movement unit setting here is an intelligent setting that includes adaptive action recognition, designed to provide the best interactive experience for different application scenarios.

[0070] S610 controls the final confirmation of movement through the confirmation feedback module. This process can be performed on a handheld device or as a separate gesture operation.

[0071] The execution process of the different modules involved in the above steps is explained below:

[0072] Figure 7 This is a flowchart of the initialization module according to a specific embodiment of the present invention. The initialization module is the first module to start the system, and its main function is to identify the hardware devices involved in 3D acquisition, identify the hardware computing power of the acquisition devices and the 3D display capabilities of the display devices, and integrate the capabilities achieved by the hardware. The process includes the following steps:

[0073] S702, software initialization;

[0074] S704, the detection and ranging hardware module. Taking a composite ranging system that includes a UWB module, a millimeter wave module, and an optical module as an example, the detection hardware module will detect these three ranging modules and initialize and start the hardware.

[0075] S706, hardware acquisition and calculation, image hardware processing, hardware detection, that is, detecting other hardware modules and initializing and starting other hardware.

[0076] These three modules can be used to measure the distance to the marker point and establish a corresponding three-dimensional coordinate system. The relative position of the three-dimensional coordinate system can be averaged based on the test data to reduce spatial systematic errors.

[0077] S708, start the hardware benchmarking program to evaluate network performance, hardware computing power, image processing power and 3D display capabilities, etc.

[0078] S710, to obtain network capability score;

[0079] S712, obtains the image processing capability score;

[0080] S714, the comprehensive score, is a general score calculated by integrating the current network operation capabilities, computing power, image processing capabilities, and 3D display capabilities, and then mapping it to the corresponding processing grid (the grid concept in 2D is similar to the unit cube concept in 3D space).

[0081] Figure 8 This is a flowchart of the ranging object locking module according to a specific embodiment of the present invention. This module mainly targets the ranging standard point as a three-dimensional spatial perception tracking point. The mark can be an optical tag, a UWB tag, or a millimeter-wave relay beacon point (not limited to the above forms), or a composite product of these tags. It mainly provides accurate marks for three-dimensional ranging. The process includes the following steps:

[0082] S802, Initialize the ranging module (corresponding to the aforementioned ranging module);

[0083] S804, lock the punctuation mark position;

[0084] S806, the trajectory of the distance measuring marker;

[0085] S808 converts the motion trajectory into driving information for the display cursor;

[0086] S810, integrate data from other ranging modules. When the system includes multiple ranging modules, this step can integrate the ranging data from multiple ranging modules, which can make the tracking of the ranging point more accurate and obtain a more accurate tracking point trajectory.

[0087] This module operates similarly to Kinect, where tracking points are calibrated by waving or using a rangefinder. However, this embodiment uses an optical tag, a UWB tag, or a millimeter-wave relay beacon, which makes tracking more accurate. It can track the trajectory of the calibrated point well without the need for optical shape recognition, and it can average the error of various ranging data to achieve a better fit with reality.

[0088] Figure 9 This is a flowchart of the three-dimensional coordinate system correspondence module according to a specific embodiment of the present invention. The three-dimensional coordinate system correspondence module mainly performs calculations on the composite ranging data, averages the three-dimensional coordinate system error, fits the control motion curve, and interacts with the display interface. The process includes the following steps:

[0089] S902, based on the comprehensive score obtained in S714 of the aforementioned initialization module workflow, inform the acquisition system to select 3D grid sampling information, that is, determine 3D grid sampling information based on the hardware capabilities of the acquisition module.

[0090] S904, determine the minimum 3D display grid corresponding to the score area, that is, determine the minimum display grid unit corresponding to the score area, that is, the minimum grid size that can be displayed;

[0091] S906, begin collecting and mapping reality, that is, while collecting ranging information, map the ranging results of the real three-dimensional space to the virtual three-dimensional space;

[0092] The S908 encapsulates the distance measurement results in real three-dimensional space to output standard format data.

[0093] The above steps complete the format setting and control unit processing grid of 3D reality (the grid concept in 2D is similar to the unit cube concept in 3D space), thereby completing the operation and feedback scene of real-time motion and 3D virtual reality.

[0094] The confirmation process of the confirmation feedback module in step S512 or step S610 is described below. Figure 10 This is an example diagram of the operation of a confirmation feedback module according to a specific embodiment of the present invention. This example takes a single gesture operation as an example, and the process includes the following steps:

[0095] S1002, Module initialization;

[0096] S1004, the module enters a sleep timer state;

[0097] S1006, perform acceleration detection, that is, detect the acceleration of the target object (such as a finger) movement;

[0098] S1008, perform gesture recognition;

[0099] S1010, determine whether it is a valid gesture;

[0100] S1012, if the judgment result is a valid gesture, send information encoding to confirm the gesture operation, and then enter the sleep timer state; if the judgment result is negative, that is, if it is not a valid gesture, continue to perform the acceleration detection of the gesture action.

[0101] Figure 11 This is an example diagram of the operation of a confirmation feedback module according to another specific embodiment of the present invention. This example takes a handheld device performing confirmation as an example, and the process includes the following steps:

[0102] S1102, Module initialization;

[0103] S1104, the module enters a sleep timer state;

[0104] S1106, perform signal detection to detect operational actions;

[0105] S1108, the curve for fitting the operation action;

[0106] S1110, determine whether it is a valid code;

[0107] S1112: If the judgment result is yes, a control signal is transmitted to the host, and then the system enters a sleep timer state. If the judgment result is no, that is, if the operation is invalid, the signal detection continues.

[0108] In the above embodiments of the present invention, the operation of the cursor (comment point) is a feedback operation based on three-dimensional sampling data; after the system is started, the embodiments of the present invention need to initialize the hardware for recognizing and measuring the distance in the real scene and match the three-dimensional screen capability for real-world operation, which is different from the homogenization of the mouse operation interface in the existing backbone technology; after initialization, the software in the embodiments of the present invention will rasterize the displayed three-dimensional space units (similar to a cubic grid of pixels), and use these unit grids as the smallest unit for cursor (comment point) operation. This rasterization depends on the sampling capability of the acquisition device and the processing capability of the display device. The system performs a balancing calculation on this through the movement unit setting module; in the embodiments of the present invention, in order to prevent the cursor from approaching the icon (target) and causing erroneous operation, the image... The target will draw a response range in three-dimensional space. Only when the cursor enters the response area will the confirmation action or key response be monitored, thus improving the confirmation efficiency. The confirmation feedback module will promptly prompt the operator with sound, light, touch and other devices based on the response of the screen, and record the operation error so as to modify the mapping method equation, so as to achieve the effect of hierarchical optimization of complex control. In addition, in the embodiments of the present invention, as a three-dimensional spatial perception tracking point, the target can be an optical tag, a UWB tag, or a millimeter wave relay beacon (not limited to the above forms), or a composite product of these tags. It mainly provides accurate targets for three-dimensional ranging, and the control trajectory is mapped by the movement trajectory of the target in three-dimensional space.

[0109] In related technologies, Microsoft's Kinect system solves the problem of mapping and interaction between real and virtual reality spaces. However, due to the camera's resolution and the fact that interaction is performed on a 2D interface, it suffers from slow confirmation speed, inability to perform fine-grained operations, and a lack of feedback mechanisms for complex application scenarios. Compared to existing technologies, this invention overcomes the shortcomings of the Kinect system. It is a spatial mapping interaction based on high-precision 3D ranging, utilizing more advanced ranging and angle measurement technologies, such as UWB and millimeter waves, to establish a standard mapping between virtual and real spaces. Based on this, it employs hand motion tracking or peripheral devices for confirmation, thus solving the interaction efficiency problem. This invention is an interactive system that enables fine-grained operations similar to a mouse in 3D space.

[0110] Through the embodiments of this invention, an AR (VR) interaction method based on a high-precision coordinate system is presented. Compared to the current Kinect system, which uses hardware to acquire 3D data and map it to a virtual reality coordinate system, Kinect uses a binocular (or multi-view) system to synthesize a 3D system using algorithms. The interaction method of this invention is based on a high-precision coordinate system, which offers higher accuracy than the single (binocular) 3D system constructed by Kinect. Furthermore, the confirmation interface is in 3D space, not just 2D surface displacement confirmation. Therefore, this invention has significant advantages in terms of timeliness and accuracy in mapping to virtual reality. From a user experience perspective, the interaction method of this invention enables more realistic operation experiences such as double-clicking icons, motion proximity, and touch control in 3D scenes. The interaction method also shows significant improvements in confirmation speed and accuracy. It is an interaction method that follows the operating system upgrade after the 3D acquisition system has been precisely improved. In this implementation process, the program relies heavily on real-time measured 3D spatial data and adjusts the measurement refinement unit (a grid concept in 2D, similar to a unit cube concept in 3D space) according to the approach of the action and the target, so as to provide more refined feedback to the hand (handle or sensor) approaching the target.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0112] This embodiment also provides an interactive device in three-dimensional space, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0113] Figure 12 This is a structural block diagram of an interactive device in three-dimensional space according to an embodiment of the present invention, such as... Figure 12 As shown, the device includes:

[0114] The first display module 1202 is used to display a target augmented reality 3D image on the target augmented reality device, and to display a target virtual interactive object in the virtual 3D space of the target augmented reality 3D image. The target augmented reality 3D image is a 3D image formed by superimposing a real 3D image captured in real 3D space with a virtual 3D image in the virtual 3D space. The real 3D space is rasterized into a set of real cells, and the virtual 3D space is rasterized into a set of virtual cells. There is a mapping relationship between the set of real cells and the set of virtual cells.

[0115] The first acquisition module 1204 is used to acquire the target movement trajectory of the target object in the real three-dimensional space when the target object is identified in the real three-dimensional space.

[0116] The first determining module 1206 is used to determine the set of real cells through which the target movement trajectory passes in the set of real cells;

[0117] The second determining module 1208 is used to determine, based on the mapping relationship between the set of real cells and the set of virtual cells, the first set of virtual cells that corresponds to the set of real cells that have been passed in the set of virtual cells.

[0118] The third determining module 1210 is used to determine whether the target movement trajectory triggers the target interaction operation of the target virtual interaction object based on the positional relationship between the first virtual cell set and the second virtual cell set, wherein the second virtual cell set includes the virtual cells occupied by the target virtual interaction object in the set of virtual cells;

[0119] The second display module 1212 is used to display the screen corresponding to the execution of the target interaction operation on the target augmented reality device when the target movement trajectory triggers the target interaction operation of the target virtual interaction object.

[0120] In an optional embodiment, the acquisition module 1204 includes: a first acquisition unit, configured to acquire ranging information obtained by the target acquisition module measuring the distance to the target object at each of a set of times; a first determination unit, configured to determine the position of the target object in the real three-dimensional space at each time based on the ranging information acquired at each time; and a fitting unit, configured to fit the position of the target object in the real three-dimensional space at each time into the target movement trajectory.

[0121] In an optional embodiment, the first determining unit includes: an execution subunit, configured to perform the following operations for the ranging information acquired at each time point, wherein, when performing the following operations, each time point is the current time point: when the target acquisition module is a ranging module and the ranging information acquired at the current time point represents the current distance between the target object and a preset marker point in the real three-dimensional space, the position of the target object in the real three-dimensional space at the current time point is determined based on the position of the preset marker point in the real three-dimensional space and the current distance represented by the ranging information; when the target acquisition module includes multiple ranging modules and the current time point... The acquired ranging information includes multiple ranging information obtained by the multiple ranging modules measuring the target object at the current time, and each of the multiple ranging information represents the current distance between the target object and the preset marker point in the real three-dimensional space. Based on the position of the preset marker point in the real three-dimensional space and the multiple current distances represented by the multiple ranging information, multiple positions of the target object in the real three-dimensional space at the current time are determined. Based on the multiple positions of the target object in the real three-dimensional space at the current time, the position of the target object in the real three-dimensional space at the current time is determined.

[0122] In an optional embodiment, the execution subunit may determine the position of the target object in the real three-dimensional space at the current moment by averaging the three-dimensional coordinates of the plurality of positions in the real three-dimensional space to obtain a first target three-dimensional coordinate, and determining the position represented by the first target three-dimensional coordinate in the real three-dimensional space as the position of the target object in the real three-dimensional space at the current moment; or by performing a weighted average of the three-dimensional coordinates of the plurality of positions in the real three-dimensional space to obtain a second target three-dimensional coordinate, and determining the position represented by the second target three-dimensional coordinate in the real three-dimensional space as the position of the target object in the real three-dimensional space at the current moment.

[0123] In an optional embodiment, the above apparatus further includes: a second acquisition module, configured to acquire hardware resource information of a target acquisition module associated with the target augmented reality device and processing capability information of the target augmented reality device before displaying the target augmented reality 3D image on the target augmented reality device, wherein the target acquisition module is configured to measure the distance of the target object to obtain distance information, and the distance information is used to determine the target movement trajectory of the target object in the real 3D space; and a rasterization module, configured to rasterize the real 3D space according to the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device to obtain the set of real cells, and rasterize the virtual 3D space to obtain a set of virtual cells.

[0124] In an optional embodiment, the rasterization module includes: a second determining unit, configured to determine the value of a target processing capability parameter based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device; a first rasterization unit, configured to rasterize the real three-dimensional space according to a first cell size corresponding to the value of the target processing capability parameter to obtain the set of real cells, and rasterize the virtual three-dimensional space to obtain a set of virtual cells, wherein the value of the target processing capability parameter is negatively correlated with the first cell size; or, a second rasterization unit, configured to rasterize the real three-dimensional space according to a second cell size corresponding to the value range of the target processing capability parameter to obtain the set of real cells, and rasterize the virtual three-dimensional space to obtain a set of virtual cells, wherein the boundary value of the value range of the target processing capability parameter is negatively correlated with the second cell size.

[0125] In an optional embodiment, the first determining module 1206 includes: a search unit, used to search for the cell containing the location of the target movement trajectory in the set of real cells, to obtain the set of real cells.

[0126] In an optional embodiment, the third determining module 1210 includes: a first detection unit, configured to detect whether a virtual cell in the first virtual cell set is located inside the third virtual cell set when a target response range is set for the target virtual interactive object in the virtual three-dimensional space and the target response range includes a third virtual cell set in the virtual three-dimensional space, wherein the second virtual cell set is located inside the third virtual cell set in the virtual three-dimensional space; a third determining unit, configured to determine whether the target movement trajectory triggers a target interaction operation of the target virtual interactive object based on the positional relationship between the virtual cell subset in the first virtual cell set and the second virtual cell set when a subset of virtual cells in the first virtual cell set is detected to be located inside the third virtual cell set; or, a fourth determining unit, configured to determine whether the target movement trajectory triggers a target interaction operation of the target virtual interactive object based on the positional relationship between the first virtual cell set and the second virtual cell set when no target response range is set for the target virtual interactive object in the virtual three-dimensional space.

[0127] In an optional embodiment, the third determining unit includes: a first determining subunit, configured to determine whether the subset of virtual cells in the first virtual cell set is located in the second virtual cell set; a first determining subunit, configured to determine that the target movement trajectory triggered the target interaction operation of the target virtual interaction object when it is determined that there are virtual cells in the subset of virtual cells located in the second virtual cell set; or, a second determining subunit, configured to determine the number of virtual cells in the first virtual cell set located in the second virtual cell set; a third determining subunit, configured to determine the number of virtual cells in the subset of virtual cells located in the second virtual cell set. If the number of virtual cells in the set of cells is greater than or equal to a predetermined threshold, it is determined that the target movement trajectory triggered the target interaction operation of the target virtual interaction object; or, the second determination subunit is used to determine whether the subset of virtual cells in the first set of virtual cells is located in the second set of virtual cells; the fourth determination subunit is used to determine that the target movement trajectory triggered the target interaction operation of the target virtual interaction object if it is determined that there are virtual cells located in the second set of virtual cells in the subset of virtual cells, and the dwell time of the virtual cells in the subset of virtual cells in the second set of virtual cells is greater than or equal to a predetermined duration threshold.

[0128] In an optional embodiment, the above apparatus further includes: a search module, configured to search for interaction logs between the target object and the target virtual interactive object in the log information of the target augmented reality device; a fourth determination module, configured to determine abnormal interaction operations between the target object and the target virtual interactive object based on the interaction logs between the target object and the target virtual interactive object; and an adjustment module, configured to adjust the triggering conditions of the target interaction operation based on the abnormal interaction operation.

[0129] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0130] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to perform the steps in any of the above method embodiments.

[0131] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0132] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0133] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0134] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0135] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0136] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An interaction method in three-dimensional space, characterized in that, include: A target augmented reality 3D image is displayed on the target augmented reality device, and a target virtual interactive object is displayed in the virtual 3D space of the target augmented reality 3D image. The target augmented reality 3D image is a 3D image formed by superimposing a real 3D image captured in real 3D space with a virtual 3D image in the virtual 3D space. The real 3D space is rasterized into a set of real cells, and the virtual 3D space is rasterized into a set of virtual cells. There is a mapping relationship between the set of real cells and the set of virtual cells. When a target object is identified in the real three-dimensional space, the target object's movement trajectory in the real three-dimensional space is obtained; Determine the set of real cells through which the target movement trajectory passes within the set of real cells; Based on the mapping relationship between the set of real cells and the set of virtual cells, determine the first set of virtual cells that corresponds to the set of real cells passed through in the set of virtual cells; Based on the positional relationship between the first set of virtual cells and the second set of virtual cells, it is determined whether the target movement trajectory triggers the target interaction operation of the target virtual interaction object, wherein the second set of virtual cells includes the virtual cells occupied by the target virtual interaction object in the set of virtual cells; When the target movement trajectory triggers the target interaction operation of the target virtual interaction object, the screen corresponding to the execution of the target interaction operation is displayed on the target augmented reality device.

2. The method according to claim 1, characterized in that, The step of obtaining the target object's movement trajectory in the real three-dimensional space includes: The target acquisition module obtains distance measurement information from the target object at each time point in a set of time points; based on the distance measurement information obtained at each time point, the position of the target object in the real three-dimensional space at each time point is determined. The position of the target object in the real three-dimensional space at each time moment is fitted to the target movement trajectory.

3. The method according to claim 2, characterized in that, Determining the position of the target object in the real three-dimensional space at each time step based on the ranging information acquired at each time step includes: For the ranging information acquired at each of the aforementioned time points, the following operations are performed, wherein each of the aforementioned time points is the current time point: When the target acquisition module is a ranging module, and the ranging information obtained at the current moment represents the current distance between the target object and the preset marker point in the real three-dimensional space, the position of the target object in the real three-dimensional space at the current moment is determined according to the position of the preset marker point in the real three-dimensional space and the current distance represented by the ranging information. In a target acquisition module comprising multiple ranging modules, where the ranging information acquired at the current moment includes multiple ranging information obtained by the multiple ranging modules measuring the target object at the current moment, and each of the multiple ranging information represents the current distance between the target object and a preset marker point in the real three-dimensional space, the multiple positions of the target object in the real three-dimensional space at the current moment are determined based on the position of the preset marker point in the real three-dimensional space and the multiple current distances represented by the multiple ranging information; and the position of the target object in the real three-dimensional space at the current moment is determined based on the multiple positions of the target object in the real three-dimensional space at the current moment.

4. The method according to claim 3, characterized in that, Determining the position of the target object in the real three-dimensional space at the current moment based on the multiple positions of the target object in the real three-dimensional space at the current moment includes: The three-dimensional coordinates of the multiple locations in the real three-dimensional space are averaged to obtain the three-dimensional coordinates of the first target. The position represented by the three-dimensional coordinates of the first target in the real three-dimensional space is determined as the position of the target object in the real three-dimensional space at the current moment; or The weighted average of the three-dimensional coordinates of the multiple locations in the real three-dimensional space is used to obtain the three-dimensional coordinates of the second target. The position represented by the three-dimensional coordinates of the second target in the real three-dimensional space is determined as the position of the target object in the real three-dimensional space at the current moment.

5. The method according to claim 1, characterized in that, Before displaying the target augmented reality 3D image on the target augmented reality device, the method further includes: The hardware resource information of the target acquisition module associated with the target augmented reality device and the processing capability information of the target augmented reality device are acquired. The target acquisition module is used to measure the distance to the target object to obtain distance information, which is used to determine the target object's movement trajectory in the real three-dimensional space. Based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device, the real three-dimensional space is rasterized to obtain a set of real cells, and the virtual three-dimensional space is rasterized to obtain a set of virtual cells.

6. The method according to claim 5, characterized in that, The method involves rasterizing the real 3D space based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device to obtain a set of real cell units, and rasterizing the virtual 3D space to obtain a set of virtual cell units, including: The value of the target processing capability parameter is determined based on the hardware resource information of the target acquisition module and the processing capability information of the target augmented reality device. The real 3D space is rasterized according to the first cell size corresponding to the value of the target processing capability parameter to obtain the set of real cells, and the virtual 3D space is rasterized to obtain a set of virtual cells, wherein the value of the target processing capability parameter is negatively correlated with the first cell size; or, The real three-dimensional space is rasterized according to the second cell size corresponding to the value range of the target processing capability parameter to obtain the set of real cells, and the virtual three-dimensional space is rasterized to obtain a set of virtual cells, wherein the boundary value of the value range of the target processing capability parameter is negatively correlated with the second cell size.

7. The method according to claim 1, characterized in that, Determining the set of real-world cells traversed by the target movement trajectory within the set of real-world cells includes: The set of real cells is obtained by finding the cell containing the location of the target's movement trajectory within the set of real cells.

8. The method according to any one of claims 1 to 7, characterized in that, The step of determining whether the target movement trajectory triggers the target interaction operation of the target virtual interaction object based on the positional relationship between the first virtual cell set and the second virtual cell set includes: In the virtual 3D space, a target response range is set for the target virtual interactive object, and the target response range includes a third set of virtual cells in the virtual 3D space. The system detects whether virtual cells in the first set of virtual cells are located inside the third set of virtual cells, wherein the second set of virtual cells is located inside the third set of virtual cells in the virtual 3D space. If a subset of virtual cells in the first set of virtual cells is detected to be inside the third set of virtual cells, the system determines whether the target movement trajectory triggers a target interaction operation of the target virtual interactive object based on the positional relationship between the subset of virtual cells in the first set and the second set of virtual cells. Or... If no target response range is set for the target virtual interactive object in the virtual three-dimensional space, it is determined whether the target movement trajectory triggers the target interactive operation of the target virtual interactive object based on the positional relationship between the first virtual cell set and the second virtual cell set.

9. The method according to claim 8, characterized in that, The step of determining whether the target movement trajectory triggers the target interaction operation of the target virtual interaction object based on the positional relationship between the subset of virtual cells in the first virtual cell set and the second virtual cell set includes: Determine whether the subset of virtual cells in the first set of virtual cells is located in the second set of virtual cells; if it is determined that there is a virtual cell in the subset of virtual cells located in the second set of virtual cells, determine that the target movement trajectory triggered the target interaction operation of the target virtual interaction object; or Determine the number of virtual cells in the second virtual cell set that are located within the subset of virtual cells in the first virtual cell set; if the number of virtual cells in the subset of virtual cells that are located within the second virtual cell set is greater than or equal to a predetermined threshold, determine that the target movement trajectory triggered the target interaction operation of the target virtual interaction object; or Determine whether the subset of virtual cells in the first set of virtual cells is located in the second set of virtual cells; if it is determined that there is a virtual cell in the subset of virtual cells located in the second set of virtual cells, and the duration of the virtual cell in the subset of virtual cells in the second set of virtual cells is greater than or equal to a predetermined duration threshold, determine that the target movement trajectory has triggered the target interaction operation of the target virtual interaction object.

10. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Search the interaction logs between the target object and the target virtual interactive object in the log information of the target augmented reality device; Based on the interaction logs between the target object and the target virtual interaction object, determine the abnormal interaction operations between the target object and the target virtual interaction object; Based on the abnormal interaction operation, the triggering conditions of the target interaction operation are adjusted.

11. An interactive device in three-dimensional space, characterized in that, include: The first display module is used to display a target augmented reality 3D image on the target augmented reality device, and to display a target virtual interactive object in the virtual 3D space of the target augmented reality 3D image. The target augmented reality 3D image is a 3D image formed by superimposing a real 3D image captured in real 3D space with a virtual 3D image in the virtual 3D space. The real 3D space is rasterized into a set of real cells, and the virtual 3D space is rasterized into a set of virtual cells. There is a mapping relationship between the set of real cells and the set of virtual cells. The first acquisition module is used to acquire the target movement trajectory of the target object in the real three-dimensional space when the target object is identified in the real three-dimensional space. The first determining module is used to determine the set of real cells through which the target movement trajectory passes in the set of real cells; The second determining module is used to determine, based on the mapping relationship between the set of real cells and the set of virtual cells, the first set of virtual cells that has been passed and corresponds to the set of real cells that has been passed; The third determining module is used to determine whether the target movement trajectory triggers the target interaction operation of the target virtual interaction object based on the positional relationship between the first virtual cell set and the second virtual cell set, wherein the second virtual cell set includes the virtual cells occupied by the target virtual interaction object in the set of virtual cells; The second display module is used to display the screen corresponding to the execution of the target interaction operation on the target augmented reality device when the target movement trajectory triggers the target interaction operation of the target virtual interaction object.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program is executed by a processor to perform the method of any one of claims 1 to 10.

13. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 10 through the computer program.

Images