An interactive method and device for holographic sandbox

Abstract

The present application discloses an interactive method and device applied to a holographic sandbox, which creates a handle object, wherein the handle object is an object corresponding to an interactive handle in an image displayed on the holographic sandbox, and the interactive handle is used to capture the movements of a user's hand; obtains the posture information of the interactive handle; uses the posture information of the interactive handle to assign control parameters of the handle object, so that the posture of the handle object in the image displayed on the holographic sandbox matches the posture of the interactive handle; generates rays in the extension direction of the handle object; determines the intersection point of the layer where the ray and the image corresponding to the movable object are located as the end point of the ray; the image displayed on the holographic sandbox includes the image of the movable object and the image of the immovable object; collects the interactive information generated by the interactive handle; and interacts with the image of the movable object according to the interactive information.

Description

Technical Field

[0001] The present application relates to the field of simulation display technology, and in particular to an interactive method and device applied to a holographic sandbox. Background Art

[0002] Front-Projected Holographic Display, also known as virtual imaging technology, is a technology that uses the principles of interference and diffraction to record and reproduce the real three-dimensional image of an object. It has the advantage of being able to meet all the perceptions of human vision, and viewers can even watch three-dimensional images without the help of auxiliary devices such as helmets and glasses. With the continuous development of display technology, holographic display technology has received more and more attention.

[0003] Currently, common holographic projection, 3D display and other products on the market are usually only used to display specific images to users and cannot interact with users. Summary of the invention

[0004] The embodiments of the present application provide an interactive method and device applied to a holographic sandbox to at least partially solve the above-mentioned technical problems.

[0005] The present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides an interactive method applied to a holographic sandbox, the method comprising:

[0007] Create a handle object, where the handle object is an object corresponding to an interactive handle in the image displayed by the holographic sandbox, and the interactive handle is used to capture the user's hand movements;

[0008] Get the position information of the interactive handle;

[0009] Using the posture information of the interactive handle to assign control parameters of the handle object, so that the posture of the image of the handle object displayed on the sand table matches the posture of the interactive handle;

[0010] generating a ray in the extension direction of the handle object;

[0011] Determine the intersection point of the layer where the ray and the image corresponding to the movable object are located as the end point of the ray; the image displayed on the holographic sand table includes the image of the movable object and the image of the immovable object;

[0012] Collecting interaction information generated by the interactive handle;

[0013] Interact with the image of the movable object according to the interaction information.

[0014] In an optional embodiment of the present specification, the interaction information includes: posture information of the interaction handle; and according to the interaction information, the image interaction with the movable object includes:

[0015] The image of the corresponding movable object presented by the layer intersecting with the ray is highlighted.

[0016] In an optional embodiment of the present specification, after interacting with the image of the movable object according to the interaction information, the method further includes:

[0017] The position information of the interactive handle is used again to assign the control parameters of the handle object;

[0018] regenerate rays in the extension direction of the handle object;

[0019] If the regenerated ray and the layer where the image corresponding to the movable object is located no longer intersect, the highlighting of the image of the movable object is canceled.

[0020] In an optional embodiment of the present specification, the interaction information includes information generated by a user for a selection operation of an interaction handle; and according to the interaction information, interacting with the image of the movable object includes:

[0021] When a selection operation of the interactive handle is detected, the posture of the image of the movable object in the image displayed on the holographic sandbox is adjusted according to the posture information of the interactive handle.

[0022] In an optional embodiment of the present specification, after interacting with the image of the movable object according to the interaction information, the method further includes:

[0023] When a deselection operation of the interactive handle is detected, the position of the image of the movable object in the image displayed on the holographic sandbox is maintained until a selection operation of the interactive handle is detected again.

[0024] In an optional embodiment of the present specification, the layer where the image corresponding to the movable object is located is provided with an interactive interface, and the layer where the image corresponding to the immovable object is located is not provided with an interactive interface.

[0025] In an optional embodiment of this specification, wherein,

[0026] After determining that the layer intersection point where the ray and the image corresponding to the movable object are located is the end point of the ray, the method further includes: adjusting the designated variable of the handle object to a first designated value, wherein the first designated value indicates that the handle object tracks the movable object;

[0027] According to the interaction information, after interacting with the image of the movable object, the method also includes: acquiring the posture information of the interaction handle again; reassigning the control parameters of the handle object using the posture information of the interaction handle acquired again; generating rays again in the extension direction of the handle object; if the regenerated rays do not intersect with the layer where the image corresponding to the movable object is located, adjusting the designated variable of the handle object to a second designated value, the second designated value indicating that the handle object has not tracked the movable object.

[0028] In a second aspect, an embodiment of the present application further provides an interactive device applied to a holographic sandbox, for implementing any one of the methods in the aforementioned first aspect.

[0029] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0030] Processor; and

[0031] A memory arranged to store computer executable instructions which, when executed, cause the processor to perform any of the methods of the first aspect.

[0032] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs, and when the one or more programs are executed by an electronic device including multiple applications, the electronic device executes any one of the methods in the first aspect.

[0033] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: when the user interacts with the image displayed on the holographic sandbox through the interactive handle, a ray for interaction can be generated according to the hand movement. The end point of the ray is the layer where the movable object is located. It can be seen that through the process in this specification, the ray can identify the movable object that can be used for interaction, and avoid the interference of the immovable object in the image on the interaction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0035] Figure 1 A schematic diagram of a holographic display scene provided in an embodiment of this specification;

[0036] Figure 2 A schematic diagram of a holographic display process provided in an embodiment of this specification;

[0037] Figure 3A schematic diagram of the structure of an interactive device applied to a holographic sandbox provided in an embodiment of this specification;

[0038] Figure 4 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, and this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0040] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0041] The serial numbers of the components in this document, such as "second", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0042] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0043] Exemplarily, the interactive method applied to the holographic sandbox in this specification is based on a holographic display system. The holographic display system includes a holographic sandbox, customized dual-viewpoint radio frequency stereo glasses, and a display processing device. Users can watch the holographic images in the holographic sandbox by wearing customized dual-viewpoint radio frequency stereo glasses. In an optional scenario, the scene of the interaction between the holographic display system and the user can be as follows: Figure 1 shown.

[0044] In this specification, the holographic display system can capture the user's hand movements and move the image of the movable object in the holographic picture based on the user's hand movements.

[0045] The interactive method applied to the holographic sandbox in this specification may include the following steps:

[0046] S200: creating a handle object, where the handle object is an object corresponding to an interactive handle in an image displayed on a holographic sandbox, and the interactive handle is used to capture the movements of a user's hand.

[0047] The handle object in this specification is a part of the image (the image is a holographic image) displayed to the user by the holographic sandbox. The process in this specification is executed by the display processing device.

[0048] In an optional embodiment of the present specification, when it is detected that the interactive handle enters the detection range of the holographic sandbox (a preset range of a specified distance from the holographic sandbox, which can be set by the user), this step is executed. When it is detected that the interactive handle leaves the detection range of the holographic sandbox, the handle object is deleted.

[0049] In this manual, the user holds the interactive handle to interact with the images displayed on the holographic sandbox.

[0050] S202: Obtaining the position information of the interactive handle.

[0051] In this specification, the position information of the interactive handle may be collected by the interactive handle itself and sent to the display processing device; or may be actively collected by the display processing device.

[0052] The posture information of the interactive handle may include the position and posture of the interactive handle. In an optional embodiment of the present specification, a coordinate system may be established with the center of the holographic sandbox as the coordinate origin, and the posture of the interactive handle is the posture in the coordinate system.

[0053] S204: Assigning control parameters of the handle object using the posture information of the interactive handle, so that the posture of the image of the handle object displayed on the sandbox matches the posture of the interactive handle.

[0054] Specifically, the posture information of the interactive handle may be used as the control parameter of the handle object.

[0055] Steps S202 and S204 in this specification can be continuously executed, and the position and posture of the handle object can be adjusted in real time according to the position information of the interactive handle, so as to realize the continuous change of the position and posture of the handle object in the image.

[0056] S206: Generate a ray in the extension direction of the handle object.

[0057] The rays in this manual are also part of the image displayed to the user by the holographic sandbox.

[0058] S208: Determine the intersection point of the layer where the ray and the image corresponding to the movable object are located as the end point of the ray; the image displayed on the holographic sandbox includes the image of the movable object and the image of the immovable object.

[0059] The images displayed in the holographic sandbox may include images of multiple objects. Some of the multiple objects may not be movable relative to the coordinate system used when the holographic sandbox displays the holographic image, and some of the objects may be movable relative to the coordinate system used when the holographic sandbox displays the holographic image. Exemplarily, the immovable objects may be buildings, trees, etc. The movable objects may be rackets, etc. The images displayed to the user by the holographic sandbox include multiple layers, each layer corresponding to an object.

[0060] The starting point of the ray is the interactive handle. When the image displayed in the holographic sandbox contains a layer corresponding to a movable object, the intersection of the ray and the layer corresponding to the movable object is the end point. When the image displayed in the holographic sandbox does not contain a layer corresponding to a movable object, the end point of the ray is the intersection of the ray and the layer at the bottom of all layers.

[0061] S210: Collecting interaction information generated by the interactive handle.

[0062] The interactive handle interacts with the image displayed on the holographic sandbox through interactive information. Any information that enables the user to interact with the image displayed on the holographic sandbox can be used as interactive information in this manual.

[0063] In an optional embodiment of the present specification, the interaction information includes: posture information of the interaction handle, information generated by a user's selection operation of the interaction handle, information generated by a user's deselection operation of the interaction handle, and the like.

[0064] The display processing device can execute this step in real time.

[0065] S212: Interact with the image of the movable object according to the interaction information.

[0066] It can be seen that when the holographic image includes both movable objects and immovable objects, the movable objects corresponding to the user's hand movements in the image displayed on the holographic sandbox can be identified. And after the movable objects corresponding to the user's hand movements are identified, the interaction between the user and the movable objects is realized. When the interaction between the user and the image displayed on the holographic sandbox is realized through the interactive handle, a ray for interaction can be generated according to the hand movements. The end point of the ray is the layer where the movable object is located. It can be seen that through the process in this specification, the ray can identify the movable objects that can be used for interaction, and the immovable objects in the image can be prevented from interfering with the interaction process.

[0067] In an optional embodiment of the present specification, the holographic display system is based on the unity3d engine. Tracker software is used to track the position information of the interactive handle in the real scene. The position information of the interactive handle in the real scene is transmitted to unity through the RTSForUnity plug-in. A handle object (GameObject) is created in the unity scene and named HandController. Then, the acquired position information of the interactive handle is assigned to HandController.

[0068] Then, on the HandController, generate rays along the Z-axis direction of the interactive handle (the extension direction of the interactive handle). Rays can be detected according to layers (a classification method for game objects in Unity). Create a new C# script, name it MyController, and mount it on the HandController. Add a layer in the Unity editor, name it InteractiveRay (the ray emitted by the handle object), and in the Update function of MyController (a function that comes with the Unity script and will be called in every frame of the program), use the Unity function to emit rays and set the parameters so that the rays only detect objects with the InteractiveRay layer set (that is, movable objects).

[0069] In an optional embodiment of the present specification, the process of interacting with the image of the movable object may be: highlighting the image of the movable object corresponding to the layer intersecting with the ray. Specifically, a C# script may be created and named IInterface. The interface IInterface is declared, and the function BeginOverlap() is declared in the IInterface interface for calling when the ray just enters the movable object, so that the movable object is highlighted to let the user know that the movable object is selected.

[0070] Afterwards, the posture information of the interactive handle can be used again to assign the control parameters of the handle object. Regenerate the ray in the extension direction of the handle object. If the layer where the regenerated ray and the image corresponding to the movable object are located no longer intersect, the image of the movable object is highlighted. Specifically, the function EndOverlap() can be declared in the IInterface interface for calling when the ray just leaves the movable object, which can cancel the highlight of the movable object to let the user know that the selection of the movable object has been canceled.

[0071] In a further optional embodiment of the present specification, the process of interacting with the image of a movable object may be: when the selection operation of the interactive handle is detected, the pose of the image of the movable object in the image displayed on the holographic sandbox is adjusted according to the pose information of the interactive handle. Specifically, the function OnClick(MyController myController) may be declared in the IInterface interface, which is called when the ray stops at the movable object and the button corresponding to the interactive handle is clicked, so as to simulate the function of clicking a button. The function OnGrab(MyControllermyController) may be declared in the IInterface interface, which is called when the ray stops at the movable object and the button corresponding to the interactive handle is pressed, so as to simulate the function of grabbing the movable object.

[0072] Thereafter, when the deselection operation of the interactive handle is detected, the position of the image of the movable object in the image displayed in the holographic sandbox is maintained until the selection operation of the interactive handle is detected again. Specifically, the function void OnRelease() can be declared in the IInterface interface, which is called when the button corresponding to the interactive handle is released when the ray stays on the movable object, so as to simulate the function of releasing the movable object.

[0073] You can create a C# script named InteractableObject, in which the specified variable of the object is the first specified value, to implement the basic functions of the movable object, as the base class of the movable object. In the Awake function of InteractableObject (run before the program runs), set the layer of the movable object to InteractiveRay, implement the interface IInterface, and implement the interface functions BeginOverlap, EndOverlap, OnGrab, and OnRelease.

[0074] To process the user's selection operation, you can create a C# script named Button to implement the basic functions of the button. As the base class of the button, it is used to represent the object implemented by the user's selection operation. In the Button function Awake (run before the program runs), set the layer of the movable object to InteractiveRay, implement the interface IInterface, and implement the interface function OnClick, so that the button can know that it has been clicked and make a corresponding response.

[0075] In a further optional embodiment of the present specification, the layer where the image corresponding to the movable object is located is provided with an interactive interface, and the layer where the image corresponding to the immovable object is located is not provided with an interactive interface. After determining the outgoing ray, the designated variable of the handle object is adjusted to a first designated value, and the first designated value indicates that the handle object has tracked the movable object. After interacting with the image of the movable object according to the interactive information, the method further includes: acquiring the posture information of the interactive handle again; reassigning the control parameters of the handle object using the posture information of the interactive handle acquired again; generating rays again in the extension direction of the handle object; if the rays generated again do not intersect with the layer where the image corresponding to the movable object is located, adjusting the designated variable of the handle object to a second designated value, and the second designated value indicates that the handle object has not tracked the movable object.

[0076] Specifically, the object detected by the ray of the previous frame can be stored as a specified variable named LastTraceObject. If the ray does not detect any movable object, LastTraceObject is a null value (the second specified value); the object detected by the ray of the current frame is stored as a variable named CurrentTraceObject. If the ray does not detect any movable object, CurrentTraceObject is a null value. Compare whether LastTraceObject and CurrentTraceObject are consistent. If they are inconsistent, it means that the ray has left LastTraceObject and entered CurrentTraceObject. At this time, if LastTraceObject is not null (the first specified value), it means that the ray has left LastTraceObject, and it is determined whether LastTraceObject has an IInterface interface, and the interface function BeginOverlap() is called; at this time, if CurrentTraceObject is not null, it means that the ray has left CurrentTraceObject, and it is determined whether CurrentTraceObject has an IInterface interface, and the interface function EndOverlap() is called.

[0077] When the ray intersects with the layer corresponding to a movable object and has not left, press the handle button to determine whether the CurrentTraceObject has an IInterface interface, call the interface function OnGrab(), release the handle button, call the interface function OnRelease(), and the grab and release operations can be implemented.

[0078] When the ray enters a button and has not left, click the handle button, determine whether the CurrentTraceObject has an IInterface interface, call the interface function OnClick(), and then the button click operation can be realized.

[0079] In an optional embodiment of the present specification, the display processing device can be integrated with the holographic sandbox. The customized dual-viewpoint radio frequency stereo glasses can be glasses with display function.

[0080] This specification does not impose any specific restrictions on the number of holographic sand tables included in the holographic display system. In the case of multiple holographic sand tables, the placement of different holographic sand tables in the venue can be the same or different.

[0081] The holographic display process in this specification can be executed by a display processing device in a holographic display system, and the holographic display process can include at least part of the following steps.

[0082] S500: The display processing device determines the viewpoint of each customized pair of radio frequency stereo glasses supporting dual viewpoints.

[0083] The process in this specification is intended to output holographic images to users at multiple different viewpoints. Since each user's viewpoint (i.e., the viewpoint of the customized dual-viewpoint RF stereo glasses worn by the user) is different to varying degrees, the display processing device in this specification determines the viewpoint for each customized dual-viewpoint RF stereo glasses.

[0084] S502: Generate a first display signal corresponding to the viewpoint according to the viewpoint of the customized dual-viewpoint radio frequency stereo glasses.

[0085] In order to enable users to experience the stereoscopic sense of the holographic image, the display processing device outputs a first display signal corresponding to the user's viewpoint for each user. The first display signal corresponding to the user's viewpoint is used to generate a holographic image viewed from the user's viewpoint.

[0086] In an optional embodiment of the present specification, the first display signal corresponding to a certain user is composed of a left-eye first display signal corresponding to the left eye of the user and a right-eye first display signal corresponding to the right eye of the user. There is a certain difference between the left-eye first display signal and the right-eye first display signal to simulate the difference in the viewpoints of the user's binoculars, thereby achieving a display with a three-dimensional sense.

[0087] In an optional embodiment of the present specification, the image specification corresponding to the first display signal is 1920 pixels*1080 pixels. The output frequency of the first display signal is 120 Hz.

[0088] S504: Outputting the first display signal corresponding to each viewpoint to the holographic sandbox, so that the holographic sandbox outputs an image according to the first display signal.

[0089] In an optional embodiment of the present invention, the holographic sandbox is a LED (Light Emitting Diode) display device.

[0090] After receiving the first display signal, the holographic sandbox outputs an image according to the first display signal for viewing by a user wearing customized dual-viewpoint radio frequency stereo glasses.

[0091] Since the number of customized dual-viewpoint radio frequency stereo glasses is not unique, and the first display signal received by the holographic sandbox is also not unique, the holographic sandbox needs to display holographic images to different customized dual-viewpoint radio frequency stereo glasses according to different first display signals.

[0092] S506: For each customized pair of radio frequency stereo glasses supporting dual viewpoints, generate a synchronization signal corresponding to the customized pair of radio frequency stereo glasses supporting dual viewpoints, and send the synchronization signal to the customized pair of radio frequency stereo glasses supporting dual viewpoints.

[0093] As can be seen from the foregoing, the holographic display in this specification requires the coordination of the holographic sandbox and the customized dual-viewpoint radio frequency stereo glasses. This coordination is achieved to a certain extent through a synchronization signal. In an optional embodiment of this specification, a synchronization signal can be used to control all customized dual-viewpoint radio frequency stereo glasses; in another optional embodiment of this specification, a synchronization signal can be generated for each customized dual-viewpoint radio frequency stereo glasses to control different customized dual-viewpoint radio frequency stereo glasses separately.

[0094] Specifically, for a certain customized pair of radio frequency stereo glasses supporting dual viewpoints, the synchronization signal controls the customized pair of radio frequency stereo glasses supporting dual viewpoints to collect the content displayed on the holographic sandbox when the holographic image under the viewpoint of the customized pair of radio frequency stereo glasses supporting dual viewpoints is displayed on the holographic sandbox, and displays the collected results to the user.

[0095] In an optional embodiment of the present specification, the customized dual-viewpoint radio frequency stereo glasses collect the images output by the holographic sandbox according to the synchronization signal, so as to alternately display the collected images to the user wearing the customized dual-viewpoint radio frequency stereo glasses.

[0096] In an optional embodiment of the present specification, the customized dual-viewpoint radio frequency stereo glasses are shutter-type 3D glasses.

[0097] It should be noted that the execution order of step S204 and step S206 is not particular. In an optional embodiment, step S204 and step S206 are executed simultaneously.

[0098] The interactive method used in the holographic sandbox in this specification uses a holographic sandbox and customized dual-viewpoint radio frequency stereo glasses to display holographic images to users. The holographic sandbox has strong controllability, and its display brightness can be adjusted according to the brightness of the scene in which it is located. Even in darker scenes, a good holographic image display effect can be obtained. Since the holographic image in this specification is assisted by the holographic sandbox, the customized dual-viewpoint radio frequency stereo glasses in this specification do not need to perform more complex data processing compared to existing VR glasses, which is conducive to reducing the volume and weight of the customized dual-viewpoint radio frequency stereo glasses, making the user feel more comfortable.

[0099] In addition, existing 3D display products usually have only a very narrow observation viewpoint or can only see the ideal effect at a fixed position, that is, existing 3D display products only support a single viewpoint, but the 3D stereo interactive display system with a single viewpoint display is increasingly unable to meet customer needs. The interactive method applied to the holographic sandbox in this specification uses a holographic sandbox and a customized dual-viewpoint radio frequency stereo glasses to match, even if the user's viewpoint changes (which may be caused by the user's movement), it can also display a holographic image with a three-dimensional sense to the user, which is conducive to improving the user experience.

[0100] The coordination process between the customized dual-viewpoint RF stereo glasses and the holographic sandbox is as follows: the display processing device controls the switching of the RF stereo glasses through the synchronous transmitter for two pairs of customized RF stereo glasses, so that the dual-viewpoint RF stereo glasses A and the dual-viewpoint RF stereo glasses B can display their respective tracked viewpoints in a time-sharing manner; the viewpoints of the dual-viewpoint RF stereo glasses A and the dual-viewpoint RF stereo glasses B corresponding to the viewpoints are output to the holographic sandbox in a time-sharing manner, so that the holographic sandbox can display the dual-viewpoint RF stereo glasses A and the dual-viewpoint RF stereo glasses B in a time-sharing manner, and the dual-viewpoint RF stereo glasses corresponding to the viewpoints are controlled by the synchronous transmitter.

[0101] In addition, the synchronization signal is output to the 3D synchronization transmitter at the same time, and the 3D synchronization transmitter transmits the radio frequency signal to control the opening and closing of the two pairs of dual-viewpoint radio frequency stereo glasses, so that the two pairs of dual-viewpoint radio frequency stereo glasses can independently observe their corresponding stereo videos, that is, dual-viewpoint radio frequency stereo glasses A can only see stereo video signal A, and dual-viewpoint radio frequency stereo glasses B can only see stereo video signal B. By using dual-viewpoint technology, two independent tracked viewpoints can be displayed in time-sharing on the same display screen in the same VR environment, so that the dual-viewpoint radio frequency stereo glasses A and the dual-viewpoint radio frequency stereo glasses B have corresponding viewpoints, thereby improving the immersive feeling of the system.

[0102] In addition, in other optional embodiments of the present specification, when the holographic sandbox displays a holographic image according to a first display signal, if an interaction instruction is received, the hand position of the user wearing the customized dual-viewpoint radio frequency stereo glasses is identified; the viewpoint information of the user is obtained, the viewpoint information showing the position of the user's binoculars and the viewing angle of the user's binoculars; according to the hand position, a first position of a handheld end for generating an interactive handle is determined; and according to the hand position and the viewpoint information, a second position of an indicating end for generating an interactive handle is determined; the handheld end is one end of the interactive handle, and the interactive end is the other end of the interactive handle; according to the first position and the second position, a second display signal is generated and sent to the holographic sandbox, so that the holographic sandbox displays a holographic image according to the first display signal and the second display signal.

[0103] In an optional embodiment of the present specification, the method further includes: determining whether the holographic sandbox is displaying a holographic image according to a first display signal; if so, detecting an interactive instruction; if not, executing the determination again.

[0104] In an optional embodiment of the present specification, a second position of an indicating end for generating an interactive handle is determined based on the hand position and the viewpoint information, including: extending a specified length from the hand position as a starting point so that the end point obtained by the extension intersects with the line of sight of the user, and determining that the end point is the second position; the line of sight of the user is obtained based on the viewpoint information.

[0105] In an optional embodiment of the present specification, the interaction achieved through the interactive handle includes at least one of the following: zooming in, zooming out, moving, rotating, selecting, and replacing the image output by the holographic sandbox.

[0106] In an optional embodiment of the present specification, the method further includes: determining whether the outline of the holographic image displayed on the holographic sandbox intersects with the indicating end of the interactive handle; if not, enlarging the holographic image displayed on the holographic sandbox until the outline of the holographic image intersects with the indicating end of the interactive handle.

[0107] In an optional embodiment of the present specification, the method further includes: generating a left-eye first display signal and a right-eye first display signal according to the viewpoint information of the user; processing the left-eye first display signal and the right-eye first display signal to generate a first display signal, and outputting the first display signal to the holographic sandbox, so that the holographic sandbox outputs an image according to the first display signal; generating a synchronization signal, and sending the synchronization signal to the customized dual-viewpoint supporting radio frequency stereo glasses, so that the customized dual-viewpoint supporting radio frequency stereo glasses collect the image output by the holographic sandbox according to the synchronization signal, so as to alternately display the collected images to the user wearing the customized dual-viewpoint supporting radio frequency stereo glasses.

[0108] In an optional embodiment of the present specification, the holographic display system further includes: a virtual camera; the position of the virtual camera is the same as the position of the user's binoculars, and the viewing angle of the virtual camera is the same as the viewing angle of the user's binoculars; the virtual camera is used to capture images of the environment in which the customized dual-viewpoint radio frequency stereo glasses are located according to its position and viewing angle. The method further includes: acquiring the image captured by the virtual camera; generating a first display signal according to the image captured by the virtual camera, so that at least part of the content in the environment is shown in the holographic image displayed by the holographic sandbox.

[0109] In an optional embodiment of the present specification, the holographic display system further includes: a motion capture camera; and the user's hand position is obtained based on data collected by the motion capture camera.

[0110] When tracking a user, for each dynamic auxiliary camera, an image captured by the dynamic auxiliary camera is obtained; a user included in the image captured by the dynamic auxiliary camera is identified; if the image captured by the dynamic auxiliary camera does not include a user, the number of users photographed by other dynamic auxiliary cameras other than the dynamic auxiliary camera is identified; if the number of users photographed by other dynamic auxiliary cameras is greater than a preset first threshold number, the other dynamic auxiliary camera is determined to be a reference camera; and the shooting angle of the dynamic auxiliary camera is adjusted so that the difference between the adjusted shooting angle of the dynamic auxiliary camera and the shooting angle of the reference camera is not greater than a preset threshold angle.

[0111] In an optional embodiment of the present specification, after adjusting the shooting angle of the dynamic auxiliary camera, it also includes: obtaining the shooting angle of each dynamic auxiliary camera, and judging whether there is a shooting blind spot within a specified range around the holographic sandbox; if the judgment result is yes, then among the dynamic auxiliary cameras, determining the dynamic auxiliary camera whose shooting angle is the smallest different from the shooting blind spot as the first target camera; adjusting the shooting angle of the first target camera so that the shooting blind spot is within the adjusted shooting angle of the first target camera.

[0112] In an optional embodiment of the present specification, the method further includes: when it is detected that the number of users in the images captured by each dynamic auxiliary camera is less than a second threshold number, adjusting the shooting angle of each dynamic auxiliary camera so that the shooting angle of each dynamic auxiliary camera is evenly distributed around the holographic sandbox; the second threshold number is less than the first threshold number.

[0113] In an optional embodiment of the present specification, the method further includes: for each dynamic auxiliary camera, determining the moving speed of the user in the image captured by the dynamic auxiliary camera within a specified time period from the current moment; if the moving speed of the user is greater than a threshold speed, determining that the dynamic auxiliary camera is the second target camera; among the other dynamic auxiliary cameras except the second target camera, determining the dynamic auxiliary camera with the smallest difference in shooting angle with the second target camera as the third target camera; adjusting the shooting angle of the third target camera so that the image captured by the third target camera includes the user whose moving speed is greater than the threshold speed. In an optional embodiment of the present specification, the dynamic auxiliary cameras are arranged around the holographic sandbox to capture images around the holographic sandbox; or, the dynamic auxiliary cameras are suspended above the holographic sandbox to capture images around the holographic sandbox.

[0114] Through the interactive method applied to the holographic sand table in this specification, a holographic image with a strong sense of three-dimensionality and high image quality can be displayed to the user in a variety of scenes with different brightness. In actual scenes, the user may move in the venue where the holographic sand table is set according to his own needs, and there may be a phenomenon that the holographic image displayed by the holographic sand table does not match the user's actual viewpoint due to the user's movement. In addition, when there are multiple users, the viewpoints of multiple users are also different to a certain extent.

[0115] In order to provide a holographic image with excellent display effect during the movement of the user, and to display the holographic images from different viewpoints at the same time, in an optional embodiment of the present specification, the holographic display system also includes: a motion capture camera. The motion capture camera is arranged on the holographic sandbox. The number of motion capture cameras in this specification, the setting positions on the holographic sandbox, and the shooting angles are determined according to the setting placement, shape, size, etc. of the holographic sandbox. Specifically, the number of motion capture cameras is 4, which are respectively arranged at the corners of the holographic sandbox. The motion capture camera is used to capture (shoot) images of the environment around the holographic sandbox to capture the user's position in the scene, and then obtain the customized viewpoint that supports dual-viewpoint radio frequency stereo glasses.

[0116] In other implementations of this specification, a motion capture camera can be placed above the holographic sandbox to capture images of the environment from a bird's-eye view.

[0117] The motion capture camera may perform image acquisition under the control of the display processing device.

[0118] In an optional embodiment of the present specification, the motion capture camera first obtains the 3D coordinate information of the customized dual-viewpoint radio frequency stereo glasses in space, and then combines the obtained human eye viewpoint coordinate information of the user wearing the customized dual-viewpoint radio frequency stereo glasses in the motion capture space to complete the tracking and positioning of the user's viewing angle, and then transmits the positioning information to the display processing device. The position signal of the display processing device in the motion capture space is synchronized to the holographic image, and the three-dimensional position of the customized dual-viewpoint radio frequency stereo glasses in the motion capture space is used as the position of the virtual camera in the holographic image (that is, the position of the customized dual-viewpoint radio frequency stereo glasses), so as to calculate the viewpoint picture of the virtual camera.

[0119] Furthermore, the customized dual-viewpoint radio frequency stereo glasses include shutter-type active 3D glasses and a rigid body composed of at least three reflective marking points. The reflective marking points may be 4. The at least three reflective marking points are embedded in the shell structure of the shutter-type active 3D glasses. The shutter-type active 3D glasses are used to receive the viewpoint images (3D signals) transmitted by the holographic sandbox, so as to obtain the correct left and right eye images. Specifically, the shutter-type active 3D glasses receive the 3D signal emitted by the 3D signal transmitter in the holographic sandbox.

[0120] The rigid body composed of at least three reflective marking points is used to mark the position of the shutter-type active 3D glasses in the motion capture space, and can capture the position information of the customized dual-viewpoint radio frequency stereo glasses in the motion capture space, as well as the viewpoint position information of the user wearing the customized dual-viewpoint radio frequency stereo glasses in the motion capture space. By capturing the position of the shutter-type active 3D glasses in the motion capture space, the user can see the three-dimensional picture of the display content floating in the air by wearing the shutter-type active 3D glasses, and can watch the three-dimensional picture (i.e., holographic image) from various angles.

[0121] It should be noted that, in an optional embodiment of the present specification, the display processing device may be composed of a plurality of apparatuses, and the functions of the display processing device may be implemented by the cooperation of the apparatuses constituting the display processing device.

[0122] Specifically, the display processing device may include: a display server, a display processor and a synchronization transmitter. The first display signal is generated by the display server; the composite signal and / or the synchronization signal is generated by the display processor; the synchronization signal is sent by the synchronization transmitter to the customized dual-viewpoint radio frequency stereo glasses. Optionally, the aforementioned left-eye display signal and right-eye display signal are generated by the display server, and the display server sends the left-eye display signal and the right-eye display signal to the display processor after generating them. The display processor generates a first display signal based on the left-eye display signal and the right-eye display signal, and sends it to the holographic sandbox. The synchronization signal is sent by the synchronization transmitter to the customized dual-viewpoint radio frequency stereo glasses.

[0123] Optionally, the display server is electrically connected to the display processor via a DP or HDMI cable; the display processor is electrically connected to the receiving card via a network port; the display processor is electrically connected to the synchronous transmitter via a cable; and the receiving card is electrically connected to the holographic sandbox via a flat cable.

[0124] In an optional embodiment of the present specification, the holographic sandbox can be composed of a small-pitch LED display screen, and the holographic sandbox includes an LED display screen body, a three-dimensional video amalgamator, a three-dimensional signal transmitter, and the like.

[0125] The 3D video fusion device is connected to the sending card of the LED display screen. The 3D signal transmitter is connected to the 3D video fusion device, which receives the viewpoint picture information of the virtual camera (i.e., customized dual-viewpoint RF stereo glasses) processed by the display processor in the display processing device, adjusts the display frequency and displays the picture on the LED display screen through the sending card to display the left and right eye pictures. The display processor is equipped with a holographic 3D display algorithm. The 3D signal transmitter synchronously sends the display frequency to the holographic 3D glasses to realize real-time display of the picture adjusted according to the different viewpoint positions of the user wearing the holographic 3D glasses.

[0126] In this description, in a virtual three-dimensional scene, the three-dimensional position of the customized dual-viewpoint radio frequency stereo glasses in the motion capture space is used as the position of the virtual camera in the virtual space. In this way, the position of a virtual camera can be simulated to capture the image at the location of the customized dual-viewpoint radio frequency stereo glasses. The virtual three-dimensional scene is set according to the position of the virtual camera in the virtual space.

[0127] The picture taken by the virtual camera (i.e., the picture under the viewpoint of the customized dual-viewpoint RF stereo glasses) is the three-dimensional virtual scene picture (i.e., holographic image) seen from the viewpoint of the user wearing the customized dual-viewpoint RF stereo glasses, that is, the virtual three-dimensional scene shot by the virtual camera is the 3D viewpoint picture viewed by the user. The display processing device can process the picture to be displayed in real time according to the viewpoint position information of the customized dual-viewpoint RF stereo glasses in the motion capture space, and display the viewpoint picture using a holographic sandbox, which can achieve an excellent holographic three-dimensional display effect and obtain an excellent screen-out feeling and holographic visual feeling.

[0128] Compared with other products, the holographic display system provided by this application has greatly reduced costs, and the picture display is more delicate and rich, and the colors are richer. The display effect is stable, the user is more convenient to use, and the operation and control are simpler. Among them, the holographic sand table has a delicate display effect, high brightness, and low cost. It uses LED active 3D stereoscopic display technology combined with the spatial position information of the observation viewpoint to achieve a 3D stereoscopic effect of virtual objects suspended on the LED display screen, which can improve the user's viewing experience and improve the effect of holographic display.

[0129] In addition, in an optional embodiment of the present specification, the process in the present specification can not only be used to display holographic images to users, but also to achieve interaction with users. Specifically, when receiving an interaction instruction, the display processing device generates a second display signal and outputs it to the holographic sandbox, so that the holographic sandbox outputs an image for displaying an interactive handle according to the second display signal, so that the user can operate the interactive handle to achieve interaction; the interaction includes at least one of the following: zooming in, reducing, moving, rotating, selecting, and replacing the image output by the holographic sandbox.

[0130] Optionally, the interactive instruction is triggered by a specified action of the user, and the specified action may be captured by a motion capture camera. The image of the interactive handle may be a stick-shaped image, and one end of the interactive handle cooperates with the user's hand movement and moves in the image generated by the holographic sandbox following the user's hand movement.

[0131] In an optional embodiment of this specification, the display processing device adopts the UE4 processing system. The holographic display system and the interactive method applied to the holographic sandbox in this specification are widely applicable to scenarios, specifically, they can be applied to military situation deduction, industrial equipment simulation operation, simulation design, education and teaching training room, medical simulation training and other scenarios.

[0132] Based on the same idea, the present specification also provides a corresponding embodiment Figure 2 An interactive device for a holographic sand table is shown in the partial process. Figure 3 As shown, the interactive device applied to the holographic sandbox may include one or more of the following modules:

[0133] The handle object creation module 300 is configured to: create a handle object, where the handle object is an object corresponding to an interactive handle in the image displayed on the holographic sandbox, and the interactive handle is used to capture the user's hand movements.

[0134] The posture information acquisition module 302 is configured to: acquire the posture information of the interactive handle.

[0135] The assignment module 304 is configured to: use the posture information of the interactive handle to assign the control parameters of the handle object, so that the posture of the image of the handle object displayed on the sandbox matches the posture of the interactive handle.

[0136] The ray generation module 306 is configured to generate rays in the extension direction of the handle object.

[0137] The end point determination module 308 is configured to: determine the intersection point of the layer where the ray and the image corresponding to the movable object are located as the end point of the ray; the image displayed on the holographic sandbox includes the image of the movable object and the image of the immovable object.

[0138] The interaction information module 310 is configured to collect the interaction information generated by the interaction handle.

[0139] The interaction module 312 is configured to interact with the image of the movable object according to the interaction information.

[0140] In an optional embodiment of the present specification, the interaction information includes: posture information of the interaction handle.

[0141] The interaction module 312 is specifically configured to highlight the image of the corresponding movable object presented by the layer intersecting with the ray.

[0142] In an optional embodiment of the present specification, the device further includes: a highlight display module.

[0143] The highlight display module is configured as follows: using the posture information of the interactive handle again to assign control parameters of the handle object; regenerating rays in the extension direction of the handle object; if the layer where the regenerated rays and the image corresponding to the movable object are located no longer intersect, then canceling the highlight display of the image of the movable object.

[0144] In an optional embodiment of the present specification, the interaction information includes information generated by the user for a selection operation of the interaction handle. The device further includes: a moving module. The moving module is configured to: when a selection operation of the interaction handle is detected, adjust the position of the image of the movable object in the image displayed on the holographic sandbox according to the position information of the interaction handle.

[0145] In an optional embodiment of the present specification, the device further includes: an adjustment module.

[0146] The adjustment module is configured to: when a deselection operation of the interactive handle is detected, maintain the posture of the image of the movable object in the image displayed on the holographic sandbox until a selection operation of the interactive handle is detected again.

[0147] In an optional embodiment of the present specification, the layer where the image corresponding to the movable object is located is provided with an interactive interface, and the layer where the image corresponding to the immovable object is located is not provided with an interactive interface.

[0148] In an optional embodiment of the present specification, the device further includes: a tracking module.

[0149] The tracking module is configured to: adjust the specified variable of the handle object to a first specified value, the first specified value indicating that the handle object has tracked the movable object; obtain the position information of the interactive handle again; reassign the control parameters of the handle object using the position information of the interactive handle obtained again; generate rays again in the extension direction of the handle object; if the rays generated again do not intersect with the layer where the image corresponding to the movable object is located, adjust the specified variable of the handle object to a second specified value, the second specified value indicating that the handle object has not tracked the movable object.

[0150] It can be understood that the above-mentioned interactive device applied to the holographic sandbox can implement the various steps of the holographic display process performed by the interactive device applied to the holographic sandbox provided in the aforementioned embodiments, and the relevant explanations on the interactive method applied to the holographic sandbox are applicable to the interactive device applied to the holographic sandbox, which will not be repeated here.

[0151] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 4 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0152] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0153] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0154] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an interactive device applied to the holographic sandbox and / or a second interactive device applied to the holographic sandbox at the logical level. The processor executes the program stored in the memory and is specifically used to execute any of the aforementioned holographic display processes.

[0155] The above application Figure 2 The interactive device for a holographic sandbox and / or the method executed by the second interactive device for a holographic sandbox disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0156] The electronic device may also perform Figure 2 At least some of the method steps performed by the interactive device applied to the holographic sandbox, and realizing an interactive device applied to the holographic sandbox and / or a second interactive device applied to the holographic sandbox in Figure 2 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0157] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 2 The illustrated embodiments include an interactive device applied to a holographic sandbox and / or a method executed by a second interactive device applied to a holographic sandbox, and are specifically used to execute any of the aforementioned interactive methods applied to a holographic sandbox.

[0158] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0159] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0162] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0163] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0164] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0165] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0166] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0167] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An interactive method applied to a holographic sandbox, wherein: The method comprises: Create a handle object, where the handle object is an object corresponding to an interactive handle in the image displayed by the holographic sandbox, and the interactive handle is used to capture the user's hand movements; Get the position information of the interactive handle; Using the posture information of the interactive handle to assign control parameters of the handle object, so that the posture of the image of the handle object displayed on the sand table matches the posture of the interactive handle; generating a ray in the extension direction of the handle object; Determine the intersection point of the layer where the ray and the image corresponding to the movable object are located as the end point of the ray; the image displayed on the holographic sand table includes the image of the movable object and the image of the immovable object; Collecting interaction information generated by the interactive handle; interacting with the image of the movable object according to the interaction information; The interaction information includes: posture information of the interaction handle, information generated by the user for the selection operation of the interaction handle; when the interaction information is received, the viewpoint information of the user is obtained, and the viewpoint information shows the position of the user's binoculars and the viewing angle of the user's binoculars; according to the hand position indicated by the action of the user's hand, a first position of the handheld end for generating the interaction handle is determined; and according to the hand position and the viewpoint information, a second position of the indicating end for generating the interaction handle is determined; the handheld end is one end of the interaction handle, and the interaction end is the other end of the interaction handle; according to the first position and the second position, a second display signal is generated; the second display signal is used to send to the holographic sandbox, so that the holographic sandbox displays a holographic image according to the first display signal and the second display signal; the first display signal is generated according to the viewpoint of the customized dual-viewpoint radio frequency stereo glasses, and the first display signal corresponding to the viewpoint of the user is used to generate a holographic image viewed from the viewpoint of the user.

2. The method of claim 1, wherein: Interacting with the image of the movable object according to the interaction information includes: The image of the corresponding movable object presented by the layer intersecting with the ray is highlighted.

3. The method of claim 2, wherein: After interacting with the image of the movable object according to the interaction information, the method further includes: The position information of the interactive handle is used again to assign the control parameters of the handle object; regenerate rays in the extension direction of the handle object; If the regenerated ray and the layer where the image corresponding to the movable object is located no longer intersect, the highlighting of the image of the movable object is canceled.

4. The method of claim 1, wherein: Interacting with the image of the movable object according to the interaction information includes: When a selection operation of the interactive handle is detected, the posture of the image of the movable object in the image displayed on the holographic sandbox is adjusted according to the posture information of the interactive handle.

5. The method of claim 4, wherein: After interacting with the image of the movable object according to the interaction information, the method further includes: When a deselection operation of the interactive handle is detected, the position of the image of the movable object in the image displayed on the holographic sandbox is maintained until a selection operation of the interactive handle is detected again.

6. The method of claim 1, wherein: The layer where the image corresponding to the movable object is located is provided with an interactive interface, while the layer where the image corresponding to the immovable object is located is not provided with an interactive interface.

7. The method of claim 1, wherein: After determining that the layer intersection point where the ray and the image corresponding to the movable object are located is the end point of the ray, the method further includes: adjusting the designated variable of the handle object to a first designated value, wherein the first designated value indicates that the handle object tracks the movable object; According to the interaction information, after interacting with the image of the movable object, the method also includes: acquiring the posture information of the interaction handle again; reassigning the control parameters of the handle object using the posture information of the interaction handle acquired again; generating rays again in the extension direction of the handle object; if the regenerated rays do not intersect with the layer where the image corresponding to the movable object is located, adjusting the designated variable of the handle object to a second designated value, the second designated value indicating that the handle object has not tracked the movable object.

8. An interactive device applied to a holographic sandbox, the device being used to implement any of the methods described in claims 1 to 7.

9. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute any one of the methods of claims 1 to 7.

10. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 7.

Images