An application method and device applied to a holographic sand table based on UE4

Abstract

The application discloses an interactive method and device based on a UE4 holographic sand table. When an image of a movable object is detected in a holographic picture of the holographic sand table, detection of a user gesture is performed. When the user gesture is detected, it is determined whether the moving direction of the user's hand shown by the user gesture matches a specified direction. If yes, the moving time of the user's hand continuously moving in the specified direction is detected. It is determined whether the moving time is greater than a threshold time. If yes, the image of the movable object is controlled to move in a moving mode corresponding to the specified direction. The process in the specification realizes interaction based on the moving direction of the user's hand and the time of the user's hand continuously moving, which can effectively avoid invalid actions of the user and interference on the process of interaction.

Description

Technical Field

[0001] This application relates to the field of simulation display technology, and in particular to an interactive method and device applied to a UE4 holographic sand table. Background Technology

[0002] Holographic display technology, also known as virtual imaging technology, is a technique that uses the principles of interference and diffraction to record and reproduce a true three-dimensional image of an object. It has the advantage of satisfying all human visual perception needs, and viewers can even view three-dimensional images without the aid of helmets, glasses, or other auxiliary devices. With the continuous development of display technology, holographic display technology has gained increasing attention.

[0003] Currently, common products on the market such as holographic projection and 3D display are usually only used to display specific images to users, and cannot interact with users. Summary of the Invention

[0004] This application provides an interactive method and apparatus for use with a UE4 holographic sandbox, which at least partially solves the above-mentioned technical problems.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide an interactive method applied to a UE4 holographic sandbox, the method comprising:

[0007] When the holographic image of the holographic sandbox is detected to contain the image of a movable object, the user's gesture is detected.

[0008] When a user gesture is detected, it is determined whether the direction of movement of the user's hand indicated by the gesture matches a specified direction;

[0009] If so, then detect the duration of continuous movement of the user's hand along the specified direction;

[0010] Determine whether the movement time is greater than a threshold time;

[0011] If so, the image of the movable object is controlled to move in a manner corresponding to the specified direction.

[0012] In an optional embodiment of this specification, after controlling the image of the movable object to move in a movement manner corresponding to the specified direction, the method further includes:

[0013] The specified time is the moment when the image of the movable object begins to move in a manner corresponding to the specified direction;

[0014] Within a specified time range starting from the specified time, the detection of user gestures is disabled.

[0015] In an optional embodiment of this specification, when a user gesture is detected, determining whether the direction of movement of the user's hand indicated by the user gesture matches a specified direction includes:

[0016] The needle determines the degree of match between the specified direction and the direction of movement of the user's hand in each preset specified direction;

[0017] Among the preset specified directions, the specified direction that matches the direction of movement of the user's hand indicated by the user's gesture is determined based on the specified direction with the highest matching degree.

[0018] In an optional embodiment of this specification, among preset specified directions, determining a specified direction that matches the movement direction of the user's hand indicated by the user's gesture, based on the specified direction with the highest matching degree, includes:

[0019] Among the preset specified directions, the specified direction with the highest matching degree is selected as the target direction;

[0020] Determine whether the target direction matching degree is greater than the threshold matching degree;

[0021] If so, the target direction is determined to be a specified direction that matches the direction of movement of the user's hand indicated by the user's gesture.

[0022] In an optional embodiment of this specification, controlling the image of the movable object to move in a manner corresponding to the specified direction includes:

[0023] Determine whether the first movement distance of the user's hand along the specified direction is greater than a threshold distance;

[0024] If so, the image of the movable object is controlled to move in a manner corresponding to the specified direction.

[0025] In an optional embodiment of this specification, wherein,

[0026] The threshold time is 0.5 seconds; and / or,

[0027] The threshold distance is 25 centimeters.

[0028] In an optional embodiment of this specification, determining whether the direction of movement of the user's hand indicated by the user's gesture matches a specified direction includes:

[0029] For each frame of action detected by the user's gesture, determine whether the second movement distance of the action in that frame relative to the previous frame along a specified direction is within a preset distance range;

[0030] If the judgment result for each frame of action is yes, then it is determined that the direction of movement of the user's hand indicated by the user gesture matches the specified direction.

[0031] Secondly, embodiments of this application also provide an interactive device applied to a UE4 holographic sandbox, for implementing any of the methods in the first aspect mentioned above.

[0032] Thirdly, embodiments of this application also provide an electronic device, including:

[0033] Processor; and

[0034] A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform any of the methods in the first aspect.

[0035] Fourthly, embodiments of this application also provide a computer-readable storage medium storing one or more programs that, when executed by an electronic device including multiple applications, cause the electronic device to perform any of the methods in the first aspect.

[0036] The at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects: The interaction method and apparatus applied to the UE4 holographic sandbox in this specification only detects user gestures when the holographic screen of the holographic sandbox contains an image of a movable object. This helps to achieve user interaction with the holographic screen while avoiding excessive resource consumption in detecting invalid user actions. In addition, the process in this specification is based on the direction of user hand movement and the duration of continuous hand movement to achieve interaction, which can effectively avoid invalid user actions that interfere with the interaction process. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0038] Figure 1 This is a schematic diagram of a holographic display scene provided in an embodiment of this specification;

[0039] Figure 2 This is a schematic diagram of a holographic display process provided in an embodiment of this specification;

[0040] Figure 3A schematic diagram of an interactive device structure applied to a UE4 holographic sand table, provided as an embodiment of this specification;

[0041] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Similar elements in different embodiments are referred to by related similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the present application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to the present application are not shown or described in the specification. This is to avoid obscuring the core parts of the present application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0043] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0044] The serial numbers assigned to components in this document, such as "second" or "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0045] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0046] For example, the interaction method applied to the UE4 holographic sandbox in this specification is based on a holographic display system. The holographic display system includes a holographic sandbox, customized dual-viewpoint RF stereoscopic glasses, and a display processing device. Users can view holographic images in the holographic sandbox by wearing the customized dual-viewpoint RF stereoscopic glasses. In an optional scenario, the interaction between the holographic display system and the user can be as follows: Figure 1 As shown.

[0047] In this specification, the holographic display system can capture the user's gestures and, based on the user's gestures, move the images of movable objects in the holographic image.

[0048] The interactive methods applied to the UE4 holographic sandbox in this specification may include the following steps:

[0049] S200: When the holographic image of the holographic sand table is detected to contain the image of a movable object, the user's gesture is detected.

[0050] This manual does not impose specific restrictions on how the holographic sand table can be set up in the holographic display venue. For example, the holographic sand table can be hung on the wall or set up on the ground.

[0051] The images displayed in a holographic sand table can include images of multiple objects. Some of these objects may be immovable relative to the coordinate system used for holographic image display, while others may be movable relative to the coordinate system used for holographic image display. For example, immovable objects may be buildings, trees, etc., while movable objects may be rackets, etc.

[0052] During the process described in this manual, users can control the movement of the corresponding image of the movable object in the holographic image through gestures.

[0053] The process described in this manual only detects user gestures when the holographic image of the holographic sand table contains the image of a movable object. This helps to achieve user interaction with the holographic image while avoiding the excessive resource consumption of detecting invalid user actions.

[0054] S202: When a user gesture is detected, determine whether the direction of movement of the user's hand indicated by the gesture matches a specified direction. If the determination result is yes, proceed to step S204. If the determination result is no, repeat step S200.

[0055] In this specification, user gestures can be represented by changes in the position of the user's hand relative to the coordinate system of the user's environment (which can be the coordinate system used when displaying holographic images in a holographic sand table, such as the world coordinate system).

[0056] In an optional embodiment of this specification, the holographic display system may include a motion-assisted camera, and the user's gestures may be captured by the motion-assisted camera.

[0057] S204: Detect the movement time of the user's hand moving continuously in the specified direction.

[0058] In this manual, the specified direction is the preset direction of the holographic display system. Users can learn the specified direction in advance.

[0059] Optionally, the timing begins when a specified number (preset value) of consecutive user movements along a specified direction are detected. If movement after the starting point continues along the specified direction, timing continues to accumulate movement time until the movement time exceeds a threshold time. If movement after the starting point no longer follows the specified direction, timing stops.

[0060] S206: Determine whether the movement time is greater than a threshold time. If the determination result is yes, proceed to step S208. If the determination result is no, repeat step S200.

[0061] The time threshold can be an empirical value; in one optional embodiment, the threshold time is 0.5 seconds. In this step, if the determination result is yes, it indicates that the user's gesture is a valid gesture and not a mistaken touch by the user.

[0062] S208: Control the image of the movable object to move in a movement manner corresponding to the specified direction.

[0063] In this specification, movement can be along a specified direction, such as moving up or down. It can also be flipping, rotating, zooming in or out, etc.

[0064] When the user's gestures are intentional, the image of a movable object in the holographic image is controlled to move along the specified direction to achieve interaction between the user and the holographic image. It is evident that the process described in this specification is based on the direction of the user's hand movement and the duration of continuous hand movement, effectively preventing invalid user actions from interfering with the interaction process.

[0065] Furthermore, to avoid binding user hand gestures to the movable object, in an optional embodiment of this specification, the moment when the image of the movable object begins to move in a manner corresponding to the specified direction is determined as a specified moment. Within a specified time range starting from the specified moment, the detection of user gestures is disabled.

[0066] In one optional embodiment of this specification, the specified time range may be an empirical value; alternatively, the specified time range may be 1 second.

[0067] In another optional embodiment of this specification, the time for moving the movable object can be defined as a specified time range. That is, the detection of user gestures is disabled before the movable object completes its movement.

[0068] In an optional embodiment of this specification, the specified direction may not be unique. For example, the specified direction may include up (UpAnim) and down (DownAnim). To identify which specified direction the user's gesture corresponds to among multiple specified directions, in an optional embodiment of this specification, for each preset specified direction, the matching degree between the specified direction and the user's hand movement direction is determined; among the preset specified directions, the specified direction with the highest matching degree is selected as the target direction. It is determined whether the matching degree of the target direction is greater than a threshold matching degree. If so, the target direction is determined to be a specified direction that matches the user's hand movement direction indicated by the user's gesture. If not, the user's gesture is determined to be an invalid gesture, and step S200 is re-executed.

[0069] Furthermore, to improve the accuracy of identifying invalid user operations, in an optional embodiment of this specification, it is determined whether a first movement distance of the user's hand along the specified direction is greater than a threshold distance. If so, the image of the movable object is controlled to move according to a movement mode corresponding to the specified direction. The threshold distance can be an empirical value, such as 25 centimeters.

[0070] In a further optional embodiment of this specification, the process of determining whether the movement direction of the user's hand indicated by the user gesture matches the specified direction may further include: for each frame of action detected by the user gesture, determining whether the second movement distance of the frame action relative to the previous frame action along the specified direction is within a preset distance range (which may be an empirical range, such as ±5 cm). If the determination result for each frame action is yes, then it is determined that the movement direction of the user's hand indicated by the user gesture matches the specified direction.

[0071] The procedures described in this manual can be based on LeapMotion. Before executing the procedures in this manual, you can create a new UE4 third-person C++ project, add the LeapMotion plugin to the project and enable the plugin in the project settings, and then restart the project. Run the LeapMotion driver, and then run the created project. If you can see the two hands that come with the plugin, it means that LeapMotion is working correctly.

[0072] Then, add two action recognition methods, UpAnim and DownAnim, to the blueprint Tick event of the plugin's hand: these two methods implement the raise and lower gesture actions, respectively.

[0073] In an optional embodiment of this specification, before step S200, the movement mode of the movable object corresponding to each specified direction can be predetermined and stored. For example, upward (UpAnim) and downward (DownAnim) movement modes are allowed for that specified direction. Then, after determining the specified direction that matches the user's gesture, the movement mode matching the user's gesture can be directly invoked.

[0074] Specifically, an event dispatcher can be created in advance, storing the correspondence between movement methods and specified directions. Then, based on the stored correspondence, the movement method corresponding to the user's gesture can be identified. Finally, the movable object can be moved according to the movement method. For example, the movable object can be directly controlled to move along a specified direction.

[0075] As can be seen, the process described in this specification has good extensibility and can be extended by the correspondence stored in the event dispatcher, that is, it can realize the movement of movable objects in different ways.

[0076] The execution entity of the interaction method applied to the UE4 holographic sandbox in this specification can be a display processing device. To achieve holographic display, this specification provides an interaction method applied to the UE4 holographic sandbox. This interaction method is based on a holographic display system, which includes: a display processing device, a holographic sandbox, and customized dual-viewpoint RF stereoscopic glasses. Multiple customized dual-viewpoint RF stereoscopic glasses are provided, and different customized dual-viewpoint RF stereoscopic glasses are worn by different users. The customized dual-viewpoint RF stereoscopic glasses move with the user's movement. Because different users have different poses relative to the holographic sandbox, the different customized dual-viewpoint RF stereoscopic glasses have different viewpoints.

[0077] In one optional embodiment of this specification, the display processing device can be integrated with the holographic sand table. Custom-designed dual-viewpoint radio frequency stereoscopic glasses can be glasses with display functionality.

[0078] This manual does not impose a specific limit on the number of holographic sand tables included in the holographic display system. When there are multiple holographic sand tables, their placement on the site can be the same or different.

[0079] The holographic display process described in this specification can be executed by a display processing device in a holographic display system, and the holographic display process may include at least some of the following steps.

[0080] S500: The display processing device determines the viewpoint of each customized dual-viewpoint RF stereoscopic glasses.

[0081] The process described in this specification is designed to output holographic images to a user from multiple different viewpoints. Since each user's viewpoint (i.e., the viewpoint of the custom-designed dual-viewpoint RF stereoscopic glasses worn by the user) differs to varying degrees, the display processing device in this specification determines the viewpoint for each custom-designed dual-viewpoint RF stereoscopic glasses.

[0082] S502: Generate a first display signal corresponding to the viewpoint of the customized dual-viewpoint RF stereoscopic glasses. (The first display signal corresponding to the viewpoint.)

[0083] To enable users to experience the stereoscopic effect of holographic images, the display processing device outputs a first display signal corresponding to the viewpoint of each user. This first display signal is used to generate the holographic image viewed from that user's viewpoint.

[0084] In an optional embodiment of this specification, the first display signal corresponding to a user is composed of a left-eye first display signal corresponding to the user's left eye and a right-eye first display signal corresponding to the user's right eye. There is a certain difference between the left-eye first display signal and the right-eye first display signal to simulate the difference in the user's binocular viewpoints, thereby achieving a stereoscopic display.

[0085] In an optional embodiment of this specification, the image size corresponding to the first display signal is 1920 pixels * 1080 pixels. The output frequency of the first display signal is 120 Hz.

[0086] S504: Output the first display signal corresponding to each viewpoint to the holographic sand table, so that the holographic sand table outputs images according to the first display signal.

[0087] In one optional embodiment described herein, the holographic sand table is an LED (Light Emitting Diode) display device.

[0088] After receiving the first display signal, the holographic sand table outputs an image based on the first display signal for users wearing customized dual-viewpoint radio frequency stereoscopic glasses to view.

[0089] Since the number of customized dual-viewpoint RF stereoscopic glasses is not unique, and the first display signal received by the holographic sand table is also not unique, the holographic sand table needs to display holographic images to different customized dual-viewpoint RF stereoscopic glasses according to different first display signals.

[0090] S506: For each customized dual-viewpoint RF stereoscopic glasses, generate a corresponding synchronization signal for that customized dual-viewpoint RF stereoscopic glasses and send it to that customized dual-viewpoint RF stereoscopic glasses.

[0091] As described above, the holographic display in this specification requires the use of a holographic sand table and customized dual-viewpoint RF stereoscopic glasses. This integration is achieved to some extent through synchronization signals. In one optional embodiment of this specification, a single synchronization signal can be used to control all customized dual-viewpoint RF stereoscopic glasses; in another optional embodiment, a separate synchronization signal can be generated for each customized dual-viewpoint RF stereoscopic glasses to control each pair individually.

[0092] Specifically, for a customized dual-viewpoint RF stereoscopic glasses, a synchronous signal controls the customized dual-viewpoint RF stereoscopic glasses to collect the content displayed in the holographic sand table when the customized dual-viewpoint RF stereoscopic glasses display holographic images from the viewpoint of the customized dual-viewpoint RF stereoscopic glasses, and displays the content displayed in the holographic sand table based on the collected results to the user.

[0093] In one optional embodiment of this specification, customized dual-viewpoint RF stereoscopic glasses acquire images output from the holographic sand table according to the synchronization signal, and alternately display the acquired images to a user wearing the customized dual-viewpoint RF stereoscopic glasses.

[0094] In one optional embodiment of this specification, the customized dual-viewpoint radio frequency stereoscopic glasses are shutter-type 3D glasses.

[0095] It should be noted that the execution order of steps S204 and S206 is not important. In an optional embodiment, steps S204 and S206 are executed simultaneously.

[0096] The interaction method described in this manual, applied to a UE4 holographic sandbox, uses a combination of the holographic sandbox and customized dual-viewpoint RF stereoscopic glasses to display holographic images to the user. The holographic sandbox offers high controllability; its brightness can be adjusted according to the ambient light of the scene, achieving good holographic image display even in darker environments. Because the holographic images in this manual are displayed with the assistance of the holographic sandbox, the customized dual-viewpoint RF stereoscopic glasses, compared to existing VR glasses, do not require complex data processing. This reduces the size and weight of the customized dual-viewpoint RF stereoscopic glasses, resulting in a more comfortable user experience.

[0097] Furthermore, existing 3D display products typically offer only a narrow viewing angle or allow for optimal viewing only from a fixed position. In other words, current 3D display products support only a single viewpoint, and single-viewpoint 3D interactive display systems are increasingly failing to meet customer needs. The interactive method described in this specification, applied to a UE4 holographic sand table, combines a holographic sand table with customized dual-viewpoint radio frequency stereoscopic glasses. Even if the user's viewpoint changes (which could be caused by user movement), it can still display a stereoscopic holographic image, thus improving the user experience.

[0098] The process of cooperating customized dual-viewpoint RF stereoscopic glasses with a holographic sand table is as follows: The display processing device controls the on / off state of the RF stereoscopic glasses for two pairs of customized RF stereoscopic glasses through a synchronous transmitter, so that dual-viewpoint RF stereoscopic glasses A and dual-viewpoint RF stereoscopic glasses B display their respective tracked viewpoints in a time-division manner; the viewpoints corresponding to the viewpoints of dual-viewpoint RF stereoscopic glasses A and dual-viewpoint RF stereoscopic glasses B are output to the holographic sand table in a time-division manner, so that the holographic sand table displays the viewpoints of dual-viewpoint RF stereoscopic glasses A and dual-viewpoint RF stereoscopic glasses B in a time-division manner, and controls the corresponding viewpoints of the dual-viewpoint RF stereoscopic glasses through the synchronous transmitter.

[0099] In addition, a synchronization signal is simultaneously output to a 3D synchronization transmitter. The 3D synchronization transmitter transmits radio frequency signals to control the switching on and off of two pairs of dual-viewpoint RF stereoscopic glasses. This allows each pair of dual-viewpoint RF stereoscopic glasses to independently observe its corresponding stereoscopic video; that is, dual-viewpoint RF stereoscopic glasses A can only see stereoscopic video signal A, and dual-viewpoint RF stereoscopic glasses B can only see stereoscopic video signal B. By using dual-viewpoint technology, two independent tracked viewpoints can be displayed in a time-sharing manner on the same display screen within the same VR environment. This gives both dual-viewpoint RF stereoscopic glasses A and B their corresponding viewpoints, thereby enhancing the immersive experience of the system.

[0100] Furthermore, in other optional embodiments of this specification, when the holographic sand table displays holographic images according to the first display signal, if an interactive command is received, the hand position of the user wearing the customized dual-viewpoint radio frequency stereoscopic glasses is identified; the user's viewpoint information is obtained, which shows the position of the user's binoculars and the user's binocular viewing angle; based on the hand position, a first position for generating the handheld end of the interactive handle is determined; and based on the hand position and the viewpoint information, a second position for generating the pointer end of the interactive handle is determined; the handheld end is one end of the interactive handle, and the interactive end is the other end of the interactive handle; based on the first position and the second position, a second display signal is generated and sent to the holographic sand table, so that the holographic sand table displays holographic images according to the first display signal and the second display signal.

[0101] In an optional embodiment of this specification, the method further includes, before proceeding, determining whether the holographic sand table is displaying a holographic image according to a first display signal; if yes, then detecting the interactive command; if no, then performing the determination again.

[0102] In an optional embodiment of this specification, determining a second position for generating the indicator end of the interactive handle based on the hand position and the viewpoint information includes: extending a specified length from the hand position such that the endpoint of the extension intersects with the user's line of sight, and determining the endpoint as the second position; the user's line of sight is obtained based on the viewpoint information.

[0103] In one optional embodiment of this specification, the interaction implemented through the interactive handle includes at least one of the following: zooming in, zooming out, moving, rotating, selecting, and changing the images output by the holographic sand table.

[0104] In an optional embodiment of this specification, the method further includes: determining whether the outline of the holographic image displayed on the holographic sand table intersects with the indicator end of the interactive handle; if not, enlarging the holographic image displayed on the holographic sand table until the outline of the holographic image intersects with the indicator end of the interactive handle.

[0105] In an optional embodiment of this specification, the method further includes: generating a left-eye first display signal and a right-eye first display signal based on the user's viewpoint information; processing the left-eye first display signal and the right-eye first display signal to generate a first display signal, which is then output to the holographic sand table, causing the holographic sand table to output an image based on the first display signal; generating a synchronization signal and sending it to the customized dual-viewpoint RF stereoscopic glasses, causing the customized dual-viewpoint RF stereoscopic glasses to acquire the image output by the holographic sand table based on the synchronization signal, and alternately displaying the acquired image to the user wearing the customized dual-viewpoint RF stereoscopic glasses.

[0106] In an optional embodiment of this specification, the holographic display system further includes: a virtual camera; the virtual camera is positioned at the same location as the user's binoculars, and the virtual camera's viewing angle is the same as the user's binoculars' viewing angle; the virtual camera is used to acquire images of the environment in which the customized dual-viewpoint RF stereoscopic glasses are located, based on its position and viewing angle. The method further includes: acquiring the images acquired by the virtual camera; and generating a first display signal based on the images acquired by the virtual camera, such that at least a portion of the content in the environment is shown in the holographic image displayed on the holographic sand table.

[0107] In an optional embodiment of this specification, the holographic display system further includes a motion capture camera; the user's hand position is obtained based on data collected by the motion capture camera.

[0108] When tracking users, for each auxiliary camera, the image captured by that auxiliary camera is acquired; the user contained in the image captured by that auxiliary camera is identified; if the image captured by that auxiliary camera does not contain a user, the number of users captured by other auxiliary cameras besides that auxiliary camera is identified; if the number of users captured by other auxiliary cameras is greater than a preset first threshold number, then that other auxiliary camera is determined to be a reference camera; the shooting angle of the auxiliary camera is adjusted so that the difference between the adjusted shooting angle of the auxiliary camera and the shooting angle of the reference camera is not greater than a preset threshold angle.

[0109] In an optional embodiment of this specification, after adjusting the shooting angle of the auxiliary camera, the method further includes: acquiring the shooting angle of each auxiliary camera, determining whether there is a shooting blind spot within a specified range around the holographic sand table; if the determination result is yes, then among the auxiliary cameras, determining the auxiliary camera with the smallest difference between its shooting angle and the shooting blind spot as the first target camera; adjusting the shooting angle of the first target camera so that the shooting blind spot is within the adjusted shooting angle of the first target camera.

[0110] In an optional embodiment of this specification, the method further includes: when the number of users in the images captured by each auxiliary camera is less than a second threshold number, adjusting the shooting angle of each auxiliary camera so that the shooting angles of each auxiliary camera are evenly distributed around the holographic sand table; the second threshold number is less than the first threshold number.

[0111] In an optional embodiment of this specification, the method further includes: for each auxiliary camera, determining the user's movement speed in the image captured by the auxiliary camera within a specified time period from the current time; if the user's movement speed is greater than a threshold speed, then determining that the auxiliary camera is a second target camera; among all the other auxiliary cameras besides the second target camera, determining the auxiliary camera with the smallest shooting angle difference from the second target camera as a third target camera; adjusting the shooting angle of the third target camera so that the image captured by the third target camera includes the user whose movement speed is greater than the threshold speed. In an optional embodiment of this specification, each auxiliary camera is positioned around the holographic sand table to acquire images of the area around the holographic sand table; or, the auxiliary camera is suspended above the holographic sand table to acquire images of the area around the holographic sand table.

[0112] The interactive method described in this manual, applied to a UE4 holographic sandbox, can display highly immersive and high-quality holographic images to users in various scenarios with varying brightness. However, in real-world scenarios, users may move around the holographic sandbox area according to their needs. This could lead to a mismatch between the holographic image displayed on the sandbox and the user's actual viewpoint. Furthermore, when there are multiple users, their viewpoints will also differ to some extent.

[0113] To provide high-quality holographic images during user movement and to simultaneously display holographic images from different viewpoints, in an optional embodiment of this specification, the holographic display system further includes motion capture cameras. The motion capture cameras are mounted on the holographic sand table. The number of motion capture cameras, their positions on the holographic sand table, and their shooting angles are determined based on the placement, shape, and size of the holographic sand table. Specifically, four motion capture cameras are used, positioned at the corners of the holographic sand table. The motion capture cameras are used to acquire (capture) images of the environment surrounding the holographic sand table to capture the user's position within the scene, thereby obtaining a customized viewpoint supporting dual-viewpoint radio frequency stereoscopic glasses.

[0114] In other embodiments of this specification, the motion capture camera can be positioned above the holographic sand table to capture images of the environment from a bird's-eye view.

[0115] Motion capture cameras can perform image acquisition under the control of display processing devices.

[0116] In an optional embodiment of this specification, the motion capture camera first acquires the 3D coordinate information of the customized dual-viewpoint RF stereoscopic glasses in space. Then, it combines this information with the acquired coordinate information of the user's eye viewpoint in the motion capture space while wearing the customized dual-viewpoint RF stereoscopic glasses to track and locate the user's viewing angle. This positioning information is then transmitted to the display processing device. This synchronizes the position signal of the display processing device in the motion capture space to the holographic image, and uses the three-dimensional position of the customized dual-viewpoint RF stereoscopic glasses in the motion capture space as the position of the virtual camera in the holographic image (i.e., the position of the customized dual-viewpoint RF stereoscopic glasses) to calculate the viewpoint of the virtual camera.

[0117] Furthermore, the customized dual-viewpoint RF stereoscopic glasses include shutter-type active 3D glasses and a rigid body composed of at least three reflective markers. There can be four reflective markers. The at least three reflective markers are embedded in the outer shell structure of the shutter-type active 3D glasses. The shutter-type active 3D glasses are used to receive the viewpoint image (3D signal) transmitted from the holographic sand table, thereby obtaining the correct left and right eye images. Specifically, the shutter-type active 3D glasses receive the 3D signal emitted by the 3D signal transmitter in the holographic sand table.

[0118] The rigid body, composed of at least three reflective markers, is used to mark the position of the shutter-type active 3D glasses in the motion capture space. It can capture the position information of the customized dual-viewpoint RF stereoscopic glasses in the motion capture space, as well as the viewpoint position information of the user wearing the customized dual-viewpoint RF stereoscopic glasses in the motion capture space. By capturing the position of the shutter-type active 3D glasses in this way, the user can see a stereoscopic image of the displayed content floating in mid-air, and can view the 3D image (i.e., holographic image) from various angles.

[0119] It should be noted that, in an optional embodiment of this specification, the display processing device may be composed of multiple devices, and the function of the display processing device may be realized by the cooperation of the various devices constituting the display processing device.

[0120] Specifically, the display processing device may include a display server, a display processor, and a synchronization transmitter. The first display signal is generated by the display server; the composite signal and / or synchronization signal are generated by the display processor; the synchronization signal is sent by the synchronization transmitter to customized dual-viewpoint radio frequency stereoscopic glasses. Optionally, the aforementioned left-eye and right-eye display signals are generated by the display server, which sends them to the display processor after generating them. The display processor generates the first display signal based on the left-eye and right-eye display signals and sends it to the holographic sand table. The synchronization signal is sent by the synchronization transmitter to customized dual-viewpoint radio frequency stereoscopic glasses.

[0121] Optionally, the display server is electrically connected to the display processor via a DP or HDMI cable; the display processor is electrically connected to the receiver card via an Ethernet port; the display processor is electrically connected to the synchronization transmitter via a cable. The receiver card is electrically connected to the holographic sand table via a ribbon cable.

[0122] In one optional embodiment of this specification, the holographic sand table may be composed of a small-pitch LED display screen, which includes an LED display screen body, a three-dimensional video fusion unit, a three-dimensional signal transmitter, etc.

[0123] The 3D video fusion unit is connected to a transmitting card on the LED display screen. A 3D signal transmitter is connected to the 3D video fusion unit, which receives viewpoint information from a virtual camera (i.e., customized dual-viewpoint RF stereoscopic glasses) processed by the display processor in the display processing device. It adjusts the display frequency and displays the image on the LED display screen via the transmitting card to show the left and right eye views. The display processor incorporates a holographic 3D display algorithm. The 3D signal transmitter synchronously transmits this display frequency to the holographic 3D glasses to achieve real-time display of the image adjusted according to the user's viewpoint position while wearing the holographic 3D glasses.

[0124] In this description, in the virtual 3D scene, the 3D position of the customized dual-viewpoint RF stereoscopic glasses in the motion capture space is used as the position of the virtual camera in the virtual space. This simulates the position of a virtual camera to capture the image of the location of the customized dual-viewpoint RF stereoscopic glasses. The virtual 3D scene is set according to the position of the virtual camera in the virtual space.

[0125] The image captured by the virtual camera (i.e., the image seen from the perspective of the customized dual-viewpoint RF stereoscopic glasses) is the 3D virtual scene (i.e., holographic image) seen by the user wearing the customized dual-viewpoint RF stereoscopic glasses. In other words, the virtual 3D scene captured by the virtual camera is the 3D viewpoint image seen by the user. The display processing device can process the image to be displayed in real time based on the viewpoint position information of the customized dual-viewpoint RF stereoscopic glasses in the motion capture space, and simultaneously display the viewpoint image using a holographic sand table, achieving excellent holographic 3D display effects and obtaining a superb sense of out-of-screen movement and holographic visual experience.

[0126] The holographic display system provided in this application significantly reduces costs compared to other products, offering more delicate and richer image display with more vibrant colors. It boasts strong display stability, greater user convenience, and simpler operation and control. Specifically, the holographic sand table exhibits delicate display effects, high brightness, and low cost. Utilizing LED active 3D stereoscopic display technology combined with the spatial position information of the viewing point, it achieves a 3D stereoscopic effect of virtual objects floating on the LED display screen, enhancing the user experience and improving the overall holographic display effect.

[0127] In addition, in an optional embodiment of this specification, the process described herein can not only be used to display holographic images to users, but also to enable interaction with users. Specifically, when the display processing device receives an interaction command, it generates a second display signal and outputs it to the holographic sand table, causing the holographic sand table to output an image for displaying an interactive handle based on the second display signal, enabling the user to operate the interactive handle to achieve interaction; the interaction includes at least one of the following: magnifying, shrinking, moving, rotating, selecting, or changing the image output by the holographic sand table.

[0128] Optionally, the interaction command is triggered by a user-specified action, which can be captured by a motion capture camera. The image of the interactive handle can be a rod-shaped image, with one end of the handle coordinating with the user's hand movements and moving within the image generated by the holographic sand table, following the user's hand movements.

[0129] In one optional embodiment of this specification, the display processing device employs the UE4 processing system. The holographic display system and the interactive method applied to the UE4 holographic sand table described in this specification have wide applicability, specifically, they can be applied to scenarios such as military situation simulation, industrial equipment simulation operation, simulation design, educational training rooms, and medical simulation training.

[0130] Based on the same idea, the embodiments in this specification also provide corresponding... Figure 2 The illustrated part of the process is applied to an interactive device based on a UE4 holographic sandbox. For example... Figure 3As shown, the interactive device applied to the UE4 holographic sandbox may include one or more of the following modules:

[0131] The user gesture detection module 300 is configured to perform user gesture detection when the holographic image of the holographic sand table contains the image of a movable object.

[0132] The first judgment module 302 is configured to: when a user gesture is detected, determine whether the direction of movement of the user's hand shown by the user gesture matches a specified direction.

[0133] The movement time determination module 304 is configured to: if so, detect the movement time of the user's hand moving continuously along the specified direction.

[0134] The second judgment module 306 is configured to: determine whether the moving time is greater than a threshold time.

[0135] The movement mode determination module 308 is configured to: if so, control the image of the movable object to move according to the movement mode corresponding to the specified direction.

[0136] In an optional embodiment of this specification, the interactive device applied to a UE4 holographic sandbox further includes a control disable module. This module is configured to: determine the moment when the image of the movable object begins to move in a manner corresponding to the specified direction as a specified moment; and determine, within a specified time range starting from the specified moment, disable the detection of user gestures.

[0137] In an optional embodiment of this specification, the first judgment module 302 is specifically configured to: determine the matching degree between the specified direction and the movement direction of the user's hand in each preset specified direction; and among the preset specified directions, determine the specified direction that matches the movement direction of the user's hand indicated by the user's gesture based on the specified direction with the largest matching degree.

[0138] In an optional embodiment of this specification, the first judgment module 302 is specifically configured to: filter out the specified direction with the highest matching degree from the preset specified directions as the target direction; determine whether the matching degree of the target direction is greater than the threshold matching degree; if so, determine that the target direction is a specified direction that matches the movement direction of the user's hand indicated by the user's gesture.

[0139] In an optional embodiment of this specification, the movement mode determination module 308 is specifically configured to: determine whether the first movement distance of the user's hand along the specified direction is greater than a threshold distance; if so, control the image of the movable object to move according to the movement mode corresponding to the specified direction.

[0140] In one optional embodiment of this specification, the threshold time is 0.5 seconds; and / or, the threshold distance is 25 centimeters.

[0141] In an optional embodiment of this specification, the first judgment module 302 is specifically configured to: for each frame of action detected by the user gesture, determine whether the second moving distance of the frame action relative to the previous frame action along a specified direction is within a preset distance range; if the judgment result for each frame action is yes, then determine that the moving direction of the user's hand shown by the user gesture matches the specified direction.

[0142] It is understood that the above-mentioned interactive device applied to the UE4 holographic sandbox can realize the various steps of the holographic display process performed by the interactive device applied to the UE4 holographic sandbox provided in the foregoing embodiments. The relevant explanations of the interactive method applied to the UE4 holographic sandbox are applicable to the interactive device applied to the UE4 holographic sandbox, and will not be repeated here.

[0143] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 4 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0144] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0145] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0146] The processor reads the corresponding computer program from non-volatile memory into memory and then runs it, logically forming one type of interactive device applied to the UE4 holographic sandbox and / or a second type of interactive device applied to the UE4 holographic sandbox. The processor executes the program stored in memory and is specifically used to execute any of the aforementioned holographic display processes.

[0147] The above is as stated in this application. Figure 2 The embodiments shown disclose a method for use in an interactive device based on a UE4 holographic sandbox and / or a second method for use in an interactive device based on a UE4 holographic sandbox. This method can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0148] The electronic device can also perform Figure 2 The method steps are applied to at least some of the interactive steps performed on an interactive device based on a UE4 holographic sandbox, and implements an interactive device based on a UE4 holographic sandbox and / or a second interactive device based on a UE4 holographic sandbox in... Figure 2 The functions of the embodiments shown are not described in detail here.

[0149] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 2 The embodiments shown include a method for executing an interactive device based on a UE4 holographic sandbox and / or a second method for executing an interactive device based on a UE4 holographic sandbox, specifically for executing any of the aforementioned interactive methods for an interactive device based on a UE4 holographic sandbox.

[0150] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0154] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0155] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0156] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0157] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An interactive method applied to a UE4 holographic sandbox, wherein, The method includes: The movement method of a movable object corresponding to each specified direction is determined in advance and stored; When the holographic image of the holographic sandbox is detected to contain the image of a movable object, the user's gesture is detected. When a user gesture is detected, it is determined whether the direction of movement of the user's hand indicated by the gesture matches a specified direction; If so, then detect the duration of continuous movement of the user's hand along the specified direction; Determine whether the movement time is greater than a threshold time; If so, the image of the movable object is controlled to move in a manner corresponding to the specified direction; If the movement method of the movable object corresponding to each specified direction is determined in advance, then after the specified direction matching the user's gesture is determined, the movement method of the specified direction matching the user's gesture can be directly called. Specifically, a pre-built event dispatcher stores the correspondence between movement methods and specified directions in the event dispatcher. Based on the correspondence stored in the event dispatcher, the movement method corresponding to the user's gesture is identified, and then the movable object is moved according to the movement method. Among them, when a user gesture is detected, determining whether the direction of movement of the user's hand indicated by the user gesture matches a specified direction includes: For each preset specified direction, determine the degree of matching between that specified direction and the direction of movement of the user's hand; Among the preset specified directions, the specified direction that matches the direction of movement of the user's hand indicated by the user's gesture is determined according to the specified direction with the highest matching degree; Among the preset specified directions, the specified direction that matches the direction of movement of the user's hand indicated by the user's gesture is determined based on the specified direction with the highest matching degree, including: Among the preset specified directions, the specified direction with the highest matching degree is selected as the target direction; Determine whether the target direction matching degree is greater than the threshold matching degree; If so, the target direction is determined to be a specified direction that matches the direction of movement of the user's hand indicated by the user's gesture.

2. The method as described in claim 1, wherein, After controlling the image of the movable object to move in a manner corresponding to the specified direction, the method further includes: The specified time is the moment when the image of the movable object begins to move in a manner corresponding to the specified direction; Within a specified time range starting from the specified time, the detection of user gestures is disabled.

3. The method as described in claim 1, wherein, Controlling the image of the movable object to move in a manner corresponding to the specified direction includes: Determine whether the first movement distance of the user's hand along the specified direction is greater than a threshold distance; If so, the image of the movable object is controlled to move in a manner corresponding to the specified direction.

4. The method as described in claim 3, wherein, The threshold time is 0.5 seconds; and / or, The threshold distance is 25 centimeters.

5. The method as described in claim 1, wherein, Determining whether the direction of movement of the user's hand indicated by the user's gesture matches a specified direction includes: For each frame of action detected by the user's gesture, determine whether the second movement distance of the action in the frame relative to the previous frame along the specified direction is within the preset distance range; If the judgment result for each frame of action is yes, then it is determined that the direction of movement of the user's hand indicated by the user gesture matches the specified direction.

6. An interactive device for use with a UE4 holographic sandbox, the device being used to implement the method of any one of claims 1 to 5.

7. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the method described in any one of claims 1 to 5.

8. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 5.

Images