Method and system for human-computer interaction

Abstract

The invention provides a method and a system for human-computer interaction. The method comprises the following steps: acquiring a first coordinate of a first real object in a depth camera coordinate system and a first eye coordinate of eyes of a first user in the depth camera coordinate system through a depth camera group; according to the first eye coordinate, based on the first coordinate, calculating a second coordinate of the first real object in a first eye coordinate system of the eyes of the first user; the first human eye coordinate system coincides with a first virtual camera coordinate system of a first virtual camera in a virtual world, and a third coordinate of a first virtual object corresponding to the first real object in the first virtual camera coordinate system is calculated; and calculating a fourth coordinate of the first virtual object in the virtual world coordinate system based on the third coordinate according to a first virtual camera coordinate of the first virtual camera in the virtual world coordinate system of the virtual world.

Description

Technical Field

[0001] This disclosure relates to the field of human-computer interaction technology, and more specifically, to methods and systems for human-computer interaction. Background Technology

[0002] In the field of human-computer interaction technology, achieving virtual-real fusion interaction is a key goal, but traditional technologies have many limitations.

[0003] In traditional extended reality (XR) or holographic scenarios, the amount of computation increases exponentially as the number of objects in the scene increases, consuming a large amount of computing resources, resulting in delayed interactive responses, affecting smoothness, and reducing user experience. Summary of the Invention

[0004] Embodiments of this disclosure provide methods and systems for human-computer interaction that can significantly reduce computational load. Furthermore, embodiments of this disclosure provide methods and systems for human-computer interaction that can improve the accuracy and realism of human-computer interaction while significantly reducing computational load.

[0005] According to embodiments of this disclosure, a method for human-computer interaction is provided, the method comprising: obtaining, through a depth camera group, a first coordinate of a first real object in a depth camera coordinate system and a first eye coordinate of a first user in the depth camera coordinate system; calculating, based on the first eye coordinate, a second coordinate of the first real object in a first human eye coordinate system of the first user's eye; calculating, by aligning the first human eye coordinate system with the first virtual camera coordinate system of a first virtual camera in a virtual world, a third coordinate of a first virtual object corresponding to the first real object in the first virtual camera coordinate system, wherein the first human eye coordinate system coincides with the first virtual camera coordinate system by aligning a reference point of the first human eye coordinate system with a reference point of the first virtual camera coordinate system; and calculating, based on the third coordinate, a fourth coordinate of the first virtual object in the virtual world coordinate system.

[0006] According to embodiments of this disclosure, the first coordinate may include the coordinates of the feature point cloud of the first real object in the depth camera coordinate system.

[0007] According to embodiments of this disclosure, the reference point of the first human eye coordinate system may be the midpoint between the eyes of the first user, and the reference point of the first virtual camera coordinate system may be the position of the first virtual camera in the virtual world.

[0008] According to embodiments of this disclosure, when a single user performs human-computer interaction, the first virtual camera coordinate system of the first virtual camera can coincide with the virtual world coordinate system of the first virtual camera in the virtual world.

[0009] According to embodiments of this disclosure, the method may further include: performing an interaction between a first real object and a virtual target based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system.

[0010] According to embodiments of this disclosure, the steps of performing an interaction between a first real object and a virtual target may include: generating a real collision box and a virtual collision box based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system, respectively; recognizing a collision in response to the overlap between the coordinate range of the real collision box and the coordinate range of the virtual collision box in the virtual world coordinate system; determining a control command corresponding to the first real object based on the recognition of the collision; and controlling the virtual object to perform an operation corresponding to the control command.

[0011] According to embodiments of this disclosure, the control commands may include at least one of click commands, move commands, zoom commands, and rotate commands.

[0012] According to embodiments of this disclosure, the step of performing interaction between a first real object and a virtual target may further include: applying a collider component to the first virtual object and the virtual target based on the recognition of a collision occurrence, and applying a rigid body component to the virtual target; and simulating the physical effects of human-computer interaction based on the properties of the collider component and the rigid body component.

[0013] According to embodiments of this disclosure, the method may further include: obtaining, through a depth camera group, the sixth coordinate of the second real object in the depth camera coordinate system and the second eye coordinate of the second user's eye in the depth camera coordinate system; calculating, based on the sixth coordinate and the second eye coordinate, the seventh coordinate of the second real object in the second human eye coordinate system of the second user's eye; calculating, by aligning the second human eye coordinate system with the second virtual camera coordinate system of the second virtual camera in the virtual world, the eighth coordinate of the second virtual object corresponding to the second real object in the second virtual camera coordinate system, wherein the second human eye coordinate system coincides with the second virtual camera coordinate system by aligning the reference point of the second human eye coordinate system with the reference point of the second virtual camera coordinate system; and calculating, based on the eighth coordinate and the second virtual camera coordinate of the second virtual camera in the virtual world coordinate system, the ninth coordinate of the second virtual object in the virtual world coordinate system.

[0014] According to embodiments of this disclosure, the method may further include: performing an interaction between a second real object, a first real object, and a virtual target based on the ninth coordinate of the second virtual object in the virtual world coordinate system, the fourth coordinate of the first virtual object in the virtual world coordinate system, and the fifth coordinate of the virtual target in the virtual world coordinate system.

[0015] According to embodiments of this disclosure, a system for human-computer interaction is provided, the system comprising: a depth camera group; a coordinate acquisition module configured to: obtain, through the depth camera group, a first coordinate of a first real object in a depth camera coordinate system and a first eye coordinate of a first user in the depth camera coordinate system; and a coordinate calculation module configured to: calculate, based on the first eye coordinate, a second coordinate of the first real object in a first human eye coordinate system of the first user's eye; calculate, by aligning the first human eye coordinate system with the first virtual camera coordinate system of a first virtual camera in the virtual world, a third coordinate of a first virtual object corresponding to the first real object in the first virtual camera coordinate system, wherein the first human eye coordinate system coincides with the first virtual camera coordinate system by aligning the reference point of the first human eye coordinate system with the reference point of the first virtual camera coordinate system; and calculate, based on the third coordinate, a fourth coordinate of the first virtual object in the virtual world coordinate system.

[0016] According to embodiments of this disclosure, the first coordinate may include the coordinates of the feature point cloud of the first real object in the depth camera coordinate system.

[0017] According to embodiments of this disclosure, the reference point of the first human eye coordinate system may be the midpoint between the eyes of the first user, and the reference point of the first virtual camera coordinate system may be the position of the first virtual camera in the virtual world.

[0018] According to embodiments of this disclosure, when a single user performs human-computer interaction, the first virtual camera coordinate system of the first virtual camera can coincide with the virtual world coordinate system of the first virtual camera in the virtual world.

[0019] According to embodiments of this disclosure, the system may further include: an interaction execution module configured to: perform an interaction between a first real object and a virtual target based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system.

[0020] According to embodiments of this disclosure, the steps of performing an interaction between a first real object and a virtual target may include: generating a real collision box and a virtual collision box based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system, respectively; recognizing a collision in response to the overlap between the coordinate range of the real collision box and the coordinate range of the virtual collision box in the virtual world coordinate system; determining a control command corresponding to the first real object based on the recognition of the collision; and controlling the virtual object to perform an operation corresponding to the control command.

[0021] According to embodiments of this disclosure, the control commands may include at least one of click commands, move commands, zoom commands, and rotate commands.

[0022] According to embodiments of this disclosure, the step of performing interaction between a first real object and a virtual target may further include: applying a collider component to the first virtual object and the virtual target based on the recognition of a collision occurrence, and applying a rigid body component to the virtual target; and simulating the physical effects of human-computer interaction based on the properties of the collider component and the rigid body component.

[0023] According to embodiments of this disclosure, the coordinate acquisition module may further be configured to: obtain the sixth coordinate of the second real object in the depth camera coordinate system and the second eye coordinate of the second user's eye in the depth camera coordinate system through the depth camera group; and the coordinate calculation module may further be configured to: calculate the seventh coordinate of the second real object in the second human eye coordinate system of the second user's eye based on the sixth coordinate and the second eye coordinate system; calculate the eighth coordinate of the second virtual object corresponding to the second real object in the second virtual camera coordinate system by aligning the second human eye coordinate system with the second virtual camera coordinate system of the second virtual camera in the virtual world, wherein the second human eye coordinate system coincides with the second virtual camera coordinate system by aligning the reference point of the second human eye coordinate system with the reference point of the second virtual camera coordinate system; and calculate the ninth coordinate of the second virtual object in the virtual world coordinate system based on the eighth coordinate and the second virtual camera coordinate system of the second virtual camera in the virtual world coordinate system.

[0024] According to embodiments of this disclosure, the interaction execution module can also be configured to: execute the interaction between the second real object, the first real object, and the virtual target based on the ninth coordinate of the second virtual object in the virtual world coordinate system, the fourth coordinate of the first virtual object in the virtual world coordinate system, and the fifth coordinate of the virtual target in the virtual world coordinate system. Attached Figure Description

[0025] The above and other aspects, features, and advantages of this disclosure will become clear from the following detailed description of some embodiments in conjunction with the accompanying drawings. In the drawings, the same reference numerals will indicate the same elements.

[0026] Figure 1 This is a flowchart illustrating a method for human-computer interaction according to an embodiment.

[0027] Figure 2A and Figure 2B This is a diagram showing a depth image according to an embodiment.

[0028] Figure 3 This is a diagram illustrating the feature point cloud of a real-world object according to an embodiment.

[0029] Figure 4 This is a diagram illustrating the calculation of the coordinates of a real-world object according to an embodiment.

[0030] Figure 5 This is a cross-sectional view showing the collision box according to an embodiment.

[0031] Figure 6 This is a diagram illustrating the interaction between a real object and a virtual target according to an embodiment.

[0032] Figure 7 This is a block diagram illustrating a system for human-computer interaction according to an embodiment.

[0033] Figures 8A to 8C This is a diagram illustrating an apparatus for performing a method for human-computer interaction according to an embodiment. Detailed Implementation

[0034] In the following description, various embodiments of the present disclosure will be described more fully with reference to the accompanying drawings. However, the present disclosure may be implemented in many different forms and should not be construed as limiting itself to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. In the drawings, the dimensions of layers and regions may be exaggerated for clarity.

[0035] The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding the disclosure of this application. For example, unless the context clearly indicates otherwise, the order of operations described herein is exemplary and not limited to those orders set forth herein, but may be changed as will become clear upon understanding the disclosure of this application. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0036] The features described herein can be implemented in various forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will become clear upon understanding the disclosure of this application.

[0037] The structural or functional descriptions of the examples disclosed herein are intended for illustrative purposes, and the examples may be implemented in various forms. The examples are not intended to be limiting, but rather to cover various modifications, equivalents, and substitutions within the scope of the claims.

[0038] Although the terms “first” or “second” can be used to describe various components, components are not limited to these terms. These terms should be used to distinguish one component from another. For example, a “first” component may be referred to as a “second” component, or similarly, a “second” component may be referred to as a “first” component.

[0039] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It should also be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0040] Unless otherwise defined, all terms used herein (including technical or scientific terms) have the same or similar meaning as commonly understood by one of ordinary skill in the art to which the examples pertain. It will also be understood that, unless expressly defined herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant field and will not be interpreted in an idealized or overly formalized sense.

[0041] Embodiments of this disclosure may be described and illustrated in terms of blocks that perform one or more of the described functions. These blocks, which may be referred to herein as units or modules, are physically implemented by analog and / or digital circuitry (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuitry, etc.) and may be driven by firmware and / or software. For example, the circuitry may be implemented in one or more semiconductor chips or on a substrate support such as a printed circuit board. The circuitry constituting a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware performing some functions of the block and a processor performing other functions of the block. Each block of an embodiment may be physically separated into two or more interacting and discrete blocks. Similarly, the blocks of an embodiment may be physically combined into more complex blocks.

[0042] In traditional extended reality (XR) or holographic scenarios, the computational load increases exponentially with the number of objects in the scene, consuming a large amount of computing resources. This not only causes delays in interactive responses, affecting the smoothness of interaction, but also seriously degrades the user experience.

[0043] Specifically, since virtual and real objects are difficult to be in the same coordinate system, a large number of complex and tedious coordinate calculations are required before and after interaction to determine their positional relationship, increasing the computational burden on the system. In addition, traditional methods do not fully consider the complex spatial relationship between the user's perspective and the position of the virtual camera. If the connection between the two cannot be accurately established and a reasonable coordinate system cannot be constructed, real objects cannot be accurately positioned in virtual space, thus affecting the accuracy and realism of human-computer interaction.

[0044] The embodiments of this disclosure are based on a specific coordinate mapping scheme. By establishing a connection between real-world objects, a depth camera group, the user's eye, a virtual camera, and virtual objects, the coordinates of real-world objects in the depth camera coordinate system are transformed to the coordinates of the corresponding virtual objects in the virtual world coordinate system. During interactive calculations, there is no need for complex transformations and calculations in different coordinate systems; instead, processing is done directly in the virtual world coordinate system, significantly reducing the computational load of human-computer interaction and improving computational efficiency.

[0045] Furthermore, the embodiments of this disclosure enable virtual and real objects to ultimately be in the same coordinate system. Real objects within the sampling range of the depth camera group can all have corresponding precise coordinates and positions in the virtual world coordinate system, accurately establishing the positional relationship between objects in the virtual world coordinate system, thereby improving the accuracy and realism of human-computer interaction.

[0046] Figure 1 This is a flowchart illustrating a method for human-computer interaction according to an embodiment.

[0047] In some embodiments, the method for human-computer interaction may include, as referred to Figure 1 Operations S100 to S400 are described above.

[0048] like Figure 1 As shown, in operation S100, the first coordinates of the first real object in the depth camera coordinate system and the first eye coordinates of the first user in the depth camera coordinate system are obtained through the depth camera group. In one embodiment, the first coordinates may include the coordinates of the feature point cloud of the first real object in the depth camera coordinate system.

[0049] According to the embodiments, real-world objects have a broad scope. Typical real-world objects may include the user themselves or user body parts (e.g., hands). For example, by capturing and analyzing hand movements and postures, a wide variety of interactive functions can be achieved. However, it should be clear that the scope of real-world objects is not limited to the user themselves and user body parts, but may also include other types of real-world objects depending on the actual application scenarios and needs of the embodiments of this disclosure. For example, in a specific interactive scenario, operating tools (e.g., chess pieces) and furniture (e.g., chairs) can be used as real-world objects.

[0050] According to the embodiment, the depth camera array can accurately acquire the position of the user's eyes (i.e., both eyes), enabling the device executing the human-computer interaction method to dynamically adjust the display screen based on the real-time position of the user's eyes. Specifically, it ensures that a certain binocular parallax is maintained when light from the screen enters the user's eyes. Binocular parallax is the visual difference caused by the different positions of the human eyes when observing objects. By precisely controlling this parallax in the display screen, a naked-eye three-dimensional (3D) visual effect can be achieved, providing the user with a more realistic and immersive interactive experience.

[0051] Furthermore, in the process of obtaining the coordinates of the user's eyes, the coordinates of the real-world object can be obtained by using a depth camera array to photograph the real-world object, thereby obtaining the coordinates of the real-world object in the depth camera coordinate system. According to an embodiment, the depth camera array may include multiple depth cameras that photograph the real-world object from different angles, enabling the acquisition of comprehensive and accurate multiple depth images of the real-world object (e.g., multi-frame, multi-view RGBD images). An RGBD image is an image format that integrates color and depth information. The RGB portion records the color information of the real-world object, presenting its color appearance, while the D portion records the distance information from the real-world object to the depth camera (i.e., depth information). The depth information allows for accurate determination of the real-world object's position in three-dimensional space.

[0052] In one embodiment, eye-tracking technology can be used to obtain the coordinates of the user's eyes in the depth camera coordinate system and the coordinates of the real-world object in the depth camera coordinate system. The apparatus for executing the human-computer interaction method can dynamically adjust the corresponding coordinates of the real-world object in the depth camera coordinate system according to the real-time position of the user's eyes. However, the coordinate acquisition method disclosed herein is not limited to this.

[0053] Furthermore, according to the embodiments, feature points of real-world objects in the depth images can be extracted using a feature point detection algorithm based on multiple captured depth images. The feature point detection algorithm can identify key points with unique features on the surface of the real-world object, and these key points exhibit relatively stable characteristics in different images. Next, feature point cloud registration technology is used to match and fuse the feature point clouds of the real-world object acquired from different viewpoints. Through this processing, a complete and accurate feature point cloud of the real-world object in the depth camera coordinate system can be obtained. Finally, based on the obtained feature point cloud, the coordinates of the real-world object in the depth camera coordinate system are calculated and processed to provide precise spatial positioning information for subsequent human-computer interaction operations.

[0054] like Figure 1 As shown, in operation S200, based on the first eye coordinates, the second coordinates of the first real object in the first user's eye coordinate system are calculated. In one embodiment, the reference point of the first eye coordinate system may be the midpoint between the first user's eyes.

[0055] For example, the reference point of the first human eye coordinate system can be set as the midpoint between the user's eyes. Using this midpoint as the origin and combining it with the pre-defined coordinate axis directions, the first human eye coordinate system can be constructed.

[0056] After establishing the first human-eye coordinate system, the second coordinates are calculated through coordinate transformation. Specifically, given the first coordinates of the first real-world object in the depth camera coordinate system and the first eye coordinates of the first user in the depth camera coordinate system, a transformation relationship is established between the depth camera coordinate system and the first human-eye coordinate system. The first coordinates of the first real-world object in the depth camera coordinate system are then transformed accordingly, and the second coordinates of the first real-world object in the first human-eye coordinate system are calculated. For example, coordinate transformation can include matrix rotation and vector translation. Matrix rotation represents the angular change between the two coordinate systems, and vector translation represents the positional offset between reference points in the two coordinate systems.

[0057] like Figure 1As shown, in operation S300, by aligning the first human eye coordinate system with the first virtual camera coordinate system of the first virtual camera in the virtual world, the third coordinate of the first virtual object corresponding to the first real object in the first virtual camera coordinate system is calculated. This is achieved by aligning the reference point of the first human eye coordinate system with the reference point of the first virtual camera coordinate system. In one embodiment, the reference point of the first virtual camera coordinate system can be the position of the first virtual camera in the virtual world.

[0058] In one embodiment, when the real-world object includes the user himself or a part of the user's body (e.g., a hand), the corresponding virtual object may include the image of the user himself or a part of the user's body (e.g., a hand) in the virtual scene. In another embodiment, when the real-world object includes an operating tool (e.g., a chess piece), the corresponding virtual object may include the image of the operating tool (e.g., a chess piece) in the virtual scene. In yet another embodiment, when the real-world object includes furniture (e.g., a chair), the corresponding virtual object may include the image of the furniture (e.g., a chair) in the virtual scene. However, the embodiments of the virtual objects disclosed herein are not limited to these.

[0059] In one embodiment, during the overlap operation, the reference point of the first human eye coordinate system is set to the midpoint between the eyes of the first user, and the reference point of the first virtual camera coordinate system is set to the position of the first virtual camera in the virtual world. Through precise adjustment, the reference points of the first human eye coordinate system and the first virtual camera coordinate system are perfectly aligned in space, thereby achieving overlap between them. Specifically, eye-tracking technology can acquire the position and movement of the user's eyes in real time, ensuring that the position of the virtual camera precisely corresponds to the user's visual center; that is, the reference points of the first human eye coordinate system and the first virtual camera coordinate system remain consistent at all times.

[0060] When the two coordinate systems coincide, the second coordinate of the first real object in the first human eye coordinate system is calculated based on the first real object. Then, the third coordinate of the first virtual object corresponding to the first real object in the first virtual camera coordinate system is calculated through coordinate transformation.

[0061] like Figure 1 As shown, in operation S400, based on the first virtual camera coordinates in the virtual world coordinate system of the virtual world, the fourth coordinates of the first virtual object in the virtual world coordinate system are calculated based on the third coordinates.

[0062] For example, in the construction system of light field images, the virtual camera's coordinates in the virtual world coordinate system are predetermined and known. Based on the third coordinate of the first virtual object corresponding to the first real object in the first virtual camera coordinate system, the fourth coordinate of the first virtual object in the virtual world coordinate system is calculated through coordinate transformation.

[0063] The embodiments of this disclosure are based on a specific coordinate mapping scheme. By establishing a connection between real-world objects, a depth camera group, the user's eye, a virtual camera, and virtual objects, the coordinates of real-world objects in the depth camera coordinate system are transformed to the coordinates of the corresponding virtual objects in the virtual world coordinate system. This significantly reduces the computational load of human-computer interaction. Because complex transformations and calculations in different coordinate systems are no longer required during interactive calculations, processing is performed directly in the virtual world coordinate system, improving computational efficiency and enhancing the accuracy and realism of human-computer interaction.

[0064] Furthermore, the embodiments of this disclosure employ a method of mapping objects in real-world scenes to virtual space. This specific mapping method is more in line with the needs of operating on data based on virtual light field images. It not only allows for various operations and interactions with real-world objects in virtual space, but also enables more complex and refined interactions between real-world and virtual objects.

[0065] In one embodiment, when a single user interacts with the computer, the first virtual camera coordinate system of the first virtual camera may coincide with the virtual world coordinate system of the first virtual camera in the virtual world.

[0066] For example, when only a single user is interacting with the computer, to simplify calculations and improve the accuracy of the interaction, the coordinate system of the first virtual camera can be set to coincide with the coordinate system of the first virtual camera in the virtual world. That is, the reference points of the two coordinate systems are selected at the same location, and the coordinate axis directions are consistent. This eliminates the need for complex cross-coordinate system transformations when processing the coordinates of virtual objects; the coordinate information of the virtual object in the virtual camera coordinate system can be directly used for related operations, further significantly reducing the computational load of human-computer interaction.

[0067] Furthermore, the method for human-computer interaction according to the embodiments may further include: performing an interaction between a first real object and a virtual target based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system.

[0068] In one embodiment, a virtual target can be a key object that enables interaction with virtual objects in a virtual world coordinate system, and the virtual target can be a carefully constructed object with specific interactive attributes within the virtual world coordinate system. Specifically, virtual targets may include city buildings, Rubik's Cubes, robots, building blocks, chess pieces, etc., but the embodiments of virtual targets disclosed herein are not limited to these.

[0069] In one embodiment, the interaction between the first real object and the virtual target can be performed through a collision box generation operation, a collision recognition operation, a control command determination operation, and a virtual object control operation.

[0070] In one embodiment, the collision box generation operation may include generating a real collision box and a virtual collision box based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system, respectively. Specifically, for the first virtual object, its position and approximate shape characteristics in the virtual world coordinate system are determined based on its fourth coordinate, and a spatial geometry that can completely enclose the first virtual object is constructed as the real collision box. For example, if the first virtual object is approximately a cuboid shape, a corresponding cuboid collision box can be constructed based on the length, width, and height information determined by its fourth coordinate; if the shape of the first virtual object is more complex, multiple simple geometric shapes can be combined to construct the real collision box to ensure that it can accurately represent the collision range of the first virtual object in the virtual world. Similarly, for the virtual target, its position and shape characteristics in the virtual world coordinate system are determined based on its fifth coordinate, and a corresponding virtual collision box is constructed.

[0071] In one embodiment, the collision detection operation may include recognizing a collision in response to the overlap between the coordinate ranges of the real-world collisionbox and the virtual collisionbox in the virtual-world coordinate system. For example, this can be achieved by continuously calculating the coordinates of each vertex of the real-world and virtual collisionboxes in the virtual-world coordinate system and determining whether at least one vertex is located within the coordinate range of the other collisionbox, or by employing other efficient collision detection algorithms, such as axis-aligned bounding box (AABB) collision detection algorithms or oriented bounding box (OBB) collision detection algorithms, to quickly and accurately determine whether a collision has occurred.

[0072] In one embodiment, the control command determination operation may include determining a control command corresponding to a first real-world object based on the detection of a collision. Once a collision is detected, the system determines the control command corresponding to the first real-world object based on the collision event.

[0073] In one embodiment, virtual object control operations may include controlling the virtual object to perform operations corresponding to control instructions.

[0074] In one embodiment, the control command may include at least one of a click command, a move command, a zoom command, and a rotate command. For example, if the control command is a click command, the virtual object can perform a corresponding click feedback operation, such as changing color, playing a specific sound effect, or displaying a prompt message; if the control command is a move command, the virtual object will move in the virtual world according to a specified direction and distance; if it is a zoom command, the virtual object will be enlarged or reduced according to the command; if it is a rotate command, the virtual object will rotate around a specified axis.

[0075] The embodiments of this disclosure allow for real-time interaction with virtual targets using only real-world objects, greatly enhancing the realism and immersion of the interaction. For example, in a virtual game scene, a user can interact with a virtual character or virtual prop, which is a virtual target, through natural hand movements (representing a real-world object), just as in the real world.

[0076] Furthermore, the interaction between the first real-world object and the virtual target can be performed through simulation. In one embodiment, the simulation operation may include: applying a collider component to the real-world and virtual collision boxes based on the recognition of a collision, and applying a rigid body component to the virtual object; and simulating the physical effects of human-computer interaction based on the properties of the collider component and the rigid body component.

[0077] For example, rigid body components are used to give virtual objects physical properties, allowing them to move and interact in the virtual world according to the laws of physics. After applying a rigid body component to a virtual object, its physical parameters such as mass, friction, and elasticity need to be set. By properly setting these parameters, the movement and interaction of the virtual object in the virtual world can be made more consistent with real-world physics.

[0078] Furthermore, the method for human-computer interaction according to the embodiments may further include: obtaining, through a depth camera group, the sixth coordinate of a second real object in the depth camera coordinate system and the second eye coordinate of the second user's eye in the depth camera coordinate system; calculating, based on the sixth coordinate and the second eye coordinate, the seventh coordinate of the second real object in the second eye coordinate system of the second user's eye; calculating, by aligning the second eye coordinate system with the second virtual camera coordinate system of the second virtual camera in the virtual world, the eighth coordinate of the second virtual object corresponding to the second real object in the second virtual camera coordinate system, wherein the second eye coordinate system coincides with the second virtual camera coordinate system by aligning the reference point of the second eye coordinate system with the reference point of the second virtual camera coordinate system; and calculating, based on the eighth coordinate and the second virtual camera coordinate of the second virtual camera in the virtual world coordinate system, the ninth coordinate of the second virtual object in the virtual world coordinate system.

[0079] In one embodiment, the interaction between the second real object, the first real object, and the virtual target can be performed based on the ninth coordinate of the second virtual object in the virtual world coordinate system, the fourth coordinate of the first virtual object in the virtual world coordinate system, and the fifth coordinate of the virtual target in the virtual world coordinate system.

[0080] In other words, when multiple users interact with the same system, similar to the method for calculating the fourth coordinate of the first virtual object corresponding to the first real object in the virtual world coordinate system, the coordinates of the second virtual object corresponding to the second real object in the virtual world coordinate system can be calculated. In the virtual world coordinate system, based on the coordinates of the first virtual object, the second virtual object, and the virtual target, interactions between the second real object, the first real object, and the virtual target are executed. The embodiments of this disclosure, while supporting multi-user collaboration, can also significantly reduce computational load while improving the accuracy and realism of human-computer interaction.

[0081] Additionally, the method for human-computer interaction according to the embodiments may further include: constructing an anchor point system based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual object in the virtual world coordinate system to synchronize and update the coordinates after interaction; and performing cloud synchronization based on the constructed anchor point system.

[0082] In one embodiment, the anchor point system can be constructed by: determining the origin of the virtual world coordinate system as the basic anchor point of the anchor point system for coordinate reference; and determining a reference point as a sub-anchor point of the anchor point system within the neighborhood of the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual object in the virtual world coordinate system to assist in the synchronization and updating of the coordinates of the anchor point system.

[0083] For example, by combining basic anchor points and sub-anchor points to construct an anchor point system, the coordinates of virtual and real objects mounted under the same anchor point system can be updated with the same coordinate transformation after changes, maintaining the relative invariance of virtual and real objects mounted under the same anchor point system, and the state of virtual and real objects can be tracked uniformly.

[0084] In one embodiment, cloud synchronization can be performed by: uploading the base anchor point, sub-anchor point, the fourth coordinate of the first virtual object in the virtual world coordinate system, and the fifth coordinate of the virtual object in the virtual world coordinate system to the cloud server; and in response to requests from other users, downloading the base anchor point, sub-anchor point, the fourth coordinate of the first virtual object in the virtual world coordinate system, and the fifth coordinate of the virtual object in the virtual world coordinate system from the cloud server.

[0085] For example, the data can be packaged in a specific format and sent to a cloud server via a wireless network. The cloud server then receives, stores, and manages the data. Other users can initiate synchronization requests on their devices. The devices send request commands to the cloud server via the network, and the cloud server, upon receiving the request, sends the corresponding data back to the user devices.

[0086] The embodiments of this disclosure utilize anchor point system construction and cloud synchronization to update the coordinates of the interaction, further improving the continuity and realism of the interaction. The interaction results are uploaded to the cloud, and the view segmentation characteristics of the device (e.g., a light field display device) itself, which performs the method for human-computer interaction, support the synchronization and fusion of light field images from multiple perspectives, multiple users, and even multiple devices. This overcomes the geographical limitations of interaction result feedback and rendering in the light field image, significantly improving the immersiveness of the light field interaction scene.

[0087] Figure 2A and Figure 2B This is a diagram showing a depth image according to an embodiment.

[0088] The depth image according to the embodiment is based on multi-frame, multi-angle RGBD images of the real environment. Specifically, the same real-world scene is captured by a depth camera group. The depth camera group performs multiple capture operations at different angles and similar time intervals.

[0089] like Figure 2A As shown, in an indoor scene, multiple frames of RGBD images are acquired by a depth camera group at similar time points and from different angles. These images are then processed by subsequent image processing algorithms (such as image registration and fusion algorithms) to generate a depth image that accurately reflects the depth information of the indoor scene.

[0090] like Figure 2B As shown, a depth image that accurately reflects the posture of a real-world object (e.g., a user's hand) is generated by using a depth camera array to capture images of the object at different angles and similar time intervals.

[0091] The depth images obtained through the above methods, by incorporating information from multiple angles, can more comprehensively and accurately present the spatial position and depth relationship of objects in the real scene, providing a more reliable data foundation for subsequent applications such as human-computer interaction based on the depth images.

[0092] Figure 3 This is a diagram illustrating the feature point cloud of a real-world object according to an embodiment.

[0093] In one embodiment, after acquiring the depth image, feature points of the real-world object are detected using a feature point detection algorithm (e.g., ORB algorithm). After processing by this algorithm, a series of feature points of the real-world object can be obtained in each frame of the image. Then, a feature point cloud registration algorithm (e.g., Normal Distribution Transform (NDT), Iterative Closest Point (ICP)) is used to quickly register the series of feature points into a feature cloud.

[0094] Through the above feature point detection and feature point cloud registration processes, the coordinates of the feature point cloud of the real object in the depth camera coordinate system can be obtained.

[0095] Furthermore, the movement of real-world objects can be tracked using their feature point clouds. During interaction, the position of the feature point cloud changes accordingly as the real-world object moves. By monitoring the positional changes of the feature point cloud in real time and utilizing coordinate mapping relationships, the movement trajectory and state of the real-world object can be accurately tracked.

[0096] like Figure 3 As shown, the feature point cloud of a real-world object is a collection of multiple representative feature points (or keypoints) of that object. These feature points each have unique coordinate positions in the depth camera coordinate system. For example, for a user's hand, its surface joints, fingertips, etc., can all serve as keypoints. By acquiring the coordinate information of these feature points in the depth camera coordinate system, the state of the real-world object in space can be more accurately located and described. For instance, knowing the coordinates of each feature point of the hand allows us to determine the hand's position, posture, and the relative positional relationships between different parts of the hand, thus providing accurate data support for subsequent applications such as gesture recognition and human-computer interaction.

[0097] Figure 4 This is a diagram illustrating the calculation of the coordinates of a real-world object according to an embodiment.

[0098] like Figure 4 As shown, return to reference Figure 1 Operation S100 can obtain the first coordinates (x3, y3, z3) of the real object in the depth camera coordinate system and the eye coordinates (x1, y1, z1) and (x2, y2, z2) of the user's eye in the depth camera coordinate system through the depth camera group. At this time, the coordinates of the depth camera group (e.g., the center of multiple depth cameras included in the depth camera) in the depth camera coordinate system can be (0, 0, 0).

[0099] Return to reference Figure 1Operation S200 can calculate the second coordinates (x3', y3', z3') of the real object in the user's eye coordinate system based on the first coordinates (x3, y3, z3). In one embodiment, the reference point (0, 0, 0) of the eye coordinate system can be the midpoint between the user's eyes. In this case, the coordinates of the depth camera group in the eye coordinate system can be (0, 0, 0).

[0100] Return to reference Figure 1 By operating S300, the third coordinates (x3', y3', z3') of the virtual object corresponding to the real object in the virtual camera coordinate system can be calculated by aligning the human eye coordinate system with the virtual camera coordinate system of the virtual camera in the virtual world. This is achieved by aligning the reference point (0, 0, 0) of the human eye coordinate system with the reference point (0, 0, 0) of the virtual camera coordinate system.

[0101] When only a single user is interacting with the computer, the virtual camera's coordinate system can be set to coincide with the virtual world coordinate system of the virtual camera. Interactions between the real-world object and the virtual target can be performed based on the third coordinate (x3', y3', z3') of the virtual object in the virtual camera coordinate system and the coordinates (x4', y4', z4') of the virtual target in the virtual camera coordinate system.

[0102] Figure 5 This is a cross-sectional view showing the collision box according to an embodiment.

[0103] like Figure 5 As shown, in order to detect collisions between objects more efficiently and accurately, the collision box (or bounding box) according to the embodiment has a variety of different types to adapt to objects of different shapes and characteristics and diverse application needs. Specifically, the collision box (or bounding box) according to the embodiment may include axial bounding boxes, spherical bounding boxes, rectangular bounding boxes, discrete directional bounding boxes, etc.

[0104] Axially bounding boxes are a relatively simple and commonly used type of collision box. They are cuboids parallel to the coordinate axes. When constructing an axially bounding box, first determine the minimum and maximum values ​​of the object along each coordinate axis in the 3D coordinate system. Then, using these minimum and maximum values ​​as boundaries, construct a cuboid parallel to the coordinate axes; this cuboid is the axially bounding box of the object.

[0105] A spherical bounding box is a sphere constructed with the center of the object as its center and the distance from the object to its farthest point as its radius. To construct a spherical bounding box, the coordinates of the object's center point must first be determined, which can be obtained by calculating the average of the coordinates of all the object's vertices. Then, the distances from each vertex to the center point are calculated, and the maximum value is found; this maximum value is the radius of the spherical bounding box.

[0106] A rectangular bounding box is a oriented cuboid that can optimally fit the actual shape and orientation of an object. Unlike axial bounding boxes, the sides of a rectangular bounding box are not necessarily parallel to the coordinate axes.

[0107] Discrete-oriented bounding boxes (DIBCs) are a more general type of bounding box, an extension of axial bounding boxes. When constructing a DIBC, a set of discrete direction vectors is first determined. Then, for each direction vector, the minimum and maximum values ​​of the object's projection in that direction are calculated. These minimum and maximum values ​​constitute the boundaries of the DIBC in each direction.

[0108] Figure 6 This is a diagram illustrating the interaction between a real object and a virtual target according to an embodiment.

[0109] According to the embodiment, in order to achieve efficient and accurate interaction between real objects and virtual targets, a collider component is applied to the real collision box and the virtual collision box, a rigid body component is applied to the virtual object, and the physical effects of human-computer interaction are simulated based on the properties of the collider component and the rigid body component.

[0110] Specifically, the collider component enables colliders to interact with other objects, accurately sensing collisions and reacting accordingly. The rigid body component enables virtual objects to simulate real-world physics effects in the virtual world, allowing them to move and change according to real-world physical laws. For example, the rigid body component can simulate physical properties such as mass, drag, angular drag, and gravity.

[0111] The embodiments of this disclosure achieve efficient, accurate, and realistic interaction between real-world objects and virtual targets by applying collider components and rigid body components.

[0112] Figure 7 This is a block diagram illustrating a system for human-computer interaction according to an embodiment.

[0113] like Figure 7 As shown, the system 10 for human-computer interaction may include a depth camera group (not shown), a coordinate acquisition module 11, a coordinate calculation module 12, and an interaction execution module.

[0114] In one embodiment, the coordinate acquisition module 11 may be configured to: obtain the first coordinates of the first real object in the depth camera coordinate system and the first eye coordinates of the first user's eye in the depth camera coordinate system through the depth camera group.

[0115] In one embodiment, the coordinate calculation module 12 may be configured to: calculate a second coordinate of a first real object in a first human eye coordinate system based on the first eye coordinates; calculate a third coordinate of a first virtual object corresponding to the first real object in a first virtual camera coordinate system by aligning the first human eye coordinate system with the first virtual camera coordinate system of the first virtual camera in the virtual world, wherein the first human eye coordinate system coincides with the first virtual camera coordinate system by aligning the reference point of the first human eye coordinate system with the reference point of the first virtual camera coordinate system; and calculate a fourth coordinate of the first virtual object in a virtual world coordinate system based on the third coordinate of the first virtual camera in the virtual world coordinate system.

[0116] In one embodiment, the first coordinates may include the coordinates of the feature point cloud of the first real-world object in the depth camera coordinate system.

[0117] In one embodiment, the reference point of the first human eye coordinate system may be the midpoint between the eyes of the first user, and the reference point of the first virtual camera coordinate system may be the position of the first virtual camera in the virtual world.

[0118] In one embodiment, when a single user interacts with the computer, the first virtual camera coordinate system of the first virtual camera may coincide with the virtual world coordinate system of the first virtual camera in the virtual world.

[0119] In one embodiment, the interaction execution module 13 can be configured to perform an interaction between the first real object and the virtual target based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system.

[0120] In one embodiment, the interaction between the first real object and the virtual target can be performed through the following processes: generating a real collision box and a virtual collision box based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system, respectively; recognizing a collision in response to the overlap between the coordinate range of the real collision box and the coordinate range of the virtual collision box in the virtual world coordinate system; determining a control command corresponding to the first real object based on the recognition of the collision; and controlling the virtual object to perform the operation corresponding to the control command.

[0121] In one embodiment, the control commands may include at least one of click commands, move commands, zoom commands, and rotate commands.

[0122] In one embodiment, the interaction between the first real object and the virtual target can also be performed by: applying a collider component to the real collision box and the virtual collision box based on the recognition of a collision, and applying a rigid body component to the virtual object; and simulating the physical effects of human-computer interaction based on the properties of the collider component and the rigid body component.

[0123] In one embodiment, the coordinate acquisition module 11 may also be configured to: obtain the sixth coordinate of the second real object in the depth camera coordinate system and the second eye coordinate of the second user's eye in the depth camera coordinate system through the depth camera group.

[0124] In one embodiment, the coordinate calculation module 12 may also be configured to: calculate the seventh coordinate of the second real object in the second human eye coordinate system of the second user's eyes based on the sixth coordinate, according to the second eye coordinate; calculate the eighth coordinate of the second virtual object corresponding to the second real object in the second virtual camera coordinate system by aligning the second human eye coordinate system with the second virtual camera coordinate system of the second virtual camera in the virtual world, wherein the second human eye coordinate system and the second virtual camera coordinate system are aligned to make the second human eye coordinate system coincide; and calculate the ninth coordinate of the second virtual object in the virtual world coordinate system based on the eighth coordinate, according to the second virtual camera coordinate of the second virtual camera in the virtual world coordinate system.

[0125] In one embodiment, the interaction execution module 13 can also be configured to: execute the interaction between the second real object, the first real object, and the virtual target based on the ninth coordinate of the second virtual object in the virtual world coordinate system, the fourth coordinate of the first virtual object in the virtual world coordinate system, and the fifth coordinate of the virtual target in the virtual world coordinate system.

[0126] Additionally, the system 10 for human-computer interaction may also include an anchor point management module and a cloud synchronization module. The anchor point management module can be configured to construct an anchor point system based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual object in the virtual world coordinate system, to synchronize and update the coordinates after interaction. The cloud synchronization module can be configured to perform cloud synchronization based on the constructed anchor point system.

[0127] In one embodiment, the anchor point management module can also be configured to: determine the origin of the virtual world coordinate system as the basic anchor point of the anchor point system for coordinate reference of the anchor point system; and determine a reference point as a sub-anchor point of the anchor point system within the neighborhood of the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual object in the virtual world coordinate system for assisting in the synchronization and updating of the coordinates of the anchor point system.

[0128] In one embodiment, the cloud synchronization module can also be configured to: upload the base anchor point, sub-anchor point, the fourth coordinate of the first virtual object in the virtual world coordinate system, and the fifth coordinate of the virtual object in the virtual world coordinate system to the cloud server; and in response to requests from other users, download the base anchor point, sub-anchor point, the fourth coordinate of the first virtual object in the virtual world coordinate system, and the fifth coordinate of the virtual object in the virtual world coordinate system from the cloud server.

[0129] Figures 8A to 8C This is a diagram illustrating an apparatus for performing a method for human-computer interaction according to an embodiment.

[0130] Figure 8A This is an example of a light field display panel illustrating a method for human-computer interaction according to an embodiment. The light field display panel can accurately map real-world objects (e.g., a user's hand) to a virtual light field space using the method for human-computer interaction according to the embodiment. This mapping ensures that real-world objects and virtual targets (e.g., city buildings) are in a unified coordinate system at the system level. It ensures that both real-world objects and virtual targets can be positioned and motion described in the same coordinate system, thereby significantly reducing computational load while improving the accuracy and realism of human-computer interaction.

[0131] Figure 8B This is an example of an interactive chessboard illustrating a method for human-computer interaction according to an embodiment. The interactive chessboard can accurately map real-world objects (e.g., a user's hand) to a virtual space of light field using the method for human-computer interaction according to the embodiment. This mapping ensures that real-world objects and virtual targets (e.g., chess pieces) are in a unified coordinate system at the system level. It ensures that both real-world objects and virtual targets can be positioned and motion described in the same coordinate system, thereby significantly reducing computational load while improving the accuracy and realism of human-computer interaction.

[0132] Furthermore, to maintain the accurate position of the pieces on the chessboard, the center of the chessboard is selected as the reference anchor point for processing on-screen feedback. The point clouds of both the chessboard and the pieces are attached to this anchor point, and specific algorithms and mechanisms are used to maintain their relative pose invariance. In this way, regardless of how the user manipulates the pieces, their position and pose on the chessboard remain relatively stable, providing the user with a more realistic and accurate interactive experience.

[0133] In single-device, multi-user scenarios, view segmentation technology allows different users to view the same chessboard from different perspectives. For example, in a two-player game, when one user moves their hand holding a piece and places it on a square on the board, this interaction is viewed from the opposite user's perspective as the opponent making the move. This view segmentation technology enables different users to have different visual experiences on the same device, enhancing the fun and realism of the interaction.

[0134] In multi-user, multi-device scenarios, two players (e.g., player A and player B) view the same chessboard through different devices. Without light field image synchronization, the two devices display the same chessboard from different perspectives. However, after light field image synchronization, the images seen by each player are consistent with those seen by each player in a single-device, multi-user scenario—that is, they are viewing the "same" light field chessboard from opposite perspectives. Light field image synchronization technology ensures a consistent light field chessboard interaction experience for users on different devices, breaking down device limitations and enabling cross-device human-computer interaction.

[0135] Figure 8C This is an example of an interactive video device illustrating a method for human-computer interaction according to an embodiment. The interactive video device can accurately map a real-world object (e.g., a user) to a virtual space of light field using the method for human-computer interaction according to the embodiment. This mapping ensures that the real-world object and another real-world object using another interactive video device are located in a unified coordinate system at the system level. It ensures that both the real-world object of this interactive video device and another real-world object of another interactive video device can be positioned and motion described in the same coordinate system, thereby significantly reducing computational load while improving the accuracy and realism of human-computer interaction.

[0136] Furthermore, due to the view segmentation feature of interactive video devices, they not only support interaction between two users through two interactive video devices, but also support interactive video conferencing for multiple users. The maximum number of users they can support is limited by a combination of factors, including the rendering and processing performance of the interactive video device and the computing power of the cloud synchronization server.

[0137] Although some exemplary embodiments have been described, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this disclosure as set forth in the claims.

Claims

1. A method for human-computer interaction, comprising: Using a depth camera group, obtain the first coordinates of the first real object in the depth camera coordinate system and the first eye coordinates of the first user in the depth camera coordinate system; Based on the first eye coordinates, calculate the second coordinates of the first real object in the first eye coordinate system of the first user's eyes; By aligning the first human eye coordinate system with the first virtual camera coordinate system of the first virtual camera in the virtual world, the third coordinate of the first virtual object corresponding to the first real object in the first virtual camera coordinate system is calculated. The first human eye coordinate system and the first virtual camera coordinate system are aligned to make the first human eye coordinate system coincide. as well as Based on the coordinates of the first virtual camera in the virtual world coordinate system, and based on the third coordinate, calculate the fourth coordinate of the first virtual object in the virtual world coordinate system.

2. The method according to claim 1, wherein, The first coordinates include the coordinates of the feature point cloud of the first real object in the depth camera coordinate system.

3. The method according to claim 1, wherein, The reference point of the first-person eye coordinate system is the midpoint between the first user's eyes, and The reference point of the first virtual camera coordinate system is the position of the first virtual camera in the virtual world.

4. The method according to claim 1, wherein, When a single user interacts with the computer, the coordinate system of the first virtual camera coincides with the coordinate system of the first virtual camera in the virtual world.

5. The method according to claim 1, further comprising: Based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system, the interaction between the first real object and the virtual target is executed.

6. The method according to claim 5, wherein, The steps for performing the interaction between the first real object and the virtual target include: The real-world collision box and the virtual collision box are generated based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system, respectively. The collision is identified when the coordinate range of the real-world collision box in the virtual world coordinate system overlaps with the coordinate range of the virtual collision box in the virtual world coordinate system. Based on the occurrence of a collision, the control command corresponding to the first real-world object is determined; and Control the virtual object to perform the operation corresponding to the control command.

7. The method according to claim 6, wherein, The control commands include at least one of click commands, move commands, zoom commands, and rotate commands.

8. The method according to claim 6, wherein, The steps for performing the interaction between the first real object and the virtual target also include: Based on the collision detection, a collision body component is applied to the first virtual object and the virtual target, and a rigid body component is applied to the virtual target; and Based on the properties of collider and rigid body components, simulate the physical effects of human-computer interaction.

9. The method according to claim 1, further comprising: Using a depth camera group, obtain the sixth coordinate of the second real object in the depth camera coordinate system and the second eye coordinate of the second user in the depth camera coordinate system; Based on the second eye coordinates and the sixth coordinate, calculate the seventh coordinate of the second real object in the second eye coordinate system of the second user's eyes; By aligning the second human eye coordinate system with the second virtual camera coordinate system of the second virtual camera in the virtual world, the eighth coordinate of the second virtual object corresponding to the second real object in the second virtual camera coordinate system is calculated. The second human eye coordinate system and the second virtual camera coordinate system are aligned to make the second human eye coordinate system coincide. as well as Based on the coordinates of the second virtual camera in the virtual world coordinate system, and using the eighth coordinate, calculate the ninth coordinate of the second virtual object in the virtual world coordinate system.

10. The method of claim 9, further comprising: Based on the ninth coordinate of the second virtual object in the virtual world coordinate system, the fourth coordinate of the first virtual object in the virtual world coordinate system, and the fifth coordinate of the virtual target in the virtual world coordinate system, the interaction between the second real object, the first real object, and the virtual target is executed.

11. A system for human-computer interaction, comprising: Depth camera group; The coordinate acquisition module is configured to: obtain the first coordinates of the first real object in the depth camera coordinate system and the first eye coordinates of the first user in the depth camera coordinate system through the depth camera group; as well as The coordinate calculation module is configured as follows: Based on the first eye coordinates, calculate the second coordinates of the first real object in the first eye coordinate system of the first user's eyes; By aligning the first human eye coordinate system with the first virtual camera coordinate system of the first virtual camera in the virtual world, the third coordinate of the first virtual object corresponding to the first real object in the first virtual camera coordinate system is calculated. The first human eye coordinate system and the first virtual camera coordinate system are aligned to make the first human eye coordinate system coincide. and Based on the coordinates of the first virtual camera in the virtual world coordinate system, and based on the third coordinate, calculate the fourth coordinate of the first virtual object in the virtual world coordinate system.

12. The system according to claim 11, wherein, The first coordinates include the coordinates of the feature point cloud of the first real object in the depth camera coordinate system.

13. The system according to claim 11, wherein, The reference point of the first-person eye coordinate system is the midpoint between the first user's eyes, and The reference point of the first virtual camera coordinate system is the position of the first virtual camera in the virtual world.

14. The system according to claim 11, wherein, When a single user interacts with the computer, the coordinate system of the first virtual camera coincides with the coordinate system of the first virtual camera in the virtual world.

15. The system according to claim 11, further comprising: The interaction execution module is configured to execute the interaction between the first real object and the virtual target based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system.

16. The system according to claim 15, wherein, The steps for performing the interaction between the first real object and the virtual target include: The real-world collision box and the virtual collision box are generated based on the fourth coordinate of the first virtual object in the virtual world coordinate system and the fifth coordinate of the virtual target in the virtual world coordinate system, respectively. The collision is identified when the coordinate range of the real-world collision box in the virtual world coordinate system overlaps with the coordinate range of the virtual collision box in the virtual world coordinate system. Based on the occurrence of a collision, the control command corresponding to the first real-world object is determined; and Control the virtual object to perform the operation corresponding to the control command.

17. The system according to claim 16, wherein, The control commands include at least one of click commands, move commands, zoom commands, and rotate commands.

18. The system according to claim 16, wherein, The steps for performing the interaction between the first real object and the virtual target also include: Based on the collision detection, a collision body component is applied to the first virtual object and the virtual target, and a rigid body component is applied to the virtual target; and Based on the properties of collider and rigid body components, simulate the physical effects of human-computer interaction.

19. The system according to claim 11, wherein, The coordinate acquisition module is also configured to: obtain, via the depth camera group, the sixth coordinate of the second real-world object in the depth camera coordinate system and the second eye coordinate of the second user's eye in the depth camera coordinate system, and The coordinate calculation module is also configured as follows: Based on the second eye coordinates and the sixth coordinate, calculate the seventh coordinate of the second real object in the second eye coordinate system of the second user's eyes; By aligning the second human eye coordinate system with the second virtual camera coordinate system of the second virtual camera in the virtual world, the eighth coordinate of the second virtual object corresponding to the second real object in the second virtual camera coordinate system is calculated. This is achieved by aligning the reference point of the second human eye coordinate system with the reference point of the second virtual camera coordinate system, thus ensuring that the two coordinate systems coincide. Based on the coordinates of the second virtual camera in the virtual world coordinate system, and using the eighth coordinate, calculate the ninth coordinate of the second virtual object in the virtual world coordinate system.

20. The system according to claim 19, wherein, The interactive execution module is also configured as follows: Based on the ninth coordinate of the second virtual object in the virtual world coordinate system, the fourth coordinate of the first virtual object in the virtual world coordinate system, and the fifth coordinate of the virtual target in the virtual world coordinate system, the interaction between the second real object, the first real object, and the virtual target is executed.

Images