Man-machine interaction method, device and equipment based on medium-free holographic technology and medium

Through medium-free holographic technology, the projection of stereoscopic images and sensors to collect interactive data is achieved, contactless human-computer interaction is solved, and the operation inconvenience and information leakage problems of the existing technology in specific occasions is improved, and the diversity and accuracy of the interaction is improved.

CN119002689APending Publication Date: 2024-11-22江西像航科技有限公司 +1
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410934907.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing human-computer interaction technology is inconvenient to operate in specific occasions (such as surgery, sterile laboratories, high temperatures or radiation environments) and lacks diversity, which may lead to command errors and personal information leakage.

Method used

Using a human-computer interaction method based on medium-free holographic technology, a three-dimensional image of a target object is projected and a sensor is used to collect user interaction data, and interactive operation instructions are generated to achieve contactless three-dimensional interaction operation.

Benefits of technology

It improves the operational diversity and accuracy of human-computer interaction, reduces operation difficulty, avoids the limitations of wearable devices and environmental factors, and enhances the user's interactive experience and security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119002689A_ABST
    Figure CN119002689A_ABST
Patent Text Reader

Abstract

The invention provides a human-computer interaction method, device and equipment based on a medium-free holographic technology and a medium, the method projects a target object to a projection display area and displays a three-dimensional image of a target user through the medium-free holographic technology, so that the user can naturally and visually interact with the three-dimensional image without physical contact, and the user experience is improved. And the immersive experience is improved. The interactive operation data of the target user is collected through the sensor device, the interactive action of the target user can be accurately captured, and accurate interactive operation is achieved. Through the interaction operation instruction, the corresponding interaction operation is executed on the stereo image, non-contact interaction operation identification is realized, user operation is simplified, operation difficulty is reduced, limitation of other wearable devices and environmental factors on user operation is avoided, and applicability and operation diversity of man-machine interaction operation are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a human-computer interaction method, device, equipment and medium based on medium-free holographic technology. Background Art

[0002] Human-Computer Interaction Techniques refers to the technology that enables people to communicate with computers in an effective way through computer input and output devices. Human-Computer Interaction Techniques include machines providing people with a large amount of relevant information and prompts through output or display devices, and people inputting relevant information to machines through input devices, answering questions and prompts, etc.

[0003] However, existing human-computer interaction technologies require users to operate on computer devices through physical contact to achieve human-computer interaction, such as operating through touch screens or buttons. However, in certain specific occasions, such as surgical procedures, sterile laboratories, high-temperature places, mine operations, and radiation environments, users are required to wear protective or isolating clothing, especially when wearing thick gloves. The existing human-computer interaction methods will bring great inconvenience, and there will be problems such as inaccurate touch recognition leading to command errors. In addition, human-computer interaction operations in public places may leave fingerprint information, and there is a risk of user personal information being leaked. Therefore, the interactive operations of existing human-computer interaction technologies lack diversity and cannot meet the usage needs of different users.

[0004] Therefore, how to solve the poor diversity of interactive operations in current human-computer interaction technology has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a human-computer interaction method, device, equipment and medium based on medium-free holographic technology, aiming to improve the diversity of interactive operations of human-computer interaction technology.

[0006] In a first aspect, the present application provides a human-computer interaction method based on medium-free holographic technology, the method comprising:

[0007] Based on the medium-free holographic technology, the target object is projected into the projection display area to obtain a three-dimensional image of the target object;

[0008] Based on sensor devices, collect interactive operation data of target users and generate interactive operation instructions;

[0009] Based on the interactive operation instruction, a corresponding interactive operation is performed on the stereoscopic image to complete contactless human-computer interaction.

[0010] In a second aspect, the present application further provides a human-computer interaction device based on medium-free holographic technology, wherein the human-computer interaction device based on medium-free holographic technology comprises:

[0011] A holographic projection module is used to project a target object onto a projection display area based on a medium-free holographic technology to obtain a stereoscopic image of the target object;

[0012] An interactive instruction generation module is used to collect interactive operation data of a target user based on a sensor device and generate interactive operation instructions;

[0013] The human-computer interaction module is used to perform corresponding interactive operations on the stereoscopic image based on the interactive operation instruction to complete contactless human-computer interaction.

[0014] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the human-computer interaction method based on the medium-free holographic technology as described above are implemented.

[0015] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the human-computer interaction method based on the medium-free holographic technology as described above are implemented.

[0016] The present application provides a human-computer interaction method, device, equipment and storage medium based on medium-free holographic technology. The method of the present application includes projecting a target object onto a projection display area based on medium-free holographic technology to obtain a stereoscopic image of the target object; based on a sensor device, collecting the interactive operation data of the target user and generating an interactive operation instruction; based on the interactive operation instruction, performing a corresponding interactive operation on the stereoscopic image to complete contactless human-computer interaction. In the above manner, the present application projects the target object onto a projection display area through medium-free holographic technology to display a stereoscopic image of the target user, so that the user can interact with the three-dimensional stereoscopic image naturally and intuitively without physical contact, thereby improving the immersive experience. By collecting the interactive operation data of the target user through the sensor device, the interactive actions of the target user can be accurately captured to achieve precise interactive operation. Through the interactive operation instruction, the corresponding interactive operation is performed on the stereoscopic image to achieve contactless interactive operation recognition, simplifying the user operation, reducing the difficulty of operation, avoiding the restrictions on user operation by other wearable devices and environmental factors, and improving the applicability and operation diversity of human-computer interactive operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a first embodiment of a human-computer interaction method based on medium-free holographic technology provided by the present application;

[0019] Figure 2 A schematic flow chart of a second embodiment of a human-computer interaction method based on medium-free holographic technology provided by the present application;

[0020] Figure 3 It is a structural schematic diagram of a first embodiment of a human-computer interaction device based on medium-free holographic technology provided by the present application;

[0021] Figure 4 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0022] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0025] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0026] Please refer to Figure 1 , Figure 1 A flowchart of a first embodiment of a human-computer interaction method based on medium-free holographic technology provided in this application.

[0027] like Figure 1As shown, the human-computer interaction method based on medium-free holographic technology includes steps S101 to S103.

[0028] S101, projecting a target object onto a projection display area based on a medium-free holographic technology to obtain a stereoscopic image of the target object;

[0029] In one embodiment, free-medium holographic technology is a technology that can reproduce image information in the air without a carrier or medium. Its core component is a microchannel matrix optical waveguide plate (Microchannel matrix optical waveguide plate, referred to as ), through the principle of optics, the optical micro-mirror structure is used to replicate the light field and reproduce a 100% three-dimensional stereoscopic real image in three-dimensional space. The optical micro-mirror structure records the intensity, angle, wavelength and other information of each light from the real light source in the "object space" to the imaging unit, and replicates the light mirrored with the recorded light in the "image space" on the other side of the array. These replicated light rays are refocused to form a real image mirrored with the object in the "object space" at a symmetrical position in the "image space". This real image does not rely on any medium and can be directly observed with the naked eye.

[0030] In one embodiment, the display medium of the projection display area may be air, and the three-dimensional structure of the target object is projected into the air for display through medium-free holographic technology, so that the user can view the stereoscopic image of the target object with naked eyes.

[0031] In one embodiment, based on a three-dimensional scanning method, three-dimensional scanning data of the target object is collected.

[0032] Wherein, the three-dimensional scanning data includes shape and size; based on the three-dimensional scanning data, the three-dimensional structure data corresponding to the target object is constructed.

[0033] In one embodiment, a suitable three-dimensional scanning technology is selected according to the size, shape complexity and accuracy requirements of the target object, such as laser scanning, structured light scanning or white light scanning.

[0034] In one embodiment, a 3D scanning device is used to scan a target object. The device emits light and captures reflected light to obtain point cloud data of the object surface. The object can be scanned from multiple angles during the scanning process to ensure that complete 3D data is obtained.

[0035] In one embodiment, the raw data obtained by scanning may contain noise and incomplete parts, and preprocessing operations such as filtering, denoising and completion are required. The point cloud model of the target object is generated by the preprocessed scan data. The point cloud is a collection of each point on the surface of the object, including the spatial coordinates of each point. The point cloud data is converted into a three-dimensional mesh model using three-dimensional modeling software. The size and shape of the three-dimensional mesh model are corrected as needed to ensure the accuracy of the model. The surface texture information of the object can also be applied to the three-dimensional mesh model to improve the realism of the model.

[0036] Furthermore, three-dimensional structure data and projection display data of the target object are acquired; based on the medium-free holographic technology, the three-dimensional structure data and the projection display data, the target object is projected into the projection display area to obtain the stereoscopic image.

[0037] In one embodiment, the three-dimensional structure data of the target object may be constructed by scanning the three-dimensional information of the actual object using a three-dimensional scanning technology, or may be three-dimensional model data created through a three-dimensional modeling network.

[0038] In one embodiment, the projection display data includes but is not limited to a scaling ratio, a projection position, and a projection display posture.

[0039] In one embodiment, a suitable scaling ratio is calculated according to the size of the projection display area and the actual size of the target object.

[0040] In one embodiment, the projection position can be determined based on the shape and size of the object and the coverage area of ​​the projection area to determine the center position or characteristic position of the object in the projection display area.

[0041] In one embodiment, the projection display posture includes the orientation, tilt angle, etc. of the target object in space.

[0042] In one embodiment, before the medium-free holographic technology performs holographic imaging, high-resolution three-dimensional scanning technology can be used to capture the three-dimensional information of the target object. Then, a laser and an interferometer are used to record the light wave information of the object, that is, by interfering the light wave of the object with the reference light wave to generate an interference pattern. The captured interference pattern is then converted into a hologram, which is a two-dimensional image containing the light wave information of the object. The hologram is illuminated with an appropriate light source (such as a laser, etc.) to reproduce the light wave of the object, and when the observer views it from different angles, a three-dimensional image of the target object can be seen. By projecting the reproduced light wave onto a specific display area, the user can see a three-dimensional medium-free holographic image, and the user can observe and interact with the holographic image from different angles without contacting any physical medium.

[0043] S102, based on the sensor device, collecting the interactive operation data of the target user and generating interactive operation instructions;

[0044] In one embodiment, the sensor device may include but is not limited to an optical sensor, a sound sensor, a motion tracking sensor, a depth sensor, and the like.

[0045] Exemplarily, optical sensors can be cameras for capturing gestures and movements of users, as well as optical tracking sensors. Sound sensors can be used to capture sound signals emitted by the target user. Motion tracking sensors can be used to track the movement trajectory of the user's body or hand, and may include accelerometers and gyroscopes. Depth sensors can be used to measure distances and create 3D spatial mapping, such as structured light sensors, time-of-flight (ToF) sensors, etc.

[0046] In one embodiment, the sensor device collects and recognizes the target user's gestures, voice, or interactive operation data such as actions made with the help of other operating devices (such as a control ring, a stylus, etc.), understands the target user's interactive intentions, and generates corresponding interactive operation instructions to facilitate the execution of corresponding interactive operations on the currently displayed stereoscopic image, thereby realizing contactless human-computer interaction.

[0047] For example, when the target user's gesture changes, the optical sensor detects the changes in light and shadow at different positions in the relevant area during the gesture change, such as the reflected / refracted light formed by the blocking of light by the gesture change, thereby obtaining a preliminary gesture change recognition result. Subsequently, the sound sensor detects whether the target user issues a voice control command or detects whether other operating devices (such as the aforementioned control ring and stylus) have changed position, and the gesture change recognition result is corrected according to the voice control command or the position change results of the other operating devices. Specifically, the position change results of other operating devices can be matched with the gesture change results, or the operation results indicated by the voice control command can be matched with the operation results corresponding to the gesture change recognition results, thereby achieving accurate judgment of the interactive operation. It can also be a combination of gesture change recognition results, voice control commands and other operations. A three-dimensional result set is constructed based on the position change results of the device, and then the three-dimensional result set is subjected to content analysis and matching through the operation instruction model to correct the operation instruction model, thereby obtaining operation instructions that match the actual needs of the target user; it is easy to understand that when there are more than three operating devices, the result set can be of a higher dimension, not limited to three dimensions; it should be noted that in the process of judging interactive operations, in addition to emphasizing the recognition of interactive operations by a single sensor device, the present application is also applicable to the mutual interaction between different sensor devices and the mutual correction of the interactive operation recognition results of the target user, that is, the operation interaction recognition of the present application can be realized by a single sensor device, and data can also be exchanged between different interactive devices to correct the recognition results of different interactive devices, thereby being applicable to complex interactive environments, thereby expanding the scope of application of the present application.

[0048] S103: Based on the interactive operation instruction, perform a corresponding interactive operation on the stereoscopic image to complete contactless human-computer interaction.

[0049] In one embodiment, the medium-free holographic technology projects a stereoscopic image of a target object into an air medium, and a user does not need to view and operate the stereoscopic image by contacting a physical structure, but can interact with the projected stereoscopic image through a sensor device.

[0050] It is understandable that the human-computer interaction system is indispensable for helping people in their lives, work, research, etc. However, in some scenarios, it is inconvenient for users to interact with the human-computer interaction system through direct contact. For example, when working underground, working in a high-temperature environment, or in a radiation area, it is usually necessary to wear heavy protective clothing, but when wearing thicker gloves, users cannot operate the human-computer interaction system with their fingers as usual, and contacting other devices is not sensitive enough, making the human-computer interaction inefficient and easy to identify wrong instructions. For another example, on smart operating devices that allow multiple people to use, people will leave fingerprints when using them, which will pose a risk of personal fingerprint information leakage, thereby endangering personal information and personal property safety. In addition, for scientific researchers and teaching assistants, especially those who are engaged in the research and teaching of designing complex mechanical objects, such as large ship structures, building structures, spacecraft structures, etc., during the research and teaching process, although the human-computer interaction system can display the three-dimensional structure diagrams of these mechanical structures, they are still displayed on a flat screen. Through the medium-free holographic technology, complex mechanical structures can be projected and displayed in the form of real-like models, making it easier for users to observe and understand the structural information of real objects more intuitively, thereby improving scientific research efficiency and teaching efficiency.

[0051] In one embodiment, the sensor device converts the collected interactive operation data into specific interactive operation instructions through signal processing and data analysis. The human-computer interaction system can adjust the position, angle, size, display content and other attributes of the stereoscopic image according to the generated interactive operation instructions.

[0052] In one embodiment, the interactive operation includes operations such as moving, rotating, scaling, and selecting.

[0053] For example, if the user performs a zooming action through a gesture, the human-computer interaction system can recognize the action and zoom in or out the stereoscopic image accordingly. If the user performs a moving operation through a gesture, the human-computer interaction system will move the stereoscopic image to a position specified by the user in the real area.

[0054] This embodiment provides a human-computer interaction method based on medium-free holographic technology. The method projects the target object into the projection display area through medium-free holographic technology to display the stereoscopic image of the target user, so that the user can interact with the three-dimensional image naturally and intuitively without physical contact, thereby improving the immersive experience. By collecting the interactive operation data of the target user through sensor equipment, the interactive actions of the target user can be accurately captured to achieve precise interactive operations. Through interactive operation instructions, corresponding interactive operations are performed on the stereoscopic image to achieve contactless interactive operation recognition, simplify user operations, reduce the difficulty of operation, avoid restrictions on user operations caused by other wearable devices and environmental factors, and improve the applicability and diversity of human-computer interactive operations.

[0055] Please refer to Figure 2 , Figure 2 This is a flow chart of the second embodiment of the human-computer interaction method based on medium-free holographic technology provided in this application.

[0056] In this embodiment, based on the above Figure 1 In the illustrated embodiment, S102 specifically includes:

[0057] S201, collecting interaction position information and interaction actions of characteristic parts of the target user based on the sensor device;

[0058] In one embodiment, the sensor device may be one or more sensors such as an optical sensor, a sound sensor, a motion tracking sensor, and a depth sensor.

[0059] In one embodiment, the characteristic parts may be key points of the hand such as fingertips and knuckles, or may be eye movement data, or may be operating devices such as a stylus pen and an operating ring.

[0060] Exemplarily, a gesture recognition sensor (a visual sensor such as a camera) can recognize the gestures of the target user, such as grabbing, pushing, rotating, etc. A multi-eye visual sensor can be used to collect the gesture interaction actions of the target user, and the landing position of the characteristic part of the target user can be identified through a depth sensor. The interaction position information can be obtained based on the landing position, and the interaction operation instructions can be identified based on the gesture interaction action. For example, if the stereoscopic image is the operating interface of a certain operating system, the multi-eye visual sensor recognizes that the gesture interaction action of the target user is selection (such as double-clicking, single-clicking, etc.), and the depth sensor recognizes that the target user's finger landing position is a certain function button, then it can be determined that the target user's interaction operation intention is to select a certain function item of the operating system.

[0061] For example, the motion tracking sensor may be a sensor such as an accelerometer and a gyroscope, or a visual sensor. Through the motion tracking sensor, the continuous movement of the target user's hand or body part can be tracked, fast or subtle movements can be captured, and the target user's hand movement trajectory can be identified.

[0062] For example, the sound sensor can capture the voice commands of the target user and convert them into specific operation commands through voice recognition technology.

[0063] S202, determining an interaction target based on the interaction location information;

[0064] In one embodiment, the interaction position information can be identified by sensor devices such as multi-eye vision sensors, depth sensors, motion tracking sensors, etc.

[0065] In one embodiment, when the human-computer interaction system projects a stereoscopic image of the target object, a three-dimensional coordinate system can be constructed according to the projection coverage area, and the interactive point position information in the stereoscopic image can be counted. When the sensor device collects the landing point position of the target user's characteristic part, the three-dimensional coordinate point of the landing point position in the projection coverage area is calculated according to the landing point position and the three-dimensional coordinate system, and then the interactive point position information is matched according to the three-dimensional coordinate point, so as to determine the interactive target corresponding to the matched point position information.

[0066] S203, determining an interactive operation based on the interactive action;

[0067] Further, based on the operation attributes of the target object, the preset interactive operation is configured; based on the operation preferences of the target user, the preset interactive actions corresponding to each preset interactive operation are configured; based on the correspondence between the preset interactive operation and the preset interactive action, the action instruction library is created.

[0068] In one embodiment, the corresponding action instruction library can be pre-set according to the operation attributes of the target object and the operation preferences of the target user. For example, if the projected stereoscopic image is the operation interface of a certain operating system, the operation instructions can be constructed according to the operation mode of the operating system, and the interactive actions corresponding to each operation instruction can be designed. For example, selection, scaling, movement, etc. If the projected stereoscopic image is a structural model of a real object, operation instructions such as scaling, disassembly, movement, selection, assembly, etc. can be set, and the interactive actions corresponding to each operation instruction can be designed, and the action instruction library can be created according to the correspondence between the operation instructions and the interactive actions.

[0069] Furthermore, based on a preset action instruction library, the feature similarity between the interactive action and the preset interactive action is calculated; when the feature similarity reaches a preset similarity threshold, the preset interactive operation corresponding to the preset interactive action is determined as the interactive operation corresponding to the interactive action.

[0070] In one embodiment, the captured interactive actions may be subjected to feature extraction, including gesture, speed, direction, amplitude, duration, movement trajectory, etc. Then, a similarity algorithm (such as a machine learning model, a similarity measurement algorithm, etc.) is used to calculate the feature similarity between the target user's interactive action and each preset interactive action in the preset action instruction library.

[0071] In one embodiment, the gesture action can be represented as a vector, and the cosine similarity is used to calculate the similarity between the vector corresponding to the interactive action and the vector corresponding to the preset interactive action. Alternatively, the Euclidean distance can be used to calculate the distance between the interactive action and the preset interactive action in the multi-dimensional feature space of the gesture action, and the similarity between the two can be represented by the distance. The similarity can also be calculated by comparing the motion trajectory of the interactive action with the motion trajectory of the preset interactive action.

[0072] Furthermore, based on the action instruction library, a first feature vector corresponding to each of the preset interactive actions is obtained; based on feature vectorization, the interactive action is vectorized to obtain a second feature vector; based on a similarity measurement algorithm, the similarity between the first feature vector and the second feature vector is calculated to obtain a feature similarity between the interactive action and the preset interactive action.

[0073] In one embodiment, feature vectorization is a calculation method for converting raw data into a numerical vector form that can be used for machine learning algorithms. By extracting feature data from interactive actions and preset interactive actions, such as the coordinates of key points, movement trajectories, etc., the feature data is scaled to a uniform range, such as [0,1], by normalization, the mean of the feature data is adjusted to 0 and the standard deviation is adjusted to 1, and then the classification features are encoded using a feature encoding method (such as One-Hot Encoding), thereby obtaining a first feature vector corresponding to the preset interactive action and a second feature vector corresponding to the interactive action; then, in order to ensure the accuracy of the feature vector, the first feature vector and the second feature vector can be constructed into a feature vector matrix according to a preset matrix construction rule, and then the correlation value of the feature vector matrix is ​​calculated according to a preset Pearson correlation coefficient. When the correlation value is within a preset threshold range, it is determined that the correlation between the first feature vector and the second feature vector meets the requirements, that is, the first feature vector and the second feature vector can be used to calculate the interactive action.

[0074] Exemplarily, gesture action data is collected by sensors, and features for identifying interactive actions are extracted from the gesture action data, such as the spatial coordinates of a feature point, motion features (such as speed, direction, trajectory) or timing features (characteristics of changes in gesture actions over time), etc. The extracted features are converted into numerical vectors. For example, the (x, y, z) coordinates of each key point can form a three-dimensional vector, and multiple key points can form a matrix or high-dimensional vector. Based on the correlation judgment, the similarity between the first feature vector and the second feature vector is calculated using a similarity measurement method.

[0075] For example, using the cosine similarity metric:

[0076]

[0077] Among them, p and q represent the first eigenvector and the second eigenvector respectively, p·q represents the dot product of two vectors, and ‖p‖‖q‖ represents the modulus (i.e., length) of the two vectors respectively. The cosine value between the angle between two vectors in a vector space is used as a measure of the difference between two individuals. When the cosine value is close to 1 and the angle tends to 0, the more similar the two vectors are. When the cosine value is close to 0 and the angle tends to 90 degrees, the more dissimilar the two vectors are.

[0078] In one embodiment, a similarity threshold may be set to determine whether the target user's interactive action is close enough to the preset interactive action to trigger the corresponding interactive operation instruction. When the feature similarity between the target user's interactive action and a preset interactive action is greater than or equal to the preset similarity threshold, the interactive action may be determined to be the preset interactive action.

[0079] In one embodiment, the human-computer interaction system can provide real-time feedback to let the target user know that his / her interaction action has been recognized and executed, and adjust the interaction operation according to the target user's further actions or feedback.

[0080] In one embodiment, the interaction habits and interaction patterns of the target user can be continuously learned through a machine learning algorithm to optimize the accuracy of action recognition and the setting of the similarity threshold.

[0081] In one embodiment, the target user may be allowed to customize or expand the action instruction library according to his or her own preferences and needs, so that the human-computer interaction system is more personalized and flexible.

[0082] Furthermore, based on the action instruction library, it is determined whether the interaction target and the interaction operation match; when the interaction target and the interaction operation do not match, an interaction rejection instruction is generated, and interaction rejection reminder information is generated according to the interaction rejection instruction.

[0083] In one embodiment, before generating an interactive operation instruction, the rationality and feasibility of the instruction can be verified to ensure the accuracy of the interaction. For example, a function button can be selected and moved, but cannot be scaled or rotated. If the interactive target is the function button, but the interactive operation is scaling or rotating, an interactive operation instruction cannot be generated; if the interactive operation is moving or selecting, a selection operation instruction or a move operation instruction for the function button can be generated.

[0084] In one embodiment, when the interaction target and the interaction operation do not match, it means that it is unreasonable to perform the interaction operation on the interaction target and cannot be performed. At this time, an interaction rejection instruction can be generated to reject the interaction operation and generate interaction rejection reminder information to remind the target user that the current interaction operation cannot be performed.

[0085] In one embodiment, when the combination of the interaction target and the interaction operation is verified to exist through the action instruction library, that is, the interaction target and the interaction operation match, the current interaction operation can be performed on the interaction target, and at this time, the corresponding interaction operation instruction can be generated.

[0086] S204: Generate the interactive operation instruction based on the interactive target and the interactive operation.

[0087] In one embodiment, by locating the interaction target and the interaction operation, it is possible to clearly identify the holographic image or virtual object that the target user wants to interact with, as well as the purpose that the target user wants to achieve through the interaction.

[0088] Exemplarily, the interactive operation instruction may be a specific operation instruction for a certain interactive target, such as moving, scaling, rotating, selecting, etc.

[0089] This embodiment can adapt to the action habits of different users, recognize various interactive actions through a preset action instruction library, and accurately recognize the user's intention by calculating the feature similarity between the interactive action and the preset action, thereby reducing erroneous operations caused by inaccurate action recognition.

[0090] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of the first embodiment of a human-computer interaction device based on medium-free holographic technology provided by the present application, and the human-computer interaction device based on medium-free holographic technology is used to execute the aforementioned human-computer interaction method based on medium-free holographic technology. The human-computer interaction device based on medium-free holographic technology can be configured in a server.

[0091] like Figure 3 As shown, the human-computer interaction device 300 based on the medium-free holographic technology includes: a holographic projection module 301, an interaction instruction generation module 302 and a human-computer interaction module 303.

[0092] The holographic projection module 301 is used to project the target object onto the projection display area based on the medium-free holographic technology to obtain a stereoscopic image of the target object;

[0093] An interactive instruction generating module 302 is used to collect interactive operation data of a target user based on a sensor device and generate interactive operation instructions;

[0094] The human-computer interaction module 303 is used to perform corresponding interactive operations on the stereoscopic image based on the interactive operation instruction to complete contactless human-computer interaction.

[0095] In one embodiment, the holographic projection module 301 includes:

[0096] A data acquisition unit, used to acquire three-dimensional structure data and projection display data of the target object;

[0097] A holographic projection unit is used to project the target object into the projection display area based on the medium-free holographic technology, the three-dimensional structure data and the projection display data to obtain the stereoscopic image.

[0098] In one embodiment, the human-computer interaction device 300 based on medium-free holographic technology further includes a three-dimensional structure data construction module, including:

[0099] A three-dimensional scanning unit, used for collecting three-dimensional scanning data of the target object based on a three-dimensional scanning method, wherein the three-dimensional scanning data includes shape and size;

[0100] The three-dimensional structure data construction unit is used to construct the three-dimensional structure data corresponding to the target object based on the three-dimensional scanning data.

[0101] In one embodiment, the projection display data includes a scaling ratio, a projection position, and a projection display posture.

[0102] In one embodiment, the interaction instruction generating module 302 includes:

[0103] An interaction information collection submodule, used to collect interaction position information and interaction actions of characteristic parts of the target user based on the sensor device;

[0104] An interaction target determination submodule, used to determine an interaction target based on the interaction position information;

[0105] An interactive operation determination submodule, used to determine an interactive operation based on the interactive action;

[0106] The instruction generation submodule is used to generate the interactive operation instruction based on the interactive target and the interactive operation.

[0107] In one embodiment, the interactive operation determination submodule includes:

[0108] A similarity calculation unit, used for calculating the feature similarity between the interactive action and a preset interactive action based on a preset action instruction library;

[0109] The interactive operation determining unit is used to determine, when the feature similarity reaches a preset similarity threshold, that the preset interactive operation corresponding to the preset interactive action is the interactive operation corresponding to the interactive action.

[0110] In one embodiment, the interaction instruction generating module 302 further includes:

[0111] A matching verification unit, used to determine whether the interaction target and the interaction operation match based on the action instruction library;

[0112] The interaction rejection unit is used to generate an interaction rejection instruction when the interaction target and the interaction operation do not match, and generate interaction rejection reminder information according to the interaction rejection instruction.

[0113] In one embodiment, the similarity calculation unit includes:

[0114] A first feature vector acquisition subunit, configured to acquire a first feature vector corresponding to each of the preset interactive actions based on the action instruction library;

[0115] A second feature vector acquisition subunit is used to vectorize the interaction action based on feature vectorization to obtain a second feature vector;

[0116] The similarity calculation subunit is used to calculate the similarity between the first feature vector and the second feature vector based on a similarity measurement algorithm, and obtain the feature similarity between the interaction action and the preset interaction action.

[0117] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned embodiment of the human-computer interaction method based on medium-free holographic technology, and will not be repeated here.

[0118] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 4 Runs on the computer device shown.

[0119] See also Figure 4 , Figure 4 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0120] See also Figure 4 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0121] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any human-computer interaction method based on the medium-free holographic technology.

[0122] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0123] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any human-computer interaction method based on the medium-free holographic technology.

[0124] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0126] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0127] Based on the medium-free holographic technology, the target object is projected into the projection display area to obtain a three-dimensional image of the target object;

[0128] Based on sensor devices, collect interactive operation data of target users and generate interactive operation instructions;

[0129] Based on the interactive operation instruction, a corresponding interactive operation is performed on the stereoscopic image to complete contactless human-computer interaction.

[0130] In one embodiment, when the processor implements the medium-free holographic technology to project the target object onto the projection display area to obtain a stereoscopic image of the target object, the processor is used to implement:

[0131] Acquiring three-dimensional structure data and projection display data of the target object;

[0132] Based on the medium-free holographic technology, the three-dimensional structure data and the projection display data, the target object is projected into the projection display area to obtain the stereoscopic image.

[0133] In one embodiment, before acquiring the three-dimensional structure data and projection display data of the target object, the processor is further configured to implement:

[0134] Based on a three-dimensional scanning method, collecting three-dimensional scanning data of the target object, wherein the three-dimensional scanning data includes shape and size;

[0135] The three-dimensional structure data corresponding to the target object is constructed based on the three-dimensional scanning data.

[0136] In one embodiment, the projection display data includes a scaling ratio, a projection position, and a projection display posture.

[0137] In one embodiment, when the processor collects the target user's interactive operation data based on the sensor device and generates the interactive operation instruction, it is used to implement:

[0138] Based on the sensor device, collecting the interaction position information and interaction actions of the characteristic parts of the target user;

[0139] Determining an interaction target based on the interaction location information;

[0140] Based on the interactive action, determining an interactive operation;

[0141] The interactive operation instruction is generated based on the interactive target and the interactive operation.

[0142] In one embodiment, when implementing the determining the interaction operation based on the interaction action, the processor is used to implement:

[0143] Based on a preset action instruction library, calculating the feature similarity between the interactive action and the preset interactive action;

[0144] When the feature similarity reaches a preset similarity threshold, a preset interaction operation corresponding to the preset interaction action is determined as the interaction operation corresponding to the interaction action.

[0145] In one embodiment, after implementing the determining of the interaction operation based on the interaction action, the processor is further configured to implement:

[0146] Based on the action instruction library, determining whether the interaction target and the interaction operation match;

[0147] When the interaction target and the interaction operation do not match, an interaction rejection instruction is generated, and interaction rejection reminder information is generated according to the interaction rejection instruction.

[0148] In one embodiment, when the processor implements the preset action instruction library based on the preset action instruction library and calculates the feature similarity between the interactive action and the preset interactive action, it is used to implement:

[0149] Based on the action instruction library, obtaining a first feature vector corresponding to each of the preset interactive actions;

[0150] Based on feature vectorization, the interactive action is vectorized to obtain a second feature vector;

[0151] Based on a similarity measurement algorithm, the similarity between the first feature vector and the second feature vector is calculated to obtain the feature similarity between the interaction action and the preset interaction action.

[0152] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the human-computer interaction methods based on medium-free holographic technology provided in the embodiments of the present application.

[0153] The computer-readable storage medium may be an internal storage unit of the computer device described in the above embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the computer device.

[0154] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A human-computer interaction method based on medium-free holographic technology, It is characterized in that The method comprises: Based on the medium-free holographic technology, the target object is projected into the projection display area to obtain a three-dimensional image of the target object; Based on sensor devices, collect interactive operation data of target users and generate interactive operation instructions; Based on the interactive operation instruction, a corresponding interactive operation is performed on the stereoscopic image to complete contactless human-computer interaction.

2. The human-computer interaction method based on medium-free holographic technology according to claim 1, It is characterized in that The method of projecting a target object onto a projection display area based on the medium-free holographic technology to obtain a stereoscopic image of the target object includes: Acquiring three-dimensional structure data and projection display data of the target object; Based on the medium-free holographic technology, the three-dimensional structure data and the projection display data, the target object is projected into the projection display area to obtain the stereoscopic image.

3. The human-computer interaction method based on medium-free holographic technology according to claim 2, It is characterized in that Before acquiring the three-dimensional structure data and projection display data of the target object, the method further includes: Based on a three-dimensional scanning method, collecting three-dimensional scanning data of the target object, wherein the three-dimensional scanning data includes shape and size; The three-dimensional structure data corresponding to the target object is constructed based on the three-dimensional scanning data.

4. The human-computer interaction method based on medium-free holographic technology according to claim 1, It is characterized in that The method of collecting interactive operation data of a target user based on a sensor device and generating interactive operation instructions includes: Based on the sensor device, collecting the interaction position information and interaction actions of the characteristic parts of the target user; Determining an interaction target based on the interaction location information; Based on the interactive action, determining an interactive operation; The interactive operation instruction is generated based on the interactive target and the interactive operation.

5. The human-computer interaction method based on medium-free holographic technology according to claim 4, It is characterized in that The determining the interactive operation based on the interactive action includes: Based on a preset action instruction library, calculating the feature similarity between the interactive action and the preset interactive action; When the feature similarity reaches a preset similarity threshold, a preset interaction operation corresponding to the preset interaction action is determined as the interaction operation corresponding to the interaction action.

6. The human-computer interaction method based on medium-free holographic technology according to claim 5, It is characterized in that After determining the interactive operation based on the interactive action, the method further includes: Based on the action instruction library, determining whether the interaction target and the interaction operation match; When the interaction target and the interaction operation do not match, an interaction rejection instruction is generated, and interaction rejection reminder information is generated according to the interaction rejection instruction.

7. The human-computer interaction method based on medium-free holographic technology according to claim 5, It is characterized in that Based on a preset action instruction library, calculating the feature similarity between the interactive action and the preset interactive action includes: Based on the action instruction library, obtaining a first feature vector corresponding to each of the preset interactive actions; Based on feature vectorization, the interactive action is vectorized to obtain a second feature vector; Based on a similarity measurement algorithm, the similarity between the first feature vector and the second feature vector is calculated to obtain the feature similarity between the interaction action and the preset interaction action.

8. A human-computer interaction device based on medium-free holographic technology, It is characterized in that The human-computer interaction device based on medium-free holographic technology includes: A holographic projection module is used to project a target object onto a projection display area based on a medium-free holographic technology to obtain a stereoscopic image of the target object; An interactive instruction generation module is used to collect interactive operation data of a target user based on a sensor device and generate interactive operation instructions; The human-computer interaction module is used to perform corresponding interactive operations on the stereoscopic image based on the interactive operation instruction to complete contactless human-computer interaction.

9. A computer device, It is characterized in that The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the human-computer interaction method based on the medium-free holographic technology as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the human-computer interaction method based on the medium-free holographic technology as described in any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Medium-free holographic imaging equipment and control method thereof

    CN120848142A

  • A medium-free holographic imaging device and a control method thereof

    CN120848142B

  • Mistaken touch prevention interaction method based on medium-free holography

    CN120994073A

  • Multi-modal man-machine verification processing method and device and electronic equipment

    CN121012635A