Interaction method of electronic equipment and related device

Abstract

The invention discloses an interaction method and device of electronic equipment, the electronic equipment and a storage medium. The method comprises the following steps: acquiring a real scene image, and identifying a plurality of target elements in the real scene image; matching a virtual element corresponding to the target element, and superposing the target element and the virtual element to form a virtual-real superposed image; projecting the virtual-real superposition image to a target element to perform laser interference on the virtual-real superposition image, and generating and displaying a three-dimensional projection interface; in the embodiment of the invention, multiple behavior characteristics of the user responding to the three-dimensional projection interface are captured, and the three-dimensional projection interface is adjusted in combination with the multiple behavior characteristics, so that the dependence on a physical screen can be reduced, and the interaction flexibility and convenience are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an electronic device interaction method, device, electronic device and storage medium. Background Art

[0002] The advancement of science and technology has promoted fundamental changes in human lifestyles. The rapid development of information technology has brought about a surge in the amount of data, and people's demand for obtaining and processing information is growing.

[0003] Although touch screen technology has brought an intuitive operation mode to the user interface of electronic devices, with the expansion of application scope and the improvement of technical requirements, this interaction mode has gradually revealed its inherent limitations: the large-size screen increases the operation range, making one-handed operation more difficult, and users have to use both hands frequently, reducing convenience; at the same time, the large-size screen also limits flexibility in different scenarios, especially during mobility, where complicated operations bring inconvenience. Summary of the invention

[0004] The main technical problem solved by the present application is to provide an electronic device interaction method, device, electronic device and storage medium, which can solve the technical problem that touch screen technology limits the portability and flexibility of electronic devices.

[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: an interaction method of an electronic device, the interaction method comprising:

[0006] Acquire a real scene image, and identify multiple target elements in the real scene image;

[0007] Matching a virtual element corresponding to the target element, and superimposing the target element with the virtual element to form a virtual-real superimposed image;

[0008] Projecting the virtual-real superimposed image onto the target element to perform laser interference on the virtual-real superimposed image to generate and display a three-dimensional projection interface;

[0009] Capturing multiple behavioral features of a user in response to the three-dimensional projection interface, and adjusting the three-dimensional projection interface in combination with the multiple behavioral features.

[0010] Optionally, capturing a plurality of behavioral features of a user in response to the three-dimensional projection interface, and adjusting the three-dimensional projection interface in combination with the plurality of behavioral features, includes:

[0011] Performing key point detection on the user's actions and postures to obtain corresponding action features; and / or

[0012] Tracking the pupil movement of the user to determine the gaze focus of the user using a gaze point estimation algorithm to obtain corresponding eye movement features; and / or

[0013] Collecting the user's voice information and converting the voice information into voice features using natural language processing and voice recognition technology; and / or

[0014] Collecting brain wave signals of the user, and performing feature extraction on the brain wave signals to obtain brain wave features; and / or

[0015] Reading the emotional state and psychological needs of the user, and generating corresponding emotional features according to the emotional state and the psychological needs;

[0016] The three-dimensional projection interface is adjusted based on at least one of the action characteristics, eye movement characteristics, voice characteristics, brain wave characteristics, and emotion characteristics.

[0017] Optionally, performing laser interference on the virtual-real superimposed image to generate and display a three-dimensional projection interface includes:

[0018] Combining spatial positioning, inertial navigation and map matching technology to obtain the user's position and perspective;

[0019] Performing laser interference on the virtual and real superimposed image, and controlling interference parameters of the laser interference using phase modulation technology to generate a three-dimensional image;

[0020] Using light field coding technology, the user's position and viewing angle are calculated to adjust the projection angle and focal length of the three-dimensional image;

[0021] The preset photonic nanostructure is used to control the optical parameters of the three-dimensional image to generate the three-dimensional projection interface.

[0022] Optionally, matching a virtual element corresponding to the target element and superimposing the target element with the virtual element to form a virtual-real superimposed image includes:

[0023] Extracting element features of the target element to match the element features with elements in a preset virtual element library to determine the virtual element;

[0024] According to the target element, the virtual element is subjected to posture registration and geometric correction, so that the position and proportion of the virtual element are consistent with those of the target element, thereby obtaining the virtual-real superimposed image.

[0025] Optionally, before projecting the virtual-real superimposed image onto the target element to perform laser interference on the virtual-real superimposed image to generate and display a three-dimensional projection interface, the method includes:

[0026] Obtaining the identity characteristics of the user by performing iris scanning and facial recognition on the user;

[0027] Using blockchain encryption and zero-knowledge proof technology to verify the identity characteristics;

[0028] When the identity feature verification is passed, the virtual-real superimposed image is projected onto the target element to perform laser interference on the virtual-real superimposed image to generate and display a three-dimensional projection interface.

[0029] Optionally, the identifying a plurality of target elements in the real scene image includes:

[0030] Using semantic segmentation technology to divide different objects in the real scene image;

[0031] Element detection and scene understanding are performed on the divided real scene image to determine the target element.

[0032] Optionally, the interaction method further includes:

[0033] Get current task load information;

[0034] The dynamic voltage and frequency adjustment technology is used to adjust the operating frequency and voltage according to the current task load information.

[0035] In order to solve the above technical problems, another technical solution adopted by the present application is: an interactive device of an electronic device, the interactive device of the electronic device comprising:

[0036] A recognition module is used to acquire a real scene image and recognize multiple target elements in the real scene image;

[0037] A superposition module, used for matching a virtual element corresponding to the target element, and superimposing the target element with the virtual element to form a virtual-real superimposed image;

[0038] A projection module, used for projecting the virtual-real superimposed image onto the target element, so as to perform laser interference on the virtual-real superimposed image, and generate and display a three-dimensional projection interface;

[0039] The display module is used to capture multiple behavioral features of the user responding to the three-dimensional projection interface, and adjust the three-dimensional projection interface in combination with the multiple behavioral features.

[0040] In order to solve the above technical problems, another technical solution adopted by the present application is: an electronic device, the electronic device comprising:

[0041] A memory for storing executable program codes;

[0042] The processor is used to call and run the executable program code from the memory, so that the electronic device executes any one of the interaction methods described above.

[0043] To solve the above technical problems, another technical solution adopted in the present application is: a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the interaction method as described in any one of the above.

[0044] Different from the prior art, the embodiment of the present application provides an interaction method of an electronic device, and the interaction method includes:

[0045] Acquire a real scene image, identify multiple target elements in the real scene image; match the virtual elements corresponding to the target elements, and superimpose the target elements with the virtual elements to form a virtual-real superimposed image; project the virtual-real superimposed image onto the target element to perform laser interference on the virtual-real superimposed image, generate and display a three-dimensional projection interface; capture multiple behavioral characteristics of the user responding to the three-dimensional projection interface, and adjust the three-dimensional projection interface in combination with the multiple behavioral characteristics. By acquiring a real scene image, identifying and matching the target elements and virtual elements therein, generating a virtual-real superimposed image; then projecting the image into the target element, and using laser interference technology to generate and display a three-dimensional projection interface, the reliance on the physical screen can be reduced and the flexibility of interaction can be increased; and capturing multiple behavioral characteristics of the user responding to the three-dimensional projection interface, adjusting the projection interface in real time according to these behavioral characteristics can improve the convenience of interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of the first embodiment of the interaction method of the electronic device of the present application;

[0047] Figure 2 It is a detailed flow chart of step S140 of the interaction method of the electronic device of the present application;

[0048] Figure 3 It is a detailed flow chart of step S130 of the interaction method of the electronic device of the present application;

[0049] Figure 4 It is a detailed flow chart of step S120 of the interaction method of the electronic device of the present application;

[0050] Figure 5 It is a flowchart of a second embodiment of the interaction method of the electronic device of the present application;

[0051] Figure 6 is an exemplary structural block diagram of an electronic device of the electronic device interaction method of the present application;

[0052] Figure 7 It is an exemplary structural block diagram of a computer-readable storage medium of the interactive method of the electronic device of the present application. DETAILED DESCRIPTION

[0053] The present application is described in detail below with reference to the accompanying drawings and implementation methods.

[0054] See also Figure 1 , Figure 1 1 is a flowchart of the first embodiment of the interactive method of the electronic device of the present application. It should be noted that if there is substantially the same result, the method of the present application is not limited to Figure 1 The process sequence shown is limited. Figure 1 As shown, the interaction method includes:

[0055] Step S110, acquiring a real scene image, and identifying multiple target elements in the real scene image;

[0056] The execution subject of the method of this embodiment can be an interactive device of an electronic device, or it can be a holographic projection terminal device or server. This embodiment takes the interactive device of an electronic device as an example. The interactive device of the electronic device can be integrated in a terminal device such as a smart phone, a tablet computer, etc. with a data processing function.

[0057] The real scene image can be an image or video of the real environment captured by a camera or other sensor. For example, it contains objects, buildings, people, etc. around the user, which serve as the basic data that needs to be integrated with virtual content in augmented reality applications. The target element can be a specific object or scene identified in the real scene image.

[0058] In an exemplary embodiment, images or videos of the user's surroundings can be captured in real time by the camera of the electronic device; then, object detection, edge detection, image segmentation and other algorithms, such as YOLO, Faster R-CNN, etc., can be used to locate and identify multiple target elements in the real scene image.

[0059] Step S120, matching a virtual element corresponding to the target element, and superimposing the target element with the virtual element to form a virtual-real superimposed image;

[0060] Among them, the virtual elements can be virtual objects, virtual images or information pre-stored or generated by the system, such as 3D models, text descriptions, data visualizations, etc., which are used to pair with target elements in reality and superimposed on the target elements in the real scene image to form an augmented reality effect.

[0061] In an exemplary embodiment, after identifying the target element in the real scene image, the system can match the virtual element corresponding to the target element from the preset virtual element library. The system can align and superimpose the virtual element with the target element in the real world in terms of position, scale, and angle through depth perception technology and posture estimation algorithm to ensure that the virtual element is consistent with the target element, forming a superposition effect with a strong sense of reality.

[0062] Step S130, projecting the virtual-real superimposed image onto the target element to perform laser interference on the virtual-real superimposed image to generate and display a three-dimensional projection interface;

[0063] Among them, the virtual-reality superimposed image can be an image generated by combining and superimposing the target elements in reality and the virtual elements. It can also contain the content of the real world and the virtually generated elements, forming an augmented reality or mixed reality visual effect.

[0064] In an exemplary embodiment, the system projects a virtual and real superimposed image into the target element, and then interferes through the coherence of the laser beam to modulate the virtual element with the image of the real object; because the interference pattern records three-dimensional information, when light passes through these interference patterns, a three-dimensional holographic projection with a sense of depth is formed; then, by adjusting the wavelength, intensity and phase of the laser beam, the holographic projection can generate a realistic three-dimensional stereoscopic image on the target element, so that the user can see the effect of the three-dimensional projection interface from multiple angles or one angle.

[0065] Step S140 , capturing a plurality of behavioral features of the user in response to the three-dimensional projection interface, and adjusting the three-dimensional projection interface in combination with the plurality of behavioral features.

[0066] The behavioral features may be various actions, gestures or other behaviors performed by the user when interacting with the electronic device, which are used to adjust the three-dimensional projection interface, such as finger sliding, finger clicking, grasping action, eye gaze and focus movement, head posture or body movement, etc.

[0067] In an exemplary implementation, sensors can be used to monitor the user's behavior, and the system can obtain the user's current needs and intentions in real time by analyzing the captured multiple behavioral characteristics. In combination with the user's multiple behavioral characteristics, the system will adjust the 3D projection interface. For example, when the user enlarges the virtual object or adjusts the displayed content through gestures, the system will respond and adjust the holographic projection in real time to keep it synchronized with the user's operation.

[0068] This embodiment acquires a real-world scene image, identifies and matches the target elements and virtual elements therein, and generates a virtual-real superimposed image; then projects the image into the target element, and uses laser interference technology to generate and display a three-dimensional projection interface, which can reduce dependence on physical screens and increase the flexibility of interaction; and captures multiple behavioral characteristics of users responding to the three-dimensional projection interface, and adjusts the projection interface in real time according to these behavioral characteristics, which can improve the convenience of interaction.

[0069] See also Figure 2 , Figure 2 1 is a detailed flow chart of step S140 of the electronic device interaction method of the present application. Figure 2 As shown, step S140 may include:

[0070] Step S210, performing key point detection on the user's actions and postures to obtain corresponding action features;

[0071] In this embodiment, any of steps S210 to S250 may be executed simultaneously, in a certain order, or not executed. This embodiment takes the sequential execution of steps S210 to S250 as an example.

[0072] Specifically, the system can use a posture estimation algorithm (such as OpenPose) to detect the user's posture, and use a skeleton key point detection algorithm to detect the joints of the user's body parts to determine the user's gestures (such as grasping, waving, etc.) or actions (such as standing, walking, etc.) to perform interactive operations. For example, the user can operate the content on the 3D projection interface by waving his hand.

[0073] Step S220, tracking the pupil movement of the user to determine the gaze focus of the user using a gaze point estimation algorithm to obtain corresponding eye movement features;

[0074] Specifically, the user's pupil movement and eyeball change trajectory can be captured through eye tracking devices, such as infrared light sources and high frame rate cameras. The gaze point estimation algorithm determines the user's gaze focus by calculating the relative position of the pupil and the screen or projection interface. The gaze point estimation algorithm can be a 2D / 3D pupil positioning and line of sight tracking algorithm. This allows the system to track the user's gaze point and determine the area of ​​interest by the time the user's gaze stays, and automatically adjust or display related content. For example, the user only needs to look at a virtual button to activate its function.

[0075] Step S230, collecting the user's voice information, and using natural language processing and speech recognition technology to convert the voice information into voice features;

[0076] Specifically, the microphone of the electronic device can be used to collect the user's voice signal, and the voice signal can be converted into text or commands by combining voice recognition technology and natural language processing technology. Among them, voice features can include tone, speech speed, semantic information, etc., which are used to judge the user's intention. So that the user can directly interact with the system through voice commands, such as starting an application in the projection interface through voice, or adjusting the display content of the interface.

[0077] Step S240, collecting brain wave signals of the user, and performing feature extraction on the brain wave signals to obtain brain wave features;

[0078] Specifically, the brain wave signals of the user are collected through the brain-computer interface sensor of the electronic device, and the collected brain waves are feature extracted using a signal processing algorithm to analyze the user's brain electrical activity. Among them, different brain waves (such as alpha waves, beta waves, etc.) correspond to different psychological states or action intentions. The signal processing algorithm can be a frequency domain analysis, time domain analysis and other techniques. So that users can control the three-dimensional projection interface through thinking, such as selecting a virtual button on the interface by concentrating their attention, or adjusting the displayed content by relaxing their brains.

[0079] Step S250, reading the emotional state and psychological needs of the user, and generating corresponding emotional features according to the emotional state and the psychological needs;

[0080] Specifically, the system can use emotional computing technology to capture physiological signals such as facial expressions, voice intonation or skin galvanic response through cameras, and combine it with emotion recognition algorithms to determine the user's current emotional state, such as happiness, anxiety, fatigue, etc. At the same time, it can analyze the user's long-term operating habits, such as click behavior and voice analysis, to infer the user's psychological needs. The system can automatically adjust the style and atmosphere of the projection interface according to the user's emotional state. For example, when it is detected that the user is in a nervous state, the interface may become simple and soft to reduce visual burden and anxiety.

[0081] Step S260, adjusting the three-dimensional projection interface in combination with at least one of the action characteristics, eye movement characteristics, voice characteristics, brain wave characteristics, and emotion characteristics.

[0082] Specifically, by combining one or more of the extracted action features, eye movement features, voice features, brain wave features, and emotional features, the system can dynamically adjust the display content, interaction methods, and visual effects of the 3D projection interface. Through multimodal interaction, the user experience is more intelligent and personalized.

[0083] For example, in a 3D projection interface, a user can gaze at a certain area and use gestures to manipulate virtual objects in that area, or directly instruct operations through voice. Combined with the system's judgment of the user's emotions, the system can automatically optimize the interface's interactive mode and enhance the flexibility of electronic devices. For example, while driving, the driver can view navigation information through eye tracking without having to distract himself from operating the phone.

[0084] In this embodiment, by combining multiple features of the user, such as movements, eye movements, voice, brain waves, and emotions, the system can more accurately understand the user's intentions, achieve multiple natural and efficient interaction methods, and enhance the convenience and flexibility of electronic device interaction.

[0085] See also Figure 3 , Figure 3 Detailed flow chart of step S130 of the electronic device interaction method of the present application. Figure 3 As shown, step S130 may include:

[0086] Step S310, combining spatial positioning, inertial navigation and map matching technology to obtain the user's position and viewing angle;

[0087] Specifically, spatial positioning can be determined by at least one of a satellite positioning system (such as GPS), Wi-Fi signals, and Bluetooth beacons to determine the user's specific position in three-dimensional space. Inertial navigation technology can use sensors such as accelerometers and gyroscopes to detect the user's movement and posture changes, combined with the user's current location information, to predict the user's movement path and perspective changes. Map matching technology can match and calibrate the user's current location information with the map in the environment to ensure the accuracy of the position, and calculate the user's perspective by analyzing the relative position of the user and the environment. This can be used for precise positioning and image projection adjustment in augmented reality scenes.

[0088] Step S320, performing laser interference on the virtual and real superimposed image, and controlling interference parameters of the laser interference using phase modulation technology to generate a three-dimensional image;

[0089] Specifically, laser interference can be achieved by the interaction of two or more coherent laser beams to generate interference patterns, thereby realizing the superposition of virtual elements and target elements. Among them, the interference fringes of laser interference can be used to enhance the depth and stereoscopic sense of three-dimensional images. At the same time, in order to improve the brightness, contrast and resolution of three-dimensional images, the phase difference and amplitude of interference can be precisely adjusted by controlling the phase of the laser beam.

[0090] Step S330, using light field coding technology to calculate the user's position and viewing angle to adjust the projection angle and focal length of the three-dimensional image;

[0091] Specifically, by recording the intensity, direction, and light field information of the light, and then using light field coding technology, multi-view images can be generated. Combined with the user's position information and perspective, the system can dynamically adjust the projection angle and focal length of the image. Then, according to the user's position and perspective changes, the light direction and focal length of the three-dimensional image are dynamically controlled, thereby improving the display effect of the virtual and real superimposed images.

[0092] In addition, by utilizing directional display characteristics and light field coding technology, the displayed three-dimensional projection interface can only be seen when the user's eyes are within a specific angle and distance range.

[0093] Step S340: Using a preset photonic nanostructure to control the optical parameters of the three-dimensional image to generate the three-dimensional projection interface.

[0094] Specifically, the photonic nanostructure can be a photonic crystal material and a nano-optical structure, which is pre-set in a projection device of an electronic device to control the reflection, refraction and scattering of light of a three-dimensional image, thereby affecting the optical properties of the three-dimensional image, such as brightness, color and contrast.

[0095] This embodiment uses an ultra-high-resolution projection device based on the principle of laser interference, combined with phase modulation technology, which can construct colorful, detailed, and extremely realistic three-dimensional images and text in the air. By using photonic crystal materials and nano-optical structures, a high-brightness, high-contrast projection effect is achieved. At the same time, the module introduces spatial positioning technology, which can adjust the projection angle and focal length in real time according to the user's position and viewing angle, ensuring that no matter what angle and distance the user is at, a clear, distortion-free and highly immersive visual experience can be obtained. For example, when a photographer is shooting outdoors, he does not need to take out his mobile phone to operate the screen. He can quickly flip through and display previously taken photos with simple gestures, which improves the convenience of operation.

[0096] See also Figure 4 , Figure 4 Detailed flow chart of step S120 of the electronic device interaction method of the present application. Figure 4 As shown, step S120 may include:

[0097] Step S410, extracting element features of the target element to match the element features with elements in a preset virtual element library to determine the virtual element;

[0098] Specifically, key feature information can be extracted from target elements (such as objects, signs, people, etc.) in real scenes. Element features can be the edge, color, shape, texture, geometric structure, etc. of an object. Feature extraction is used to locate and extract stable features of targets from real scene images, such as edge detection, scale-invariant feature transformation, accelerated robust features, etc.

[0099] The system can then compare the features of the extracted target element with the elements in the virtual element library. The virtual element library can contain preset 3D virtual objects or models whose feature information has been stored in the database. The comparison process uses a feature matching algorithm to compare the features of the real object with the features in the virtual library to find the virtual element that is most similar to the target element.

[0100] For example, in a scenario where virtual and real objects are combined, such as when a user scans a real landmark building, the system can find the corresponding three-dimensional building model in the virtual library and match it through feature matching.

[0101] Step S420: performing posture registration and geometric correction on the virtual element according to the target element, so that the position and proportion of the virtual element are consistent with those of the target element, thereby obtaining the virtual-real superimposed image.

[0102] Specifically, posture registration can be the process of spatially aligning a virtual element with a target element in a real scene. Pose registration includes calculating the geometric transformation parameters of the virtual element, such as rotation, translation, and scaling, to ensure that the posture of the virtual element in three-dimensional space matches the angle and position of the real target element.

[0103] After posture alignment, the system can perform geometric correction on the virtual elements by adjusting the size and shape of the virtual elements to match the actual size of the real target elements, ensuring that the virtual elements match the target elements in proportion.

[0104] Through posture registration and geometric correction of virtual elements, the virtual elements can be superimposed on the real scene, and the display effect of the virtual elements can be adjusted according to the user's perspective, light and other environmental parameters to ensure that they are naturally integrated with the real scene.

[0105] In this embodiment, by extracting the feature information of the target element and matching it with the virtual object in the virtual element library, and then ensuring the consistency of the size and position of the virtual element with the real element through posture alignment and geometric correction, the system generates a virtual-reality superimposed image, allowing the user to seamlessly see the fusion effect of the virtual object and the real environment in the real scene.

[0106] See also Figure 5 , Figure 5 1 is a flow chart of the second embodiment of the interaction method of the electronic device of the present application. Figure 5 As shown, the interaction method includes:

[0107] Step S510, obtaining a real scene image, and using semantic segmentation technology to divide different objects in the real scene image;

[0108] Specifically, by classifying each target pixel in the image, the object boundaries in the image can be accurately identified to separate different objects (such as people, buildings, vehicles, etc.) from the background. For example, in an augmented reality scene, semantic segmentation technology can divide images of city streets into pedestrians, buildings, trees, vehicles, etc., which is convenient for subsequent processing.

[0109] Step S520, performing element detection and scene understanding on the divided real scene image to determine the target element.

[0110] Specifically, after semantic segmentation, target detection techniques such as Faster R-CNN, YOLO (You Only Look Once) or SSD (Single Shot MultiBox Detector) can be used to locate and identify each object in the image to determine the target element. At the same time, deep learning models, such as architectures based on convolutional neural networks (CNN) and recurrent neural networks (RNN), can be used to combine the overall information of the image to infer the context of the current environment. Therefore, it is necessary not only to identify objects in the image, but also to understand the relationship between objects and how objects interact with the environment.

[0111] Step S530, matching a virtual element corresponding to the target element, and superimposing the target element with the virtual element to form a virtual-real superimposed image;

[0112] The above step S120 may be performed and will not be described in detail here.

[0113] Step S540, obtaining the identity characteristics of the user by performing iris scanning and facial recognition on the user;

[0114] Specifically, iris scanning can be to use the camera of the electronic device to collect the texture features of the user's iris; at the same time, combined with the detection of facial geometric features (such as eyes, nose, mouth, etc.) to verify the identity of the user.

[0115] Step S550, using blockchain encryption and zero-knowledge proof technology to verify the identity characteristics;

[0116] Specifically, blockchain encryption can ensure the security of user identity features and prevent tampering or unauthorized access; zero-knowledge proof allows user identity to be verified without revealing specific information. For example, a user can prove that his or her identity has been verified without revealing specific biometric data, ensuring that the user's sensitive identity information is secure during transmission and storage, preventing information leakage.

[0117] Step S560, when the identity feature verification is passed, executing the projecting of the virtual-real superimposed image onto the target element to perform laser interference on the virtual-real superimposed image to generate and display a three-dimensional projection interface.

[0118] Specifically, the system will activate and display the 3D projection interface only after the user passes the identity authentication, which can ensure that authorized users can see the projection content, improving security and privacy protection. In addition, it also adopts an attribute-based access control model and a fine-grained permission management mechanism, so that authorized users can access and operate relevant information under specific conditions, effectively protecting the user's personal privacy and preventing information from being obtained and snooped by unauthorized third parties.

[0119] Step S570 , capturing a plurality of behavioral features of the user in response to the three-dimensional projection interface, and adjusting the three-dimensional projection interface in combination with the plurality of behavioral features.

[0120] The above step S140 may be performed and will not be described in detail here.

[0121] Furthermore, for any of the above steps, the electronic device may also obtain current task load information; and use dynamic voltage and frequency adjustment technology to adjust the operating frequency and voltage according to the current task load information.

[0122] In an exemplary embodiment, the system uses dynamic voltage and frequency adjustment technology and power gating technology to optimize the hardware's operating voltage and frequency in real time according to the current task load and performance requirements to reduce power consumption. The use of new quantum dot luminescent materials and low-power drive circuits improves projection efficiency and reduces energy loss. In addition, the system also has an intelligent ambient light sensing module that can automatically adjust the brightness and contrast of the projection according to the intensity of the surrounding light, further reducing unnecessary energy consumption.

[0123] In this embodiment, the user's biometrics are verified to ensure the security, accuracy and efficient human-computer interaction experience of the system. At the same time, through the comprehensive application of energy-saving technologies, the battery life of electronic devices is improved while ensuring the user experience.

[0124] See also Figure 6 , Figure 6 is an exemplary structural block diagram of an electronic device of the electronic device interaction method of the present application. Figure 6As shown, the electronic device 600 of the present application may include a processor 601 and a memory 602, wherein the processor 601 and the memory 602 communicate with each other via a bus. The memory 602 stores program instructions for holographic projection, and when the program instructions are executed by the processor 601, the processor executes the above-mentioned related method steps to implement an interaction method of an electronic device in the above-mentioned embodiment.

[0125] See also Figure 7 , Figure 7 is an exemplary structural block diagram of a computer-readable storage medium of the electronic device interaction method of the present application. Figure 7 As shown, the computer-readable storage medium 700 stores a computer program 701. When the computer program 701 is executed by a processor on a computer, the computer executes the above-mentioned related method steps to implement an interaction method of an electronic device in the above-mentioned embodiment.

[0126] The above scheme acquires a real scene image, identifies multiple target elements in the real scene image; matches the virtual elements corresponding to the target elements, and superimposes the target elements with the virtual elements to form a virtual-real superimposed image; projects the virtual-real superimposed image onto the target element to perform laser interference on the virtual-real superimposed image, generates and displays a three-dimensional projection interface; captures multiple behavioral characteristics of the user in response to the three-dimensional projection interface, and adjusts the three-dimensional projection interface in combination with the multiple behavioral characteristics. By acquiring a real scene image, identifying and matching the target elements and virtual elements therein, and generating a virtual-real superimposed image; then projecting the image into the target element, and using laser interference technology to generate and display a three-dimensional projection interface, it is possible to reduce dependence on physical screens and increase the flexibility of interaction; and by capturing multiple behavioral characteristics of the user in response to the three-dimensional projection interface, adjusting the projection interface in real time according to these behavioral characteristics, it is possible to improve the convenience of interaction.

[0127] In the several embodiments provided in the present application, it should be understood that the disclosed methods, electronic devices and storage media can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the interactive method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0131] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An interactive method for an electronic device, characterized in that: The interaction method includes: Acquire a real scene image, and identify multiple target elements in the real scene image; Matching a virtual element corresponding to the target element, and superimposing the target element with the virtual element to form a virtual-real superimposed image; Projecting the virtual-real superimposed image onto the target element to perform laser interference on the virtual-real superimposed image to generate and display a three-dimensional projection interface; Capturing multiple behavioral features of a user in response to the three-dimensional projection interface, and adjusting the three-dimensional projection interface in combination with the multiple behavioral features.

2. The interactive method according to claim 1, characterized in that: The capturing of multiple behavioral features of the user in response to the three-dimensional projection interface and adjusting the three-dimensional projection interface in combination with the multiple behavioral features includes: Performing key point detection on the user's actions and postures to obtain corresponding action features; and / or Tracking the pupil movement of the user to determine the gaze focus of the user using a gaze point estimation algorithm to obtain corresponding eye movement features; and / or Collecting the user's voice information and converting the voice information into voice features using natural language processing and voice recognition technology; and / or Collecting brain wave signals of the user, and performing feature extraction on the brain wave signals to obtain brain wave features; and / or Reading the emotional state and psychological needs of the user, and generating corresponding emotional features according to the emotional state and the psychological needs; The three-dimensional projection interface is adjusted based on at least one of the action characteristics, eye movement characteristics, voice characteristics, brain wave characteristics, and emotion characteristics.

3. The interactive method according to claim 1, characterized in that: The performing laser interference on the virtual-real superimposed image to generate and display a three-dimensional projection interface includes: Combining spatial positioning, inertial navigation and map matching technology to obtain the user's position and perspective; Performing laser interference on the virtual and real superimposed image, and controlling interference parameters of the laser interference using phase modulation technology to generate a three-dimensional image; Using light field coding technology, the user's position and viewing angle are calculated to adjust the projection angle and focal length of the three-dimensional image; The preset photonic nanostructure is used to control the optical parameters of the three-dimensional image to generate the three-dimensional projection interface.

4. The interactive method according to claim 1, characterized in that: The matching of the virtual element corresponding to the target element and superimposing the target element with the virtual element to form a virtual-real superimposed image includes: Extracting element features of the target element to match the element features with elements in a preset virtual element library to determine the virtual element; According to the target element, the virtual element is subjected to posture registration and geometric correction, so that the position and proportion of the virtual element are consistent with those of the target element, thereby obtaining the virtual-real superimposed image.

5. The interactive method according to claim 1, characterized in that: Before projecting the virtual-real superimposed image onto the target element to perform laser interference on the virtual-real superimposed image to generate and display a three-dimensional projection interface, the method includes: Obtaining the identity characteristics of the user by performing iris scanning and facial recognition on the user; Using blockchain encryption and zero-knowledge proof technology to verify the identity characteristics; When the identity feature verification is passed, the virtual-real superimposed image is projected onto the target element to perform laser interference on the virtual-real superimposed image to generate and display a three-dimensional projection interface.

6. The interactive method according to claim 1, characterized in that: The identifying multiple target elements in the real scene image includes: Using semantic segmentation technology to divide different objects in the real scene image; Element detection and scene understanding are performed on the divided real scene image to determine the target element.

7. The interactive method according to any one of claims 1 to 6, characterized in that: The interaction method further includes: Get current task load information; The dynamic voltage and frequency adjustment technology is used to adjust the operating frequency and voltage according to the current task load information.

8. An interactive device for an electronic device, characterized in that: The interaction device of the electronic device comprises: A recognition module is used to acquire a real scene image and recognize multiple target elements in the real scene image; A superposition module, used for matching a virtual element corresponding to the target element, and superimposing the target element with the virtual element to form a virtual-real superimposed image; A projection module, used for projecting the virtual-real superimposed image onto the target element, so as to perform laser interference on the virtual-real superimposed image, and generate and display a three-dimensional projection interface; The display module is used to capture multiple behavioral features of the user responding to the three-dimensional projection interface, and adjust the three-dimensional projection interface in combination with the multiple behavioral features.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the interaction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the interaction method according to any one of claims 1 to 7 is implemented.

Images