Medium-free holographic imaging device, and interactive method, system and program product thereof

Abstract

This disclosure provides a medium-free holographic imaging device and its interaction method, system, and program product. The medium-free holographic imaging device can establish a signal connection with a target display device. The interaction method includes: generating a suspended medium-free holographic image in a preset imaging area based on multimedia information from the target display device; the medium-free holographic image including one or more user-operable interactive areas; acquiring user operation information on the interactive areas; and sending the operation information to the target display device. Using this interaction method, interactive areas are displayed on the medium-free holographic image, allowing users to operate on the interactive areas. This liberates the interactive interface from a two-dimensional plane to a three-dimensional space, expanding the screen while significantly reducing cognitive load and improving information understanding and decision-making efficiency.

Description

Technical Field

[0001] This specification relates to the field of human-computer interaction technology, and in particular to a medium-free holographic imaging device and its interaction method, system and program products. Background Technology

[0002] Existing display devices have certain shortcomings in human-computer interaction, as their display space is strictly limited to a two-dimensional physical screen plane. All interactive content, whether text, images, or videos, can only be displayed within this closed plane. User interaction with screen content also mostly relies on indirect methods such as keyboards, mice, touchscreens, or remote controls, lacking spatial depth and a sense of presence, resulting in a high cognitive load and low efficiency in information gathering and decision-making. Summary of the Invention

[0003] This specification provides one or more embodiments of an interaction method for a medium-free holographic imaging device, wherein the medium-free holographic imaging device is capable of establishing a signal connection with a target display device, and the interaction method includes: generating a suspended medium-free holographic image in a preset imaging area based on multimedia information of the target display device, wherein the medium-free holographic image includes one or more user-operable interactive areas; acquiring operation information of the user on the interactive areas; and sending the operation information to the target display device.

[0004] In some embodiments, generating a suspended, medium-free holographic image in a preset imaging area based on multimedia information from a target display device includes: a processing program for identifying the multimedia information; determining an interaction program for the medium-free holographic imaging device based on the processing program; and generating the medium-free holographic image based on the interaction program.

[0005] In some embodiments, the multimedia information includes screen information, and generating a floating, medium-free holographic image in a preset imaging area based on the multimedia information of the target display device includes: identifying interactive elements in the screen information; and generating the medium-free holographic image based on the interactive elements.

[0006] In some embodiments, generating a floating, medium-free holographic image in a preset imaging area based on multimedia information from a target display device further includes: identifying the display type of the screen information; and generating the medium-free holographic image based on the interactive elements and the display type.

[0007] In some embodiments, the multimedia information includes displayable interactive elements; generating a floating, medium-free holographic image in a preset imaging area based on the multimedia information of the target display device includes: obtaining a first coordinate of the interactive element displayed on the target display device; determining a second coordinate in the medium-free holographic image based on the first coordinate; and generating a user-operable interactive sub-region in the medium-free holographic image based on the second coordinate.

[0008] In some embodiments, the interaction method further includes: updating the medium-free holographic image in response to feedback information from the target display device regarding the operation information.

[0009] In some embodiments, obtaining user operation information for the interactive area includes: collecting the user's gestures within the preset imaging area and generating corresponding operation information based on the gestures.

[0010] In some embodiments, generating corresponding operation information based on the gesture includes: determining the user's operation type based on the user's gesture and predefined gestures; and generating the operation information based on the user's gesture changes and operation type in the preset imaging area.

[0011] In some embodiments, obtaining user operation information for the interactive area includes: recognizing the user's voice and generating the operation information based on the user's voice.

[0012] In some embodiments, when the multimedia information includes a three-dimensional model, generating a floating, medium-free holographic image in a preset imaging area based on the multimedia information of the target display device includes: receiving model data of the three-dimensional model sent by the target display device; modeling the three-dimensional model based on the model data; and generating a floating three-dimensional model in the preset imaging area.

[0013] Some embodiments of this specification also provide an interactive system for a medium-free holographic imaging device, comprising: a holographic image generation module for generating a suspended medium-free holographic image in a preset imaging area based on multimedia information of a target display device, the medium-free holographic image including multiple user-operable interactive areas; an operation information acquisition module for acquiring operation information of the user on the interactive areas; and an operation information sending module for sending the operation information to the target display device.

[0014] Some embodiments of this specification also provide a medium-free holographic imaging device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it is able to implement the methods described in any embodiment of this specification.

[0015] Some embodiments of this specification also provide a computer program product, including a computer program that, when at least a portion of the computer program is executed by a processor, enables the implementation of the methods described in any embodiment of this specification. Attached Figure Description

[0016] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. The same numbers in the drawings denote the same structures or steps.

[0017] Figure 1 This is a schematic diagram illustrating application scenarios of a medium-free holographic imaging device according to some embodiments of this specification.

[0018] Figure 2 This is a schematic flowchart of an interactive method for a medium-free holographic imaging device according to some embodiments of this specification.

[0019] Figure 3 This is a schematic diagram illustrating the process of generating a media-free holographic image based on multimedia information, according to some embodiments of this specification.

[0020] Figure 4 This is a schematic diagram illustrating the process of generating a media-free holographic image based on multimedia information, according to other embodiments of this specification.

[0021] Figure 5 This is a schematic diagram illustrating the process of generating a media-free holographic image based on multimedia information according to some embodiments of this specification.

[0022] Figure 6 This is a flowchart illustrating the process of obtaining user operation information for interactive areas, based on some embodiments of this specification.

[0023] Figure 7 This is a schematic diagram of the process for generating media-free holographic images based on multimedia information, as shown in some embodiments of this specification.

[0024] Figure 8 This is a schematic diagram of the structure of the interactive system of a medium-free holographic imaging device according to some embodiments of this specification. Detailed Implementation

[0025] To more clearly illustrate the technical solutions of the embodiments in this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the content described below are some examples or embodiments of this specification. For those skilled in the art, without creative effort, the technical solutions or means disclosed in this specification can be applied to other scenarios based on this technical content.

[0026] It should be understood that the terms "system," "device," "equipment," "unit," and / or "module" used in this specification are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other words can achieve the same purpose, they may be replaced by other expressions.

[0027] Unless otherwise specified, the technical terms used to describe components, elements, etc. in this specification are not singular but may include plural. Generally speaking, terms such as "comprising" or "including" only indicate that explicitly identified steps, elements, or components are included, and these steps, elements, and components do not constitute an exclusive list, as the described method or apparatus may also include other steps or components.

[0028] This specification uses flowcharts to illustrate the operational steps performed by the apparatus or system of related embodiments. However, unless otherwise specified, the order in which these steps are described should not be construed as a limitation on the order of execution. Those skilled in the art can adjust the order of these steps based on the knowledge and information conveyed by the embodiments in this specification. Adjustments include, but are not limited to, reversing the order of steps, merging multiple steps, and splitting a step.

[0029] Existing display devices have certain shortcomings in human-computer interaction, as their display space is strictly limited to a two-dimensional physical screen for interaction. Interactive content on this two-dimensional screen, whether text, images, or videos, can only be displayed within this closed plane. User interaction with screen content also largely relies on indirect methods such as keyboards, mice, touchscreens, or remote controls, lacking spatial depth and a sense of presence, resulting in a high cognitive load and low efficiency in information gathering and decision-making.

[0030] In view of this, some embodiments of this specification propose a medium-free holographic imaging device and its interaction method. The medium-free holographic imaging device can establish a signal connection with a target display device and generate a corresponding medium-free holographic image based on the content displayed on the target display device, so that the user can control the target display device through interaction with the medium-free holographic image.

[0031] Figure 1 These are schematic diagrams illustrating application scenarios of a medium-free holographic imaging device according to some embodiments of this specification. For example... Figure 1As shown, application scenario 100 may include a medium-free holographic imaging device 110 and a target display device 120. The medium-free holographic imaging device 110 can establish a signal connection with the target display device 120. The medium-free holographic imaging device 110 can generate suspended images (also called medium-free holographic images) in the air using optical technology, without relying on a physical screen or medium for image display. In some embodiments, the medium-free holographic imaging device 110 may include an imaging source and optical transmission components. The imaging source generates or adjusts the light source image by modulating the phase / amplitude of light waves. In some embodiments, the imaging source may include a display and a light source, where the light source can be a laser diode or an LED light source, providing background illumination, enhancing display brightness and color gamut, and ensuring display light intensity and color accuracy. In some embodiments, the imaging source may include a display, a light source, and a modulator. The modulator is used to adjust the light source image by modulating the phase / amplitude of light waves; for example, the modulator may include a phase-type spatial light modulator. In some embodiments, the imaging source may be based on laser scanning imaging. For example, the imaging source may include a laser diode, a galvanometer, and a modulator. A galvanometer controls a laser diode to emit a laser beam in either grating or vector mode, rapidly scanning a virtual plane to form a medium-free holographic image. For example, the laser emitted by the laser diode, under the control of the galvanometer, scans pixels (e.g., the outline of a floating button) point by point on a virtual plane to form a medium-free holographic image. It should be noted that the medium-free holographic imaging device 110 based on laser scanning imaging may not include an optical transmission element, and may not generate a light source image during the imaging process. In some embodiments, the optical transmission element may include, but is not limited to, the optical elements or devices described in any of the Chinese patents with application numbers 202210060077.2, 202221492951.1, and 201920104395.8. The optical transmission element may transmit light waves or light fields (i.e., light source images) in a thin-film medium through the principle of total internal reflection, achieving transparent transmission and expansion of the light source image. The optical transmission element may also include other geometric waveguides, diffractive waveguides, etc. Geometric waveguides achieve light wave transmission and processing based on prisms or mirror arrays. Diffractive waveguides utilize surface-embossed gratings or volume holographic gratings to couple light beams, enabling light wave transmission and processing. In some embodiments, the optical transmission element may include free-form optical components, such as lens groups (e.g., aspherical lenses, Fresnel lenses), mirrors (e.g., high-speed galvanometers or faceted prisms), etc. In some embodiments, the optical transmission element may include holographic optical elements, using a holographic recording medium (e.g., photosensitive polymer) to achieve the functions of complex optical devices (e.g., lenses, beam splitters).

[0032] In some embodiments, the medium-free holographic imaging device 110 can be used to generate a medium-free holographic image 111. The medium-free holographic imaging device 110 can perform screen-based interaction based on the generated medium-free holographic image 111. Screen-based interaction refers to the two-way information transmission between the user and the device by using a display screen or virtual display screen as an interactive interface, combined with one or more input methods such as touch, gestures, and buttons. The screen-based interaction of the medium-free holographic imaging device 110 in the embodiments of this specification can be widely applied in multiple technical fields, and its typical application scenarios include, but are not limited to: consumer electronics: such as interactive control of smartphones and tablets; smart wearable devices: such as tactile interaction points for quick control; smart home and home appliance: such as smart central control screens and home appliance interactive screens; self-service terminal: such as self-service cash registers and digital signage; industrial control: such as industrial control panels and elevator control panels; medical: such as medical device interactive screens and self-service health terminals; transportation and travel: such as in-vehicle infotainment systems and public transportation ticketing terminals.

[0033] In some embodiments, the media-free holographic imaging device 110 may include a processor 112. The processor 112 may be used to process information and / or data related to the screen display interaction of the media-free holographic imaging device 110. The processor 112 may execute program instructions based on this data, information, and / or processing results to perform one or more functions described in this specification. In some embodiments, the processor 112 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processor (PPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), a microprocessor, or any combination thereof.

[0034] In some embodiments, to achieve screen interaction functionality, screen interaction sensing technology can be used to capture and interpret user operation commands. In some embodiments, this sensing technology can be implemented through one or more of the following technical solutions: 1. Optical sensor technology: This technology acquires the three-dimensional spatial information of the user's hand through optical means. Its implementation includes, but is not limited to: Solution 1: emitting infrared light through an infrared sensor and calculating the reflection time of the infrared light to generate a depth image, thereby determining the position of the user's fingers. Solution 2: performing visual positioning through a multi-camera system to reconstruct the three-dimensional (3D) position and posture of the user's hand. As a specific implementation method, taking the user's finger clicking a virtual button floating in the air as an example, the interaction process is as follows: First, the infrared camera emits infrared pulses and receives their reflections, calculating the distance between the finger and the camera based on the time difference between emission and reflection, and then constructing a depth map containing the finger; next, key point detection is performed on the depth map to establish a 3D hand model, so as to obtain the precise position of each key point of the hand in the 3D coordinate system; then, through a preset algorithm, the 3D coordinates of each key point of the hand are mapped in real time to the virtual coordinate system of the mediumless holographic image; the system continuously judges whether the fingertip coordinates are close to or reach the preset "floating button" interaction area; if it is determined that the fingertip has entered the interaction area, a feedback mechanism is triggered (e.g., the button is highlighted or deformed) and the corresponding system response event is executed (e.g., the menu is opened). 2. Millimeter-wave radar technology: This technology obtains the target distance information by emitting millimeter waves with varying frequencies and calculating the frequency difference between the echo signal and the emitted wave; at the same time, the target angle information is calculated by the phase difference of the signals received by multiple antennas. Finally, based on the acquired distance and angle information, the coordinate information of the user's finger in three-dimensional space is calculated. 3. Ultrasonic sensing technology: This technology collects information synchronously through multiple ultrasonic receivers. By calculating the time difference or phase difference between the ultrasonic waves emitted and received, the distance between the finger and each receiver is determined. Furthermore, based on the distance data from multiple receivers, algorithms such as triangulation are used to calculate the coordinates of the user's finger in three-dimensional space.

[0035] In some embodiments, the target display device 120 may be an electronic device with a display screen. In some embodiments, the target display device 120 may include, but is not limited to, a laptop computer, desktop computer, tablet computer, television, vehicle infotainment system, mobile communication device, etc. In some embodiments, the medialess holographic imaging device 110 may be mounted above, to the left, or to the right of the screen of the target display device 120, or placed independently on a desktop or the ground. In some embodiments, the target display device 120 is a laptop computer; the medialess holographic imaging device 110 may be mounted above, to the left, or to the right of the laptop computer. In some embodiments, the target display device 120 may be a television set, in which case the medialess holographic imaging device 110 may be placed on the ground or on a stand to generate the corresponding medialess holographic image. In some embodiments, the target display device 120 may be a vehicle infotainment system, in which case the medialess holographic imaging device 110 may be placed on the dashboard, instrument panel, or other locations that do not obstruct the driver's view, displaying the desired medialess holographic image to the driver or passenger.

[0036] In some embodiments, the target display device 120 may include a processing device 121, which can be used to process logical operations and image operations in the target display device 120. In some embodiments, the processing device 121 and the processor 112 in the mediumless holographic imaging device 110 can be communicatively connected. The processing device 121 is used to send data required by the processor 112 and receive user operation information acquired by the processor 112. In some embodiments, the processing device 121 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processor (PPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic circuit (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), a microprocessor, or any combination thereof.

[0037] In some embodiments, the media-free holographic imaging device 110 and the target display device 120 can establish a signal connection via a wired connection. For example, the media-free holographic imaging device 110 and the target display device 120 can be connected via a data transmission interface, data cable, etc. In some embodiments, the media-free holographic imaging device 110 and the target display device 120 can establish a signal connection via a network 130. In some embodiments, the network 130 can be any form of wired or wireless network, or any combination thereof. By way of example only, the network 130 can include a cable network, wired network, fiber optic network, telecommunications network, internal network, Internet, local area network (LAN), wide area network (WAN), wireless local area network (WLAN), metropolitan area network (MAN), public switched telephone network (PSTN), Bluetooth network, VPN, near field communication (NFC), device bus, device wiring, cable connection, etc., or any combination thereof. In some embodiments, the network 130 can include one or more network access points. For example, network 130 may include a wired or wireless network access point through which the medialess holographic imaging device 110 and the target display device 120 can connect to network 130 to exchange data and / or information.

[0038] Figure 2 This is a flowchart illustrating an interaction method for a medium-free holographic imaging device according to some embodiments of this specification. In some embodiments, the interaction method 200 of the medium-free holographic imaging device can be implemented by a processor 112. In some embodiments, the interaction method 200 of the medium-free holographic imaging device can be implemented by an interaction system 800 of the medium-free holographic imaging device. Figure 2 As shown, the interaction method 200 of the medium-free holographic imaging device may include the following steps.

[0039] Step 210: Generate a suspended, media-free holographic image in a preset imaging area based on the multimedia information of the target display device. The media-free holographic image may include one or more user-operable interactive areas. In some embodiments, step 210 may be implemented by the holographic image generation module 810.

[0040] In some embodiments, the processor 112 may acquire multimedia information sent by the target display device 120. In some embodiments, the processor 112 may send a request to the target display device for acquiring multimedia information, and the target display device 120 (such as the processing device 121) responds to the request by sending multimedia information to the medium-free holographic imaging device 110. In some embodiments, the target display device may periodically send multimedia information to the medium-free holographic imaging device.

[0041] Multimedia information may include at least one of the image information and audio information that the target display device 120 needs to display. In some embodiments, the target display device 120 may periodically send multimedia information to the mediumless holographic imaging device 110. In other embodiments, the mediumless holographic imaging device 110 may actively collect multimedia information from the target display device through methods including, but not limited to, API interfaces, mirror mapping, audio sensors, and image sensors. In some embodiments, when the target display device 120 is a vehicle or a vehicle's infotainment system, the multimedia information may include vehicle navigation, vehicle information, environmental information, multimedia entertainment, real-time weather, and traffic warning information. In some embodiments, when the target display device 120 is a mobile phone, the multimedia information may include music data, call data, etc., specifically including track names, volume levels, caller ID numbers, and contact names. In some embodiments, when the target display device 120 is a television set, in the scenario of playing a sports event, the multimedia information may include real-time scores, player information, and other data.

[0042] In some embodiments, the preset imaging area can be specifically set according to the usage scenario. For example, when the medium-free holographic imaging device is mounted on the display screen of a laptop, the preset imaging area is a position facing the user and convenient for the user to operate. In some embodiments, when the user's operation includes gesture operation, the preset imaging area is a position convenient for the user to perform gesture operation. In some embodiments, the preset imaging area can be adjusted according to the mounting position, placement position, and orientation of the medium-free holographic imaging device.

[0043] Step 220: Obtain user operation information regarding the interactive area. In some embodiments, step 220 may be implemented by the operation information acquisition module 820.

[0044] In some embodiments, the interactive elements in the interactive area can be diverse, including but not limited to buttons, draggable progress bars, and page-turning pages. During use, the operation information includes which interactive area the user operated on and the type of operation. In some embodiments, the media-free holographic imaging device can send the operation information to a target display device, which updates the multimedia information based on the operation information and sends the updated multimedia information to the media-free holographic imaging device. The media-free holographic imaging device can periodically repeat step 210.

[0045] In some embodiments, user operation information for interactive areas may include user gestures. In some embodiments, a medium-free holographic imaging device can recognize user gestures through sensors to obtain user operation information for interactive areas. In some embodiments, sensors may include optical sensors, specifically using infrared time-of-flight principles to generate depth maps or using multi-camera visual positioning technology to reconstruct a 3D model and posture of the hand. In some embodiments, sensors may include millimeter-wave radar, specifically using analysis of the frequency and phase difference of transmitted and received electromagnetic waves to calculate the distance and angle information of the hand in 3D space. In some embodiments, sensors may include ultrasonic sensors, specifically using multiple receivers to collect sound wave signals and performing triangulation based on time-of-flight differences or phase differences to determine the 3D coordinates of key points on the hand.

[0046] In some embodiments, the medium-free holographic imaging device can also recognize the user's intent based on the user's voice. In some embodiments, the processor 112 can recognize the user's voice based on preset words preset by the user or the system, or it can use a combination of preset words for recognition. In some embodiments, a large language model can be used to recognize the user's voice and analyze its semantics to achieve the purpose of recognizing the user's intent based on its semantics. For example, the medium-free holographic imaging device can recognize that when the user says "play music," the operation information is to play music; when the user says "pause video," the operation information is to pause the currently playing video; when the user says "show notification," the operation information is to request to display the content of the notification.

[0047] Step 230: Send operation information to the target display device. In some embodiments, step 230 can be implemented by the operation information sending module 830.

[0048] In some embodiments, the target display device may not perform any action after receiving the operation information. In some embodiments, the target display device may perform a corresponding operation based on the operation information after receiving the operation information. In some embodiments, the target display device may not provide feedback information to the medium-free holographic imaging device. In some embodiments, the target display device may provide feedback information on the operation information to the medium-free holographic imaging device. In some embodiments, the interaction method 200 of the medium-free holographic imaging device may further include step 240.

[0049] Step 240: In response to feedback from the target display device regarding operational information, update the media-free holographic image. In some embodiments, step 240 may be implemented by the holographic image generation module 810.

[0050] In some embodiments, after receiving the operation information, the target display device can operate on the displayed multimedia information. In this case, the feedback information can be the operation result information of the target display device after the operation. In a specific embodiment, the feedback information may be that the music has paused, the music is in a paused state, the page has been switched, or the page has been zoomed in / out. In some embodiments, the feedback information may be the multimedia information updated by the target display device based on the operation information.

[0051] Figure 3 This is a schematic flowchart illustrating the generation of a media-free holographic image based on multimedia information according to some embodiments of this specification. In some embodiments, the process 300 of generating a media-free holographic image based on multimedia information can be implemented by the processor 112 and / or the holographic image generation module 810. Figure 3 As shown, process 300 may include the following steps.

[0052] Step 310: Identify the multimedia information processing procedure.

[0053] In some embodiments, the processing program may include a video playback program, a music program, or a short video program. In some embodiments, the processor 112 may identify the multimedia information processing program based on signals emitted by the target display device. In some embodiments, the processor 112 may obtain the multimedia information processing program displayed by the target display device based on an API interface. In some embodiments, the processor 112 may identify the screen content of the multimedia information and determine the multimedia information processing program based on the screen content.

[0054] Step 320: Determine the interactive program for the medium-free holographic imaging device based on the processing program.

[0055] In some embodiments, the interactive program can be pre-set in the medium-free holographic imaging device, and different interactive programs can enable the medium-free holographic imaging device to display different medium-free holographic images. In some embodiments, the processor can determine the interactive program corresponding to the processing program as the interactive program currently being executed by the medium-free holographic imaging device. Specifically, when the processing program is a video playback program, the corresponding interactive program can be a video interactive program; when the processing program is a music program, the corresponding interactive program can be a music interactive program; when the processing program is a short video program, the corresponding interactive program can be a short video interactive program.

[0056] Step 330: Generate a medium-free holographic image based on the interactive program.

[0057] In some embodiments, different interactive programs can correspond to different media-free holographic images. Specifically, when the interactive program is a video interactive program, the generated media-free holographic image may include one or more interactive areas such as play / pause control, progress bar, screen brightness adjustment, and volume adjustment. When the interactive program is a short video interactive program, the generated media-free holographic image may include one or more interactive areas such as swipe, like, favorite, and progress bar.

[0058] By calling an interactive program corresponding to the processing program of the target display device in the mediumless holographic imaging device, the mediumless holographic image displayed by the mediumless holographic imaging device can correspond to multimedia information; thus, the user can interactively control the target display device based on the mediumless holographic image.

[0059] Figure 4 This is a schematic flowchart illustrating the generation of a media-free holographic image based on multimedia information according to other embodiments of this specification. In some embodiments, the process 400 for generating a media-free holographic image based on multimedia information can be implemented by the processor 112 and / or the holographic image generation module 810. Figure 4 As shown, process 400 may include the following steps.

[0060] Step 410: Identify interactive elements in the screen information.

[0061] In some embodiments, multimedia information includes screen information, and interactive elements can be content on the screen that can be used to prompt the user or be operated by the user. In some embodiments, interactive elements may include a "button" for playing / pausing video, a "slider" for controlling the video playback progress, and a text box for inputting text, etc. The specific interactive elements can be determined according to different screen information.

[0062] In some embodiments, the processor 112 can identify interactive elements in the screen information through a trained image recognition model. In some embodiments, the image recognition model may include, but is not limited to, one or any combination of neural network models, BERT models, or large language models. In some embodiments, the training process of the image recognition model may include one or more of the following steps: (1) Collecting screenshots: acquiring a large number of UI screenshots from different applications, websites, and operating systems (such as iOS, Android, Web, Windows, and vehicle systems); the greater the data diversity, the better the generalization ability of the model; (2) Data annotation: manually drawing all interactive elements on the screenshots using annotation tools and assigning them category labels; (3) Loading pre-trained weights: downloading the pre-trained model. The model can be a model that has already learned to recognize general shapes, textures and object concepts; (4) Configure the dataset: Create a file describing the dataset, specifying the paths to the training set, validation set and the names of all category labels; (5) Start training: Fine-tune the pre-trained model on the UI dataset. The model will use its existing general visual knowledge to quickly learn to recognize UI elements; (6) Evaluate the model: After training, evaluate the model's performance (such as precision and recall) on a test set that has never been seen before to ensure that the model meets the requirements.

[0063] Step 420: Generate a medium-free holographic image based on interactive elements.

[0064] In some embodiments, the medium-free holographic image can display interactive elements or corresponding operable options. In some embodiments, the generated medium-free holographic image can map the interactive elements one-to-one onto the medium-free holographic image, or it can generate the operable options of the interactive elements on the medium-free holographic image. Specifically, when the interactive element is a progress bar, the generated medium-free holographic image can include a progress bar, or it can be a fast-forward button and a rewind button.

[0065] In some embodiments, process 400 further includes step 430, identifying the display type of screen information.

[0066] In some embodiments, the display type may include video playback, audio playback, sports events, vehicle navigation, conference presentations, voice calls, etc. In some embodiments, the processor may identify the display type based on speech recognition and / or multimedia information processing. For example, the processor may recognize the voice prompt "Enter navigation mode" to identify the display type as vehicle navigation. The processor may also determine the display type based on multimedia information processing; for example, recognizing that the user is using video software may determine the display type as video playback; if the user is using navigation software, the display type may be determined as vehicle navigation. In some embodiments, the processor may also identify the display type based on multimedia data (such as the displayed image) of the target display device; for example, if the displayed image on the target display device is a football, the display type may be determined as a sports event.

[0067] Step 440: Generate a media-free holographic image based on the interactive elements and display type. In some embodiments, different media-free holographic images can be generated for different interactive elements and display types.

[0068] In some embodiments, when the display type is video playback, the target display device can send the currently playing video stream information (such as playback progress and volume level) to the mediumless holographic imaging device wirelessly or via wired means. After receiving the information, the mediumless holographic imaging device generates a floating mediumless holographic image containing interactive elements such as play / pause, progress bar, and volume adjustment.

[0069] In some embodiments, when the display type is vehicle navigation, the types of interactive elements may include vehicle navigation, vehicle information, environmental information, multimedia entertainment, display mode switching, exiting navigation, etc. Specifically, interactive elements may include route planning, turn prompts, remaining distance, etc. For example, the medium-free holographic image may include interactive elements such as vehicle navigation arrows (left turn / right turn / straight), remaining distance, speed limit signs, etc., which are updated in real time as the vehicle moves. Specifically, vehicle information may include core vehicle condition data such as vehicle speed, fuel / battery level, fuel / electricity consumption, tire pressure, driving range, and interior temperature. Interactive elements on the medium-free holographic image may specifically include vehicle speed (synchronized with the instrument panel), interior temperature (e.g., "23℃"), remaining driving range (e.g., "120km"), a red "low tire pressure" indicator when tire pressure is low, and a yellow "refueling reminder" when fuel level is below 10%; environmental information may include real-time weather and road condition warnings. Interactive elements on a mediumless holographic image can include weather-related symbols (such as "sun" for sunny days, "clouds" for cloudy days, "snowflakes" for snowy days), temperature numbers corresponding to temperature, and road condition warning signs ahead.

[0070] In some embodiments, multimedia information may include music data, call data, etc. Interactive elements on a media-free holographic image may include track titles, volume levels, caller ID numbers, contact names, answer buttons, hang-up buttons, etc.

[0071] In some embodiments, during vehicle navigation, the mediumless holographic imaging device can also issue corresponding prompts based on interactive elements and prompt the driver in a low-distraction manner (e.g., actively lowering the music volume when prompting the driver during music playback).

[0072] In some embodiments, when the display type is a sporting event, interactive elements on the media-free holographic image may include real-time scores, player information, etc. When the target display device generates a notification message (such as a phone call or text message), the notification message can be displayed as a floating image without interrupting the ongoing sporting event.

[0073] In some embodiments, when the display type is reading, interactive elements on the media-free holographic image may include page up / down buttons and a table of contents index, etc.

[0074] Figure 5 This is a schematic flowchart illustrating the generation of a media-free holographic image based on multimedia information according to some embodiments of this specification. In some embodiments, the process 500 of generating a media-free holographic image based on multimedia information can be implemented by the processor 112 and / or the holographic image generation module 810. Figure 5 As shown, process 500 may include the following steps.

[0075] Step 510: Obtain the first coordinates of the interactive element displayed on the target display device.

[0076] In some embodiments, the first coordinates may be determined by the target display device based on a first coordinate system in its displayed screen. For example, if the target display device's screen has a total of 1920 pixels horizontally and 1080 pixels vertically (i.e., the first coordinate system is 1920×1080), the first coordinates of the interactive element are (22, 10). In some embodiments, the first coordinates may be the coordinates of the center point of the interactive element.

[0077] Step 520: Determine the second coordinate in the mediumless holographic image based on the first coordinate.

[0078] In some embodiments, the second coordinates can be determined based on the first coordinate system mapped onto a second coordinate system on the mediumless holographic image. Specifically, the mediumless holographic image may correspond to 960 pixels horizontally and 540 pixels vertically (i.e., the second coordinate system is 960×540). In this case, the second coordinates are determined as (11, 5) based on the resolution of the target display device's display (1920×1080) and the resolution of the mediumless holographic image (960×540).

[0079] Step 530: Generate a user-operable interactive sub-region in the mediumless holographic image based on the second coordinates.

[0080] In some embodiments, the interactive sub-region is a region within the interactive region, including but not limited to the forms of circles, squares, buttons, progress bars, etc. In some embodiments, the processor 112 can enlarge or shrink the original interactive elements based on the relationship between the first coordinate system and the second coordinate system to form the aforementioned interactive sub-region. In some embodiments, the size of the interactive elements can be freely set according to needs to facilitate user gesture operation. For example, if the size of the formed medium-free holographic image is small, the size of the interactive elements can be set larger to facilitate user gesture operation.

[0081] Figure 6 This is a flowchart illustrating the process of obtaining user operation information on an interactive area according to some embodiments of this specification. In some embodiments, the process 600 for obtaining user operation information on an interactive area can be implemented by the processor 112 and / or the operation information acquisition module 820. Figure 6 As shown, process 600 may include the following steps.

[0082] Step 610: Collect the user's gestures within the preset imaging area.

[0083] In some embodiments, the preset imaging area can be specifically set according to the usage scenario. More details about the setting of the preset imaging area can be found above and will not be repeated here. In some embodiments, the medium-free holographic imaging device can recognize the user's gestures through sensors (such as optical sensors, millimeter-wave radar, ultrasonic sensors, etc.).

[0084] Step 620: Determine the user's operation type based on the user's gestures and predefined gestures.

[0085] In some embodiments, predefined gestures may include a single-finger "click," a four-finger "swipe," a two-finger "zoom out," and a two-finger "zoom in." In some embodiments, the medium-free holographic imaging device may pre-store several predefined gestures and their corresponding operation types, and determine the user's operation type by whether the user's gesture is a predefined gesture. In some optional embodiments, each predefined gesture corresponds to at least one operation type. Specifically, in the case of video playback, if the user makes a "swipe" gesture on the medium-free holographic image, the gesture sensor captures and identifies the action, determining it to be a drag operation type; if the user makes a "click" gesture, it is identified as a confirmed operation type; if the user makes a "zoom in" or "zoom out" gesture, it is determined to be a zoom in or zoom out operation type.

[0086] Step 630: Generate operation information based on the user's gesture changes and operation type in the preset imaging area.

[0087] In some embodiments, the sensor can periodically detect the user's hand movements and determine the user's gesture changes (specifically including changes in the user's hand coordinates and changes in the user's fingers, such as dragging, clenching a fist, and unfolding fingers) based on the detection results of the previous and current detection cycles. Specifically, in the case of video playback, if the user makes a "swipe" gesture on a mediumless holographic image, the gesture sensor captures the action and recognizes it as dragging, and the operation information is "adjust progress," thus changing the video playback progress, and the progress bar in the mediumless holographic image is updated synchronously. If the user makes a "click" gesture, it is recognized as a specific operation type, and the operation information is to play / pause the video. In the in-vehicle system scenario, when there is an incoming call reminder, "click" is recognized as a specific operation type, and the operation information is to answer the call; it is understood that clenching a fist can also be set to achieve the operation command of hanging up the call / mute; if the user makes a "zoom in" or "zoom out" gesture on the map, the operation type is recognized as zoom in or zoom out, and the operation command is to zoom in or zoom out on the map.

[0088] In some embodiments, obtaining user operation information for interactive areas may further include: recognizing the user's voice and generating operation information based on the user's voice.

[0089] In some embodiments, the operation information can be determined by whether a preset voice and corresponding operation information match, or a semantic understanding model can be used to generate the corresponding operation information by understanding the user's voice. For example, a medium-free holographic imaging device can recognize that when a user says "play music," the operation information is to play music; when a user says "pause video," the operation information is to pause the currently playing video; and when a user says "show notification," the operation information is to request the display of the notification content.

[0090] In some embodiments, operation information for the interactive area can be obtained based on the user's eye tracking. Specifically, operation information for a certain interactive element can be determined based on data such as auxiliary actions, gaze position, dwell time, and movement path. For example, actions such as blinking or head micro-movements can be combined to confirm the operation (e.g., "blink confirmation"); and a gaze moving to the interactive area can trigger a corresponding action (e.g., scrolling or page turning).

[0091] Figure 7 This is a schematic flowchart illustrating the generation of a media-free holographic image based on multimedia information according to some embodiments of this specification. In some embodiments, the process 700 of generating a media-free holographic image based on multimedia information can be implemented by the processor 112 and / or the holographic image generation module 810. Figure 7 As shown, in an optional embodiment, when the multimedia information includes a three-dimensional model, the process 700 for generating a media-free holographic image based on the multimedia information may include the following steps.

[0092] Step 710: Receive the model data of the 3D model sent by the target display device.

[0093] In some embodiments, the 3D model may include, but is not limited to, regular geometric models, irregular geometric models, human models, vehicle models, animal models, cartoon models, etc. In some embodiments, the model data of the 3D model may include core parameters used for 3D model modeling. For example, when the 3D model is a cuboid, the model data may include parameters such as the length, width, and height of the cuboid, and a floating cuboid model can be generated within a preset imaging area based on the length, width, and height.

[0094] Step 720: Model the 3D model based on the model data and generate a floating 3D model in the preset imaging area.

[0095] In some embodiments, the processor can model the 3D model based on various parameters of the 3D model, thereby generating a floating 3D model in a preset imaging area. In this embodiment, transforming the 3D model in the target display device into a floating 3D model generated in the preset imaging area can more vividly and effectively display the 3D model, which is beneficial for user understanding. In some embodiments, the processor can also recognize the user's preset gestures to operate the 3D model. Specifically, for example, if the gesture is detected as "zoom in," the 3D model is zoomed in; if the gesture is detected as "zoom out," the 3D model is zoomed out; if the gesture is detected as "single-finger swipe," the 3D model is moved; if the gesture is detected as "two-finger swipe," the 3D model is rotated, etc. The operation information of the 3D model is fed back to the target display device in the form of information, and the target display device can perform corresponding operations on the 3D model displayed on its screen based on the operation information.

[0096] In an optional embodiment, the media-free holographic imaging device can also serve as a second display for the target display device, actively displaying basic information. For example, the media-free holographic imaging device can invoke a built-in clock or calendar program or obtain weather information from the internet, and generate a floating visual image of commonly used information such as time, date, ambient temperature, and air quality, making it an interactive desktop information display or decoration. In some embodiments, when the media-free holographic imaging device is disconnected from the target display device, or when the target display device does not display multimedia information, the media-free holographic imaging device can function as an independent display device.

[0097] Figure 8 This is a schematic diagram of the structure of an interactive system for a medium-free holographic imaging device according to some embodiments of this specification. In some embodiments, the interactive system 800 of the medium-free holographic imaging device may be implemented by a processor 112. Figure 8 As shown, the interactive system 800 of the medium-free holographic imaging device may include a holographic image generation module 810, an operation information acquisition module 820, and an operation information transmission module 830.

[0098] In some embodiments, the holographic image generation module 810 can be used to generate a medium-free holographic image. In some embodiments, the holographic image generation module 810 is used to generate a suspended medium-free holographic image in a preset imaging area based on multimedia information of the target display device. The medium-free holographic image includes multiple user-operable interactive areas.

[0099] In some embodiments, the operation information acquisition module 820 can be used to acquire user operation information. In some embodiments, the operation information acquisition module 820 can acquire user operation information on interactive areas.

[0100] In some embodiments, the operation information sending module 830 can be used to send operation information. In some embodiments, the operation information sending module 830 can send the operation information to a target display device.

[0101] This specification also provides a medium-free holographic imaging device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the interactive method of the medium-free holographic imaging device in the foregoing embodiments.

[0102] This specification also provides a computer program product, including a computer program that, when at least a portion of the computer program is executed by a processor, enables the interactive method of the medium-free holographic imaging device as described in the foregoing embodiments.

[0103] For more information on each module, please refer to the related explanations in the other accompanying diagrams; they will not be repeated here. It should be understood that... Figure 8 The systems and modules shown can be implemented in various ways. For example, in some embodiments, the systems and modules can be implemented by hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the methods and systems described above can be implemented using computer-executable instructions and / or included in the control code of a processor, such as on a media such as a disk, CD, or DVD-ROM, or in the memory of a programmable device. The systems and modules of this specification can be implemented not only by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., but also by software executed by various types of processors, or by a combination of the aforementioned hardware circuits and software (e.g., firmware).

[0104] It should be noted that the above description of the system and its modules is for convenience only and should not be construed as limiting this specification to the embodiments described. It is understood that those skilled in the art, after understanding the principles of this system, may arbitrarily combine the various modules without departing from these principles to form subsystems connected to other modules. Alternatively, some modules may be split to obtain more modules or multiple units under a single module. Such modifications are all within the scope of this specification.

[0105] The beneficial effects that the embodiments of this specification may bring include, but are not limited to: (1) By linking with traditional display devices, interactive elements (such as virtual assistants, control menus, and 3D models) can be "extracted" from the limited flat screen and generated into interactive, medium-free holographic images in the free space in front of the screen, elevating information from a two-dimensional plane to a three-dimensional space. This not only achieves the physical separation of the display dimension and the interaction dimension, but also liberates the presentation of information from a two-dimensional plane to a three-dimensional space, allowing complex structures, data relationships, and dynamic processes to be presented in the most natural way. (2) Users do not need to switch windows back and forth on the screen or frequently perform zoom / rotation operations to process information in parallel, which greatly reduces cognitive load and improves information understanding and decision-making efficiency. (3) The medium-free holographic imaging device is designed to be lightweight and compact. It does not require any structural modifications to existing devices and can be placed independently on the desktop as a universal "accessory" or flexibly set on the bezel of various screens (such as televisions, monitors, and laptops). The non-intrusive integration method allows it to seamlessly integrate into diverse scenarios such as home, office, education, and vehicle, instantly upgrading ordinary two-dimensional display devices into mixed reality terminals with holographic interaction capabilities, effectively protecting users' existing investments and accelerating the popularization of three-dimensional interaction technology. (4) By deeply integrating high-precision gesture recognition, voice commands, and even eye tracking interaction channels, this system breaks the physical constraints of traditional peripherals such as keyboards and mice. Users can interact directly with holographic images in the air through intuitive gestures (such as clicking, dragging, and zooming) and natural language. The highly realistic interaction mode greatly enhances the user's sense of presence and control freedom, transforming human-computer interaction from a cold tool relationship to a warm, natural, and interesting "dialogue" relationship, which is particularly suitable for highly interactive scenarios such as entertainment, display, remote collaboration, and education. It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced can be any one or a combination of the above, or any other possible beneficial effects.

[0106] The basic concepts have been described above. It is obvious that the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to this specification by those skilled in the art. Such modifications, improvements, and corrections are taught in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

Claims

1. An interaction method of a medium-free holographic imaging device, characterized in that, The medium-free holographic imaging device can establish a signal connection with the target display device, and the interaction method includes: A floating, medium-free holographic image is generated in a preset imaging area based on the multimedia information of the target display device. The medium-free holographic image includes one or more user-operable interactive areas. Obtain user operation information regarding the interactive area; The operation information is sent to the target display device.

2. The interaction method of claim 1, wherein, The generation of a suspended, medium-free holographic image in a preset imaging area based on multimedia information from the target display device includes: A processing program for identifying the multimedia information; The interaction program of the medium-free holographic imaging device is determined based on the processing program. The medium-free holographic image is generated based on the interactive program.

3. The interaction method as described in claim 1, characterized in that, The multimedia information includes image information, and the generation of a suspended, medium-free holographic image in a preset imaging area based on the multimedia information of the target display device includes: Identify interactive elements in the screen information; The medium-free holographic image is generated based on the interactive elements.

4. The interaction method as described in claim 3, characterized in that, The generation of a suspended, medium-free holographic image in a preset imaging area based on multimedia information from the target display device also includes: Identify the display type of the screen information; The medium-free holographic image is generated based on the interactive elements and the display type.

5. The interaction method as described in claim 1, characterized in that, The multimedia information includes displayable interactive elements; The generation of a suspended, medium-free holographic image in a preset imaging area based on multimedia information from the target display device includes: Obtain the first coordinates of the interactive element displayed on the target display device; The second coordinate in the medium-free holographic image is determined based on the first coordinate; Based on the second coordinates, an interactive sub-region that can be operated by the user is generated in the mediumless holographic image.

6. The interaction method as described in claim 1, characterized in that, The interaction method also includes: In response to feedback from the target display device regarding the operation information, the medium-free holographic image is updated.

7. The interaction method as described in claim 1, characterized in that, The step of obtaining user operation information for the interactive area includes: The system collects the user's gestures within the preset imaging area and generates corresponding operation information based on the gestures.

8. The interaction method as described in claim 7, characterized in that, The step of generating corresponding operation information based on the gesture includes: The user's action type is determined based on the user's gestures and predefined gestures; The operation information is generated based on the user's gesture changes and operation type in the preset imaging area.

9. The interaction method as described in claim 1, characterized in that, The step of obtaining user operation information for the interactive area includes: Recognize the user's voice and generate the operation information based on the user's voice.

10. The interaction method as described in claim 1, characterized in that, When the multimedia information includes a three-dimensional model, the generation of a suspended, medium-free holographic image in a preset imaging area based on the multimedia information of the target display device includes: Receive model data of the three-dimensional model sent by the target display device; The three-dimensional model is modeled based on the model data, and a floating three-dimensional model is generated in the preset imaging area.

11. An interactive system for a medium-free holographic imaging device, characterized in that, The interactive system includes: A holographic image generation module is used to generate a suspended, medium-free holographic image in a preset imaging area based on multimedia information of a target display device. The medium-free holographic image includes multiple user-operable interactive areas. The operation information acquisition module is used to acquire operation information of the user's operation on the interactive area; An operation information sending module is used to send the operation information to the target display device.

12. A medium-free holographic imaging device, characterized in that, The medium-free holographic imaging device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it is able to implement the method as described in any one of claims 1 to 10.

13. A computer program product, characterized in that, It includes a computer program that, when at least a portion of the computer program is executed by a processor, enables the implementation of the method as described in any one of claims 1 to 10.

Images