Multi-modal content display method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-modal content display method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining multi-modal display contents of a plurality of external equipment, and adjusting the multi-modal display contents according to a preset data format to obtain an adjustment result; respectively creating UI interfaces corresponding to the plurality of external devices according to the adjustment result; and in response to a received display instruction of a target external device in the plurality of external devices, controlling the multi-mode display module to display a UI interface corresponding to the target external device according to the display instruction. Based on the method and the device, the multi-modal display content can be adjusted according to the preset data format, so that under the condition that the operating systems of a plurality of external devices are different, the data formats are unified, information transmission and display content synchronization among the devices crossing the operating systems are facilitated, the efficiency of cooperative work of multiple devices is improved, and the user experience is improved. Display of multi-mode content can be compatible, and diversified requirements of a user in different scenes are met.

Description

Technical Field

[0001] The present application relates to the field of display technology, and in particular to a multimodal content display method, device, electronic device and storage medium. Background Art

[0002] At present, most existing devices are difficult to display 2D (Two-Dimensional) data and 3D (Three-Dimensional) data at the same time. This means that users cannot flexibly switch according to actual needs and enjoy the unique experience brought by 2D and 3D at the same time in different usage scenarios. For example, when watching a movie, it may not be possible to switch from 2D images to 3D effects at the required moment to enhance the sense of immersion; when doing design work, it is also impossible to use 2D precise data and 3D intuitive models at the same time. In addition, in a cross-operating system environment, it is difficult to integrate information between external devices connected to the display device. The transmission of display content is often not interoperable, which greatly limits the efficiency and effectiveness of multi-device collaborative work. For example, when users try to share work documents or multimedia content between devices with different operating systems, they will encounter many obstacles, such as delays, format incompatibility, and content loss. Summary of the invention

[0003] Embodiments of the present application provide a multimodal content display method, device, electronic device, and storage medium to solve one or more of the above-mentioned technical problems.

[0004] In a first aspect, an embodiment of the present application provides a multimodal content display method, comprising: acquiring multimodal display content of multiple external devices, adjusting the multimodal display content respectively according to a preset data format, and obtaining adjustment results; creating UI interfaces corresponding to the multiple external devices respectively according to the adjustment results; in response to receiving a display instruction of a target external device among the multiple external devices, controlling the multimodal display module to display the UI interface corresponding to the target external device according to the display instruction.

[0005] In a second aspect, an embodiment of the present application provides a multimodal data display device, comprising: an acquisition module, used to acquire multimodal display content of multiple external devices, and adjust the multimodal display content respectively according to a preset data format to obtain adjustment results; an interface module, used to create UI interfaces corresponding to the multiple external devices respectively according to the adjustment results; and a control module, used to respond to receiving a display instruction of a target external device among the multiple external devices, and according to the display instruction, control the multimodal display module to display the UI interface corresponding to the target external device.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0008] Compared with the related art, this application has the following advantages:

[0009] The present application provides a multimodal content display method, device, electronic device and storage medium, the method comprising: obtaining multimodal display content of multiple external devices, adjusting the multimodal display content respectively according to a preset data format, and obtaining adjustment results; creating UI interfaces corresponding to the multiple external devices respectively according to the adjustment results; in response to receiving a display instruction of a target external device among the multiple external devices, controlling the multimodal display module to display the UI interface corresponding to the target external device according to the display instruction. Based on the present application, the multimodal display content can be adjusted according to the preset data format, so that when the operating systems of the multiple external devices are different, the data format can be unified, which facilitates information transmission and display content synchronization between devices across operating systems, and improves the efficiency of multi-device collaborative work. In addition, according to the adjustment results, the UI interfaces corresponding to the multiple external devices are respectively created; in response to receiving a display instruction of a target external device among the multiple external devices, controlling the multimodal display module to display the UI interface corresponding to the target external device according to the display instruction, so that the display of multimodal content can be compatible at the same time, meeting the diverse needs of users in different scenarios.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0012] Figure 1 A flow chart of a multimodal content display method provided in an embodiment of the present application is shown;

[0013] Figure 2 A multi-modal device architecture diagram provided in an embodiment of the present application is shown;

[0014] Figure 3 A schematic diagram of multimodal data processing provided in an embodiment of the present application is shown;

[0015] Figure 4 A schematic diagram of a data transmission protocol provided in an embodiment of the present application is shown;

[0016] Figure 5 A schematic diagram of various display modes provided in the embodiments of the present application is shown;

[0017] Figure 6 A schematic diagram of a floating window effect provided in an embodiment of the present application is shown;

[0018] Figure 7 A schematic diagram of a UI interface creation process based on two-view three-dimensional data provided in an embodiment of the present application is shown;

[0019] Figure 8 A schematic diagram of a UI interface creation process based on multi-view three-dimensional data provided in an embodiment of the present application is shown;

[0020] Fig. 9 A schematic diagram of a process of converting 3D content into 2D content based on two-view three-dimensional data provided in an embodiment of the present application is shown;

[0021] Fig.10 A schematic diagram of a process of converting 3D content into 2D content based on multi-view three-dimensional data provided in an embodiment of the present application is shown;

[0022] Fig.11 A structural block diagram of a multimodal content display device provided in an embodiment of the present application is shown;

[0023] Fig.12 A block diagram of an electronic device used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0025] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0026] In today's technological development trend, display interactive devices play an increasingly important role in people's work, life and entertainment. However, current display interactive devices face a series of challenges and limitations.

[0027] At present, there are many obvious defects in the presentation of multi-device information. In the existing solutions, there is a lack of 3D information presentation, and the icon presentation method is relatively simple, which cannot meet the user's needs for personalization and intuitiveness. The switching between 2D and 3D content is not smooth enough, which brings unnecessary waiting and trouble to users. Window management is also very chaotic. When processing multiple 2D and 3D windows at the same time, efficient organization and layout cannot be achieved.

[0028] In addition, the single and incompatible interaction methods are a prominent problem. When users operate different types of display content, such as 2D documents and 3D games, they can only rely on a limited number of interaction methods, and these methods cannot be intelligently adapted according to the characteristics of the display content. This not only leads to low operating efficiency, but also seriously affects the user experience, making cross-platform interaction difficult and cumbersome.

[0029] In summary, it is difficult for existing devices to be compatible with 2D and 3D displays at the same time, and they cannot meet the diverse needs of users in different scenarios. It is difficult to transmit and integrate information between devices across operating systems, and the display content transmission is not synchronized, which limits the efficiency and effect of multi-device collaboration. After adding 3D content, there are many deficiencies in information presentation, such as how to optimize the display of 2D and 3D information and improve the efficiency of information acquisition. The single and incompatible existing interaction methods, when 2D and 3D content exist at the same time, lead to poor user experience when operating different types of display content, and cannot achieve efficient and flexible interaction. Current display interaction devices have many problems that need to be solved in terms of 2D and 3D compatibility, cross-operating system information integration, multi-device information presentation, and interaction methods, and cannot meet the growing diversified and high-quality display interaction needs of users.

[0030] Based on this, the present application provides a multimodal content display method, device, electronic device and storage medium. The method can be compatible with 2D content and 3D content, solve the above problems from the aspects of multimodal system compatibility, multimodal display, multimodal interaction, etc., and improve the information acquisition efficiency and interaction efficiency when 2D and 3D information exist at the same time.

[0031] First, a possible architecture for implementing this approach is described. Figure 2 A multimodal device architecture diagram is shown, and the multimodal device used to implement the method may include the following architecture: a multimodal information communication module, a multimodal interaction module, a multimodal information processing module, a multimodal 3D system and a multimodal display module.

[0032] Among them, the multimodal information communication module is used to establish communication with external devices, and the communication method is not limited to wired or wireless. On the other hand, it is used to receive and integrate content from multiple external devices, thereby realizing the transmission and synchronization of display content across operating systems. For example, see Figure 3 A schematic diagram of multimodal data processing is shown, the peripherals include computers, mobile phones, notebooks, etc., and the processed data includes 2D content, 3D content, 2D and 3D fusion content, etc. The multimodal interaction module can be compatible with a variety of interactive devices, not limited to mouse, keyboard, gestures, eye movement, voice, etc. Multimodal information processing module: It is the operating carrier of the 3D multimodal system; it is responsible for processing the interactive instructions transmitted by the multimodal interaction module and generating corresponding operations; it is responsible for processing the multi-device content transmitted by the multimodal information communication module. The multimodal display module is used to display the information processed by the multimodal 3D system. The display mode supports 2D display, 3D display, and 2D & 3D fusion display.

[0033] The multimodal 3D system can identify different display content. The multimodal 3D system calls the interactive device according to the different forms of content identified. According to the information interface transmission protocol in the multimodal information communication module, the received content type, basic content information, 3D content area range and content data are parsed; the system creates the system UI interface according to the type of display content. In addition, the multimodal 3D system supports the use of multimodal display devices directly as external ordinary display modes. For example, after the cloud box is connected to the display, it can be used as an ordinary external display; the multimodal 3D system supports the 3D AI assistant to be presented in the form of a 3D virtual image, and the interactive method supports voice, text, etc.; the 3D AI assistant can record the local input information, form a local information library model, prevent external information leakage, and has iterative evolution functions; the multimodal 3D mode supports quick exit, such as shortcut keys, quick voice, etc.

[0034] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0035] The present application embodiment provides a multi-modal content display method. Figure 1 FIG. 1 is a flowchart of a multimodal content display method according to an embodiment of the present application. The method may include:

[0036] Step S101, obtaining multimodal display contents of a plurality of external devices, and adjusting the multimodal display contents respectively according to a preset data format to obtain adjustment results.

[0037] In an embodiment of the present application, the execution subject of the method can be connected to multiple external devices, and the communication mode is not limited to wired communication or wireless communication. The external devices may include but are not limited to the following devices: tablet computer (Pad), virtual reality device (Virtual Reality Device, VR), augmented reality device (Augmented Reality Device, AR), computer, all-in-one machine, projector, speaker, tactile feedback device, force feedback handle, mouse, keyboard, device controlled by gesture, device controlled by eye movement, voice, 3D AI (Artificial Intelligence, artificial intelligence) assistant, mobile phone and notebook, etc. Multimodal display content can be a content form that simultaneously presents information through multiple sensory channels (such as vision, hearing, touch, etc.) to enhance user experience and information communication effect, specifically not limited to 2D content, 3D content, 2D and 3D fusion content.

[0038] In the embodiments of the present application, different external devices can process different display contents. For example, some external devices can process 2D contents, some can process 3D contents, and some can process 2D and 3D fusion contents. Therefore, the multiple display contents of multiple external devices are referred to as multimodal display contents.

[0039] The preset data format can be a set of standards and specifications for defining data exchange rules and formats in computer networks and communication systems. The preset data format can be determined according to actual needs so as to adjust the multimodal display content of multiple external devices to the same form and standardize data from different sources, that is, adjust the multimodal display content separately according to the preset data format to obtain the adjustment result.

[0040] In this step, when the operating systems of multiple external devices are different, the data format can be unified, which facilitates information transmission and display content synchronization between devices across operating systems and improves the efficiency of multi-device collaborative work.

[0041] Step S102: creating UI interfaces (User Interface) corresponding to the plurality of external devices respectively according to the adjustment results.

[0042] In the embodiment of the present application, the adjustment result includes information that can be used to identify the external device corresponding to the current adjustment result, and includes information about the content to be displayed. Therefore, according to the adjustment result, a UI interface corresponding to each external device can be created respectively, thereby creating corresponding UI interfaces for different external devices.

[0043] It should be noted that in the embodiments of the present application, the UI interface refers to the interface for users to interact with computer systems, software applications or devices. The UI interface presents information to users through multiple sensory channels such as vision, touch and hearing, and accepts user input and instructions. Based on the UI interface, an easy-to-use, intuitive and efficient way can be provided for users to interact and operate with the system.

[0044] Step S103, in response to receiving a display instruction of a target external device among the multiple external devices, controlling the multimodal display module to display a UI interface corresponding to the target external device according to the display instruction.

[0045] In an embodiment of the present application, the execution subject of the method can communicate with the external devices respectively, and when receiving the display instruction of one or several external devices among the multiple external devices, the one or several external devices are used as the target external devices. The display instruction can be used to determine how to display and which external device's UI interface to display. Therefore, according to the actual instruction, the multimodal display module can be controlled to display the UI interface corresponding to the target external device. Among them, the multimodal display module can be connected to the execution subject of the embodiment of the present application in communication.

[0046] In this step, different external devices can display different content, and multiple external devices can display multimodal content. In response to receiving a display instruction from a target external device among the multiple external devices, the multimodal display module is controlled to display the UI interface corresponding to the target external device according to the display instruction, thereby being compatible with the display of multimodal content and meeting the diverse needs of users in different scenarios.

[0047] The present application provides a multimodal content display method, the method comprising: obtaining multimodal display content of multiple external devices, adjusting the multimodal display content respectively according to a preset data format, and obtaining adjustment results; creating UI interfaces corresponding to the multiple external devices respectively according to the adjustment results; in response to receiving a display instruction of a target external device among the multiple external devices, controlling the multimodal display module to display the UI interface corresponding to the target external device according to the display instruction. Based on the present application, the multimodal display content can be adjusted according to the preset data format, so that when the operating systems of the multiple external devices are different, the data format can be unified, which facilitates information transmission and display content synchronization between devices across operating systems, and improves the efficiency of multi-device collaborative work. In addition, according to the adjustment results, the UI interfaces corresponding to the multiple external devices are respectively created; in response to receiving a display instruction of a target external device among the multiple external devices, controlling the multimodal display module to display the UI interface corresponding to the target external device according to the display instruction, so that the display of multimodal content can be compatible at the same time, meeting the diverse needs of users in different scenarios.

[0048] In a possible implementation, the multimodal display content includes two-dimensional data, three-dimensional data, and two-dimensional and three-dimensional mixed data; and creating UI interfaces corresponding to the multiple external devices respectively according to the adjustment results can be performed according to the following steps: generating a two-dimensional UI interface for the two-dimensional data, and generating a three-dimensional UI interface for the three-dimensional data; generating a two-dimensional mixed layer and a three-dimensional mixed layer corresponding to the two-dimensional and three-dimensional mixed data, and generating a two-dimensional and three-dimensional mixed UI interface according to the two-dimensional mixed layer and the three-dimensional mixed layer.

[0049] In this possible implementation, two-dimensional data is data organized and represented in two dimensions. For example, it includes but is not limited to: images, two-dimensional coordinates, tables, matrices, etc. Three-dimensional data is data organized and represented in three dimensions. For example, it includes but is not limited to: three-dimensional coordinates, three-dimensional point clouds, volume data, etc. Two-dimensional and three-dimensional mixed data is a data set containing two-dimensional and three-dimensional information. For example, it includes but is not limited to: three-dimensional terrain data on a two-dimensional map, three-dimensional annotations on a two-dimensional image, three-dimensional data points in a two-dimensional icon, etc.

[0050] When creating multiple UI interfaces corresponding to external devices according to the adjustment results, for two-dimensional data, a two-dimensional UI interface can be created to display the two-dimensional data. In a two-dimensional user interface, all elements are arranged on a plane, and user interface elements (such as buttons, menus, and text boxes) can be plane graphics. For three-dimensional data, a three-dimensional UI interface can be created to display three-dimensional data. In a three-dimensional UI interface, interface elements can be arranged in three-dimensional space, with a sense of depth, which can provide a more realistic and immersive user experience.

[0051] For 2D and 3D mixed data, generate 2D mixed layers and 3D mixed layers for the 2D and 3D data respectively. When generating 2D mixed layers, refer to Figure 7 The following is a schematic diagram of the UI interface creation process based on two-view 3D data. The 2D UI layer and 2D content are rendered normally. After rendering, layer synthesis is performed to synthesize the 2D UI layer and 2D content into the same image frame buffer to obtain a two-dimensional hybrid layer. When generating a three-dimensional hybrid layer, refer to Figure 7 , the 3DUI layer and the 3D content are synthesized into another image frame buffer to obtain a three-dimensional mixed layer. Figure 8 The schematic diagram of the UI interface creation process based on multi-view 3D data is shown. The 3D UI layer and 3D content can be divided into dual-view and multi-view modes. Finally, the back-end processing performs corresponding image processing according to the 2D and 3D area position information. The 2D area is presented in 2D form, and the 3D area is presented in 3D form. The 2D mixed layer and the 3D mixed layer are mixed again into the same image buffer area to obtain a 2D and 3D mixed UI interface, that is, a 2D and 3D mixed UI interface is generated according to the 2D mixed layer and the 3D mixed layer.

[0052] In this step, when creating a UI interface, corresponding UI interfaces are created for two-dimensional data, three-dimensional data, and two-dimensional and three-dimensional mixed data. Figure 5 and Figure 6 , the execution subject and external device of this method can present the icons of 3D applications in a floating manner on the screen, and the icons of 2D applications can be presented in pure 2D or in a floating 2D space window suspended outside the screen; 3D application content can be presented in full-screen 3D or partial 3D (depth of field about 10cm), and 2D application content can be presented in ordinary 2D or in a floating 2D space window suspended outside the screen. When 3D and 2D icons / content are presented at the same time, a layer is formed, and the 3D icons / content float on the 2D icons / content. 2D and 3D can be switched in real time.

[0053] In a possible implementation, generating a two-dimensional and three-dimensional hybrid UI interface based on the two-dimensional hybrid layer and the three-dimensional hybrid layer can be performed according to the following steps: determining the first level of the two-dimensional hybrid layer and the second level of the three-dimensional hybrid layer according to the depth data of the two-dimensional hybrid layer and the depth data of the three-dimensional hybrid layer; determining the order of the two-dimensional hybrid layer and the three-dimensional hybrid layer according to the first level and the second level, and constructing a two-dimensional and three-dimensional hybrid UI interface.

[0054] In this possible implementation, the depth of field data is used to describe the information about the distance between different objects in the scene and the camera. These data can help determine which parts are clear and which parts are blurred, thereby achieving a more realistic effect in image processing. According to the depth of field data of the two-dimensional mixed layer and the depth of field data of the three-dimensional mixed layer, the three-dimensional data and the two-dimensional data can be layered, that is, the first level of the two-dimensional mixed layer and the second level of the three-dimensional mixed layer are determined, so as to determine the order of the two-dimensional mixed layer and the three-dimensional mixed layer, and construct a two-dimensional and three-dimensional mixed UI interface in this order. For example, when 3D and 2D icons / content are presented at the same time, layers are formed according to the depth of field, and 3D icons / content float on top of 2D icons / content. When multimodal interactive 2D and 3D content are superimposed and exist at the same time, the upper layer information shall prevail, and content that is close to the interaction shall be given priority.

[0055] In a possible implementation, controlling the multimodal display module to display the UI interface corresponding to the target external device may be performed according to the following steps: displaying a three-dimensional UI interface, displaying a two-dimensional UI interface, and / or displaying a two-dimensional and three-dimensional mixed UI interface.

[0056] In this possible implementation, see Figure 5 The diagram of various display modes shown in the figure shows that the full-screen 3D mode is an example of displaying a 3D UI interface; the full-screen 2D mode shown in the figure is an example of displaying a 2D UI interface; the local 3D mode and the spatial floating window 2D mode are examples of displaying a 2D and 3D hybrid UI interface. Figure 6 The schematic diagram of a floating window effect shown is also an example of a two-dimensional three-dimensional hybrid UI interface.

[0057] It should be noted that in the display module, only the three-dimensional UI interface, only the two-dimensional UI interface, or only the two-dimensional and three-dimensional mixed UI interface can be displayed. In the reality module, different display areas can be used to simultaneously display one or more of the three-dimensional UI interface, the two-dimensional UI interface, and the two-dimensional and three-dimensional mixed UI interface.

[0058] In a possible implementation, displaying a two-dimensional and three-dimensional hybrid UI interface may be performed according to the following steps: determining a first display layer and a second display layer in the two-dimensional and three-dimensional hybrid data; wherein a depth value of the first display layer is greater than a depth value of the second display layer; when displaying the second display layer, displaying the first display layer in a designated window; wherein the designated window and the second display layer have the same display area.

[0059] In this possible implementation, the depth value of the display layer determines the stacking order of elements in the three-dimensional space. The larger the depth value, the closer the display layer is to the observer and the higher it is in the upper layer; the smaller the depth value, the farther the display layer is from the observer and the lower it is in the lower layer. The first display layer closer to the observer is displayed on top of the second display layer. The first display layer can be displayed in the form of a window in the display area of ​​the second display layer. For details, please refer to Figure 5 In the local 3D display form, the first display layer can also be presented in the form of a window floating on the screen (the floating screen height is ≤ 10mm, which is convenient for interaction). For details, please refer to Figure 5 The display form of the floating window in the middle space or Figure 6 Display format.

[0060] When the first display layer can be displayed in the display area of ​​the second display layer in the form of a window, the same display area where the window and the second display layer exist is the area of ​​the window. When the first display layer is presented in the form of a window floating on the screen, the same display area where the window and the second display layer exist is the area where the window and the second display layer intersect. That is, the specified window and the second display layer have the same display area. It should be noted that the display area refers to the range of coordinates of the area used to display content determined in the display screen.

[0061] It should be noted that the first display layer can be used to display two-dimensional content as well as three-dimensional content.

[0062] In a possible implementation, according to the display instruction, controlling the multimodal display module to display the UI interface corresponding to the target external device can be performed according to the following steps: according to the display instruction, controlling the multimodal display module to switch from displaying the UI interface currently corresponding to the target external device to displaying the specified UI interface of the target external device.

[0063] In this possible implementation, the UI interface currently corresponding to the target external device can be used to display two-dimensional content or three-dimensional content, and the designated UI interface of the target external device can be used to display three-dimensional or two-dimensional content. In this step, according to the display instruction, the multimodal display module is controlled to switch from displaying the UI interface currently corresponding to the target external device to displaying the designated UI interface of the target external device, so that real-time switching between 2D and 3D display modes can be achieved.

[0064] When 3D content is switched to 2D content, it is switched to 2D through the content source. The content source supports, but is not limited to, sideByside (side-by-side comparison), lineByline (line-by-line comparison), updown (up-down comparison) and multi-view (multi-view comparison) formats; see Fig. 9 The schematic diagram of the process of converting 3D content into 2D content based on two-view 3D data is shown. For two-view 3D data (2View), the left view or right view of the 3D content can be processed by difference to obtain 2D content. Fig.10 The schematic diagram of the process of converting 3D content into 2D content based on multi-view 3D data is shown. For multi-view 3D data (multi-View), any one of View1 to Viewn in the 3D content can be converted into 2D content.

[0065] When 2D content is switched to 3D content, it supports converting 2D content to 3D content, and the format is not limited to sideByside, lineByline, updown, and multi-View. Specifically, it can be the reverse operation of switching 3D content to 2D content, which will not be repeated here.

[0066] In a possible implementation, the multimodal display contents are adjusted respectively according to a preset data format to obtain an adjustment result, which can be performed according to the following steps: respectively determining the content type of the multimodal display contents; the content type is used to determine the data dimension of the external device; respectively determining the resolution and frame rate of the multimodal display contents; respectively determining the coordinates of the three-dimensional area of ​​the multimodal display contents; and using the content type, the resolution and the frame rate, the coordinates of the three-dimensional area, and the value of the display content corresponding to the designated external device as the adjustment result of the display content of the designated external device.

[0067] In this possible implementation, the preset data format can be used as a transmission interface protocol, see Figure 4 A schematic diagram of a data transmission protocol is shown, where the transmission interface protocol specifically includes a content type area, a content basic information area, a 3D area area and a data area.

[0068] Determine the content type of the multimodal display content respectively, that is, determine the corresponding content type for the multimodal display content of each device. The content type is used to determine the data dimension of the external device. For example, the content type area uses data m to represent the type of content to be transmitted, which is used to identify whether the content to be transmitted is 2D, 3D, or 2D and 3D fusion content. m is not limited to distinguishing numbers, letters and combinations thereof, such as m=0 represents 2D content, m=1 represents 3D content, and m=2 represents 2D and 3D fusion content.

[0069] The resolution and frame rate of the multimodal display content are determined respectively, that is, the corresponding resolution and frame rate are determined for the multimodal display content of each device. Among them, the resolution is the number of pixels in the image, which can be expressed as the number of pixels of width and height, and is used to describe the clarity and fineness of the image. The frame rate is used to describe the number of image frames displayed per second. The higher the frame rate, the smoother and more fluent the visual experience. The resolution and frame rate can be used as data in the basic information area of ​​the content. For example, the content information area identifies the resolution size (w0, h0) and frame rate fps of the content image to be transmitted.

[0070] The coordinates of the three-dimensional area of ​​the multimodal display content are determined respectively, that is, the coordinates of the corresponding three-dimensional area are determined for the multimodal display content of each device. For example, the 3D area identifies the range of the transmitted 3D content. If the content type is 2D content, the 3D content area (x, y, w0, h0) = (0, 0, 0, 0), if the content type is 3D content, the 3D content area (x, y, w1, h1) = (0, 0, w0, h0), if the content type is 2D & 3D fusion content, then the 3D content area (x, y, w1, h1) is determined according to the actual 3D area position, where (x, y) is the coordinate of the upper left corner of the 3D content area, and (w1, h1) is the width and height of the 3D content area.

[0071] The value of the displayed content is the data stored in the data area. For example, when the multimodal displayed content is a picture, the value of the displayed content may be a pixel value.

[0072] In this step, the part that needs 3D display can be specially marked in the data format. After the mark is recognized, the content that needs 3D display is processed and merged with the 2D display content, and then transmitted to the light field 3D display screen through the SOC chip. The 2D content and 3D content transmission interface protocol (distinguishing between 2D, 3D, and 2D & 3D fusion) creates the system UI interface according to the type of display content. In this step, various information sources are standardized.

[0073] In a possible implementation, receiving a display instruction from a target external device among the multiple external devices can be performed according to the following steps: determining an interactive device according to the multimodal display content; and upon receiving a display instruction sent by the interactive device for controlling the target external device, determining that a display instruction from the target external device among the multiple external devices is received.

[0074] In this possible implementation, a variety of interactive devices can be compatible, including but not limited to mouse, keyboard, gesture, eye movement, voice, 3D AI assistant, etc. Determining the interactive device according to the multimodal display content can be performed according to the following steps: First, identify different multimodal display contents, such as but not limited to: 2D content, 3D content, 2D and 3D fusion content, and then call the interactive device according to the identified different forms of content, for example: when the content source is detected to be 2D, the priority interactive device is preferably mouse, keyboard, touch, and active pen; when the content source is detected to be 3D, the interactive device is preferably gesture, eye movement, and voice; when the content source is detected to be a fusion of 2D and 3D, the preferred interaction method is 3D AI assistant, which can automatically identify the 2D and 3D areas when the content is fused, and automatically recommend the interaction method according to the area to be interacted.

[0075] The interactive device is communicatively connected with the external device, and the interactive device sends a control instruction to the external device. The external device generates a display instruction based on the control instruction of the interactive device, and the external device then sends the display instruction to the execution subject of the embodiment of the present application. At this time, the external device, that is, the target external device, that is, the execution subject, receives the display instruction sent by the interactive device for controlling the target external device, and determines that the display instruction of the target external device among the multiple external devices is received.

[0076] In this step, information collection and localization models are supported. According to different types of display content (2D and / or 3D), the operating system calls different interaction methods, such as 2D using touch and active pen interaction, and 3D using gestures or eye tracking. Cross-platform interactive operations can be realized, and the operation commands on the external device are projected to this device for execution. It should be noted that the interface transmitted by the external device can also be operated directly on this device, and the embodiments of the present application will not go into details. In a possible implementation, when multimodal interactive 2D content and 3D content exist at the same time, the upper layer information shall prevail, and the closer one shall take priority.

[0077] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a multimodal content display device. Fig.11 FIG. 1 is a block diagram of a multimodal content display device according to an embodiment of the present application. The device may include:

[0078] The acquisition module 1101 is used to acquire multimodal display contents of multiple external devices, and adjust the multimodal display contents respectively according to a preset data format to obtain adjustment results; the interface module 1102 is used to create UI interfaces corresponding to the multiple external devices respectively according to the adjustment results; the control module 1103 is used to respond to receiving a display instruction of a target external device among the multiple external devices, and control the multimodal display module to display the UI interface corresponding to the target external device according to the display instruction.

[0079] Based on the present application, an acquisition module is used to acquire multimodal display contents of multiple external devices, and adjust the multimodal display contents respectively according to a preset data format. Therefore, the multimodal display contents can be adjusted according to the preset data format, so that when the operating systems of the multiple external devices are different, the data format can be unified, which facilitates information transmission and display content synchronization between devices across operating systems and improves the efficiency of multi-device collaborative work. In addition, an interface module is used to create UI interfaces corresponding to the multiple external devices respectively according to the adjustment results; a control module is used to respond to a display instruction received from a target external device among the multiple external devices, and according to the display instruction, control the multimodal display module to display the UI interface corresponding to the target external device, so that the display of multimodal content can be compatible with the same time, meeting the diverse needs of users in different scenarios.

[0080] In a possible implementation, the multimodal display content includes two-dimensional data, three-dimensional data, and two-dimensional and three-dimensional mixed data; UI interfaces corresponding to the multiple external devices are created respectively according to the adjustment results, including: generating a two-dimensional UI interface for the two-dimensional data, and generating a three-dimensional UI interface for the three-dimensional data; generating a two-dimensional mixed layer and a three-dimensional mixed layer corresponding to the two-dimensional and three-dimensional mixed data, and generating a two-dimensional and three-dimensional mixed UI interface according to the two-dimensional mixed layer and the three-dimensional mixed layer.

[0081] In a possible implementation, a two-dimensional and three-dimensional hybrid UI interface is generated based on the two-dimensional hybrid layer and the three-dimensional hybrid layer, including: determining a first level of the two-dimensional hybrid layer and a second level of the three-dimensional hybrid layer based on depth data of the two-dimensional hybrid layer and depth data of the three-dimensional hybrid layer; determining an order of the two-dimensional hybrid layer and the three-dimensional hybrid layer according to the first level and the second level, and constructing a two-dimensional and three-dimensional hybrid UI interface.

[0082] In a possible implementation, the multimodal display module is controlled to display a UI interface corresponding to the target external device, including: displaying a three-dimensional UI interface, displaying a two-dimensional UI interface, and / or displaying a two-dimensional and three-dimensional mixed UI interface.

[0083] In one possible implementation, displaying a two-dimensional three-dimensional hybrid UI interface includes: determining a first display layer and a second display layer in the two-dimensional three-dimensional hybrid data; wherein a depth value of the first display layer is greater than a depth value of the second display layer; when displaying the second display layer, displaying the first display layer in a designated window; wherein the designated window and the second display layer have the same display area.

[0084] In one possible implementation, according to the display instruction, the multimodal display module is controlled to display the UI interface corresponding to the target external device, including: according to the display instruction, the multimodal display module is controlled to switch from displaying the UI interface currently corresponding to the target external device to displaying the specified UI interface of the target external device.

[0085] In a possible implementation, the multimodal display content is adjusted respectively according to a preset data format to obtain an adjustment result, including: respectively determining the content type of the multimodal display content; the content type is used to determine the data dimension of the external device; respectively determining the resolution and frame rate of the multimodal display content; respectively determining the coordinates of the three-dimensional area of ​​the multimodal display content; and using the content type, the resolution and the frame rate, the coordinates of the three-dimensional area, and the value of the displayed content corresponding to the specified external device as the adjustment result of the display content of the specified external device.

[0086] In a possible implementation, receiving a display instruction of a target external device among the multiple external devices includes: determining an interactive device according to the multimodal display content; and determining that a display instruction of the target external device among the multiple external devices is received when receiving a display instruction sent by the interactive device for controlling the target external device.

[0087] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0088] Fig.12 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Fig.12 As shown, the electronic device includes: a memory 1201 and a processor 1202. The memory 1201 stores a computer program that can be run on the processor 1202. When the processor 1202 executes the computer program, the method in the above embodiment is implemented. The number of the memory 1201 and the processor 1202 can be one or more.

[0089] The electronic device also includes:

[0090] The communication interface 1203 is used to communicate with external devices and perform data exchange transmission.

[0091] If the memory 1201, the processor 1202 and the communication interface 1203 are implemented independently, the memory 1201, the processor 1202 and the communication interface 1203 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0092] Optionally, in a specific implementation, if the memory 1201, the processor 1202 and the communication interface 1203 are integrated on a chip, the memory 1201, the processor 1202 and the communication interface 1203 can communicate with each other through an internal interface.

[0093] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0094] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0095] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0096] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0097] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0098] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0099] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0100] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0101] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0102] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0103] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0104] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0105] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0106] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0107] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A multimodal content display method, comprising: Acquire multimodal display contents of multiple external devices, and adjust the multimodal display contents respectively according to a preset data format to obtain adjustment results; Creating UI interfaces corresponding to the multiple external devices respectively according to the adjustment results; In response to receiving a display instruction of a target external device among the multiple external devices, according to the display instruction, the multimodal display module is controlled to display a UI interface corresponding to the target external device.

2. The method according to claim 1, wherein: The multimodal display content includes two-dimensional data, three-dimensional data, and two-dimensional and three-dimensional mixed data; Creating UI interfaces corresponding to the multiple external devices respectively according to the adjustment results includes: generating a two-dimensional UI interface for the two-dimensional data, and generating a three-dimensional UI interface for the three-dimensional data; A two-dimensional mixed layer and a three-dimensional mixed layer corresponding to the two-dimensional and three-dimensional mixed data are generated, and a two-dimensional and three-dimensional mixed UI interface is generated according to the two-dimensional mixed layer and the three-dimensional mixed layer.

3. The method according to claim 2, wherein: Generating a two-dimensional and three-dimensional hybrid UI interface according to the two-dimensional hybrid layer and the three-dimensional hybrid layer includes: Determining a first level of the two-dimensional mixed layer and a second level of the three-dimensional mixed layer according to the depth data of the two-dimensional mixed layer and the depth data of the three-dimensional mixed layer; According to the first level and the second level, the order of the two-dimensional hybrid layer and the three-dimensional hybrid layer is determined to construct a two-dimensional and three-dimensional hybrid UI interface.

4. The method according to claim 2, wherein: Controlling the multimodal display module to display the UI interface corresponding to the target external device includes: Display a three-dimensional UI interface, display a two-dimensional UI interface, and / or display a two-dimensional and three-dimensional mixed UI interface.

5. The method according to claim 4, wherein: Displays a 2D and 3D hybrid UI interface, including: Determining a first display layer and a second display layer in the two-dimensional and three-dimensional mixed data; wherein a depth value of the first display layer is greater than a depth value of the second display layer; When the second display layer is displayed, the first display layer is displayed in a designated window; wherein the designated window and the second display layer have the same display area.

6. The method according to claim 1, wherein: According to the display instruction, controlling the multimodal display module to display the UI interface corresponding to the target external device includes: According to the display instruction, the multimodal display module is controlled to switch from displaying the UI interface currently corresponding to the target external device to displaying the designated UI interface of the target external device.

7. The method according to claim 1, wherein: The multimodal display contents are adjusted respectively according to a preset data format to obtain adjustment results, including: respectively determining the content type of the multimodal display content; the content type is used to determine the data dimension of the external device; respectively determining a resolution and a frame rate of the multimodal display content; respectively determining coordinates of three-dimensional regions of the multimodal display content; The content type, the resolution and the frame rate, the coordinates of the three-dimensional area and the value of the displayed content corresponding to the designated external device are used as the adjustment result of the displayed content of the designated external device.

8. The method according to claim 1, wherein: Receiving a display instruction from a target external device among the multiple external devices includes: Determining an interactive device according to the multimodal display content; When a display instruction for controlling a target external device sent by the interactive device is received, it is determined that a display instruction of a target external device among the multiple external devices is received.

9. A multimodal data display device, comprising: An acquisition module, used for acquiring multimodal display contents of multiple external devices, adjusting the multimodal display contents respectively according to a preset data format, and obtaining adjustment results; An interface module, used to create UI interfaces corresponding to the multiple external devices respectively according to the adjustment results; The control module is used to control the multimodal display module to display a UI interface corresponding to the target external device in response to receiving a display instruction of the target external device among the multiple external devices according to the display instruction.

10. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images