A multi-convolution visual bearing interactive method and device

Abstract

The embodiment of the present application provides a kind of multi complex visual bearing interactive method and device, the method includes: according to user demand, obtain two-dimensional image and identify the two-dimensional information of two-dimensional image;Three-dimensional information is obtained by the serialization neural network processing to the two-dimensional information, and the three-dimensional information includes three-dimensional visual sequence information, three-dimensional interaction information, three-dimensional light field information, three-dimensional sound field information and three-dimensional space information;Three-dimensional change data is obtained by analog operation and operation in the same virtual three-dimensional space to the three-dimensional information;Dynamic virtual three-dimensional space is obtained by dynamic deduction to virtual three-dimensional space according to the three-dimensional change data;According to the user demand, the dynamic virtual three-dimensional space is rendered as two-dimensional material and is displayed in real time.This method realizes the efficient interaction of two-dimensional image and three-dimensional space.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and video game technology and Unreal Engine film production, and specifically to a method and apparatus for multi-dimensional visual interactive experiences. Background Technology

[0002] Interactive games via mobile electronic devices are becoming increasingly popular. Some interactive games (such as mobile games, PC games, offline games, and online games) can be downloaded to mobile devices from remote servers, allowing users to choose their game content. During gameplay, users can interact with virtual characters on the mobile device's screen using one or more input buttons. For example, games like Cities: Skylines and No Man's Sky create virtual characters to interact with the virtual world, but this method offers limited interactivity. Other games rely on gesture recognition for control. For instance, gestures simulate actions to elicit interactions from virtual characters. The interaction path is stored in a memory location; the interaction path is selected based on the amplitude and direction of the recognized action. While this method increases interactivity, it suffers from low flexibility and accuracy. Summary of the Invention

[0003] The purpose of this invention is to provide a multi-dimensional visual interactive method and apparatus, which enables efficient interaction between two-dimensional images and three-dimensional space.

[0004] To achieve the above objectives, embodiments of the present invention provide a multi-dimensional visual interactive method, comprising: acquiring a two-dimensional image and identifying two-dimensional information of the two-dimensional image according to user needs; performing sequential neural network processing on the two-dimensional information to obtain three-dimensional information, the three-dimensional information including three-dimensional visualization sequence information, three-dimensional interactive information, three-dimensional light field information, three-dimensional sound field information, and three-dimensional spatial information; simulating and running the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data; dynamically interpreting the virtual three-dimensional space according to the three-dimensional change data to obtain a dynamic virtual three-dimensional space; and rendering the dynamic virtual three-dimensional space as two-dimensional material for real-time display according to the user needs.

[0005] Optionally, the sequenced neural network is a recurrent neural network, and the neurons of the recurrent neural network have memory; the recurrent neural network is used to predict time series sequences.

[0006] Optionally, the step of simulating and running the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data includes: simulating the actions of three-dimensional entities based on the three-dimensional visualization sequence information and three-dimensional spatial information, synchronizing the actions of the three-dimensional entities through pixel spatial positioning to obtain three-dimensional entity action data; simulating the physical collisions generated by the three-dimensional entities in the scene space based on the three-dimensional interaction information to obtain three-dimensional entity collision data; identifying the material of the three-dimensional entities based on the three-dimensional light field information to obtain three-dimensional entity material data; and simulating the changes in the three-dimensional sound field based on the three-dimensional sound field information to obtain three-dimensional sound field data; the three-dimensional change data includes the three-dimensional entity action data, three-dimensional entity collision data, three-dimensional entity material data, and three-dimensional sound field data.

[0007] Optionally, the step of dynamically deducing the virtual three-dimensional space based on the three-dimensional change data to obtain the dynamic virtual three-dimensional space includes: identifying continuous images of the real space, establishing a three-dimensional spatial information structure, and incorporating the three-dimensional change data into the three-dimensional spatial information structure to obtain the dynamic virtual three-dimensional space.

[0008] Optionally, the method further includes: setting a three-dimensional simulation entity and projecting the three-dimensional simulation entity into the dynamic virtual three-dimensional space, wherein the dynamic virtual three-dimensional space changes as the three-dimensional simulation entity changes.

[0009] Optionally, the three-dimensional spatial information is at least one of spatial positioning information, spatial structure information, spatial stability information, spatial change information, and physical feedback information.

[0010] On the other hand, the present invention provides a multi-dimensional visual interactive device, which includes: an acquisition unit for acquiring a two-dimensional image and recognizing two-dimensional information of the two-dimensional image according to user needs; a first processing unit for performing sequential neural network processing on the two-dimensional information to obtain three-dimensional information, the three-dimensional information including three-dimensional visualization sequence information, three-dimensional interactive information, three-dimensional light field information, three-dimensional sound field information, and three-dimensional spatial information; a second processing unit for simulating and running the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data; a third processing unit for dynamically interpreting the virtual three-dimensional space according to the three-dimensional change data to obtain a dynamic virtual three-dimensional space; and a fourth processing unit for rendering the dynamic virtual three-dimensional space into two-dimensional material for real-time display according to user needs.

[0011] Optionally, the sequenced neural network is a recurrent neural network, and the neurons of the recurrent neural network have memory; the recurrent neural network is used to predict time series sequences.

[0012] Optionally, the step of simulating and running the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data includes: simulating the actions of three-dimensional entities based on the three-dimensional visualization sequence information and three-dimensional spatial information, synchronizing the actions of the three-dimensional entities through pixel spatial positioning to obtain three-dimensional entity action data; simulating the physical collisions generated by the three-dimensional entities in the scene space based on the three-dimensional interaction information to obtain three-dimensional entity collision data; identifying the material of the three-dimensional entities based on the three-dimensional light field information to obtain three-dimensional entity material data; and simulating the changes in the three-dimensional sound field based on the three-dimensional sound field information to obtain three-dimensional sound field data; the three-dimensional change data includes the three-dimensional entity action data, three-dimensional entity collision data, three-dimensional entity material data, and three-dimensional sound field data.

[0013] Optionally, the device further includes: a setting unit for setting a three-dimensional simulation entity and projecting the three-dimensional simulation entity into the dynamic virtual three-dimensional space, wherein the dynamic virtual three-dimensional space changes as the three-dimensional simulation entity changes.

[0014] The multi-dimensional visual interactive method of the present invention includes: acquiring a two-dimensional image and identifying the two-dimensional information of the image according to user needs; performing sequential neural network processing on the two-dimensional information to obtain three-dimensional information, wherein the three-dimensional information includes three-dimensional visualization sequence information, three-dimensional interactive information, three-dimensional light field information, three-dimensional sound field information, and three-dimensional spatial information; simulating and running the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data; dynamically interpreting the virtual three-dimensional space based on the three-dimensional change data to obtain a dynamic virtual three-dimensional space; and rendering the dynamic virtual three-dimensional space as two-dimensional material for real-time display according to user needs. This method can switch display modes in real time according to customer needs, realizing real-time communication and interaction between two-dimensional graphics and the three-dimensional virtual world.

[0015] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a multi-layered visual interactive method according to the present invention; Figure 2 This is a schematic diagram of an embodiment of a multi-complex visual interactive method according to the present invention; Figure 3 This is a schematic diagram of the visual sequence simulation prediction of the present invention; Figure 4This is a schematic diagram of the simulated three-dimensional entity of the present invention; Figure 5 This is a schematic diagram of pixel spatial positioning according to the present invention; Figure 6 This is a schematic diagram of the three-dimensional spatial vector field of the present invention; Figure 7 This is a schematic diagram of real-time asset management according to the present invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0018] Figure 1 This is a flowchart illustrating a multi-layered visual interactive method according to the present invention, as shown below. Figure 1 As shown, the multi-dimensional visual interactive method includes: Step S101, which involves acquiring a two-dimensional image and recognizing its two-dimensional information according to user needs. The two-dimensional image can be a human figure; for example, this method can place a real person into a virtual world and allow them to interact with the virtual world, creating interactive activities similar to a game. Users can edit three-dimensional spatial assets and models as needed, and can edit and change the three-dimensional space in real time. Users can select the interaction mode according to their needs. Users can randomly switch display modes, and the system generates the final effect in real time.

[0019] Step S102 involves processing the two-dimensional information using a serialization neural network to obtain three-dimensional information. The three-dimensional information includes three-dimensional visual sequence information, three-dimensional interactive information, three-dimensional light field information, three-dimensional sound field information, and three-dimensional spatial information. Specifically, the serialization neural network is a recurrent neural network, and the neurons in the recurrent neural network have memory; the recurrent neural network is used to predict temporal sequences. Specifically, the serialization neural network can convert two-dimensional information such as visual sequence information, light field information, interactive information, spatial information, and sound field information into three-dimensional information.

[0020] In one specific implementation, the 3D visualization sequence information is the visualization sequence information captured by a camera. The visualization sequence information refers to the 2D information of a continuous 2D image, which is then converted into 3D information in real time by a serialization neural network. The visualization sequence information includes 3D information such as the depth, material, and texture of the simulated 3D entity. Specifically, it involves selecting, creating, sending, forwarding, receiving, viewing, and destroying 3D entities, and using the visualization information to observe the movement of entities between cloud interactive visualization modules during the simulation of the 3D entity model. The visualization sequence information displays the simulated 3D entities according to timecodes, and multiple simulated 3D entity display events can occur simultaneously. The visualization sequence can display time linearly and non-linearly. The time grid displays time in a non-linear manner, with each time grid containing events occurring at the same simulation time. The timecode represents the time of the simulated 3D entity events.

[0021] Based on the dynamic movement between simulated 3D entities, the physical collisions they generate in 3D space are simulated, and the interaction information between the simulated 3D entities and various 3D models in 3D space is obtained as the 3D interaction information. The path to obtain the 3D interaction information is as follows: 3D information source, 3D information, channel or network for transmitting 3D information, receiving 3D information, feedback 3D information from the receiving end, and generated noise. The feedback from the receiving end is used to verify the quality of the 3D information interaction result.

[0022] The three-dimensional light field information is the light image information of the identified continuous image. The method for obtaining the three-dimensional light field information includes the spatial light field reconstruction technology of material identification of simulated three-dimensional entities, and the serialized neural network processing unit first converts the two-dimensional information into three-dimensional information in real time.

[0023] Visualizing sequence information extends the laws of light fields in the real world to the virtual world. Deep learning methods are used to solve the problem of extracting and fusing light field structural features through a precise fusion network. By utilizing the novel features of light field information, simulated 3D entities from the real world are placed into the virtual world, so that as the simulated 3D entities dynamically change, the virtual light field space simulates the light field response of the real world.

[0024] Light field information is a dynamic simulation of the spatial information of the light field, scaled proportionally to the passage of time in the real world. For example, the light field space in the morning, at sunset, and on sunny or rainy days changes accordingly with time and weather in three-dimensional space. Physical reactions such as light wave reflection, refraction, interference, and diffraction generate interactive information that dynamically changes, causing the light field throughout the three-dimensional space to change as well. Light field information is primarily used to achieve ray tracing, positioning, and simulation functions.

[0025] In the real world, sounds from a distance are perceived as softer, while sounds from close range at the same volume are perceived differently. This invention uses sound field information to simulate changes in a three-dimensional sound field. For example, in the real world, sounds are louder when closer to a sound source and softer when farther away. This invention recreates these changes in a three-dimensional sound field. A real person uses video and real-time voice input from the user, along with motion capture from a three-dimensional visualization sequence, to project the user's expressions and movements onto a simulated three-dimensional entity (virtual digital human), enabling interaction with the user. An intelligent system automatically reads, analyzes, and identifies external input information, determining the digital human's subsequent output data based on the analysis results. This drives the simulated three-dimensional entity model to generate corresponding voice and actions, allowing the digital human to interact with the user and synthesize audio and video for presentation.

[0026] The three-dimensional spatial information refers to both real-world and virtual-world spatial information. This spatial information includes spatial positioning information, spatial structure information, information ensuring spatial stability, spatial change information, and physical feedback information. Acquiring this spatial information is a crucial component of modeling and simulating three-dimensional entities. To address the complexity of modeling and simulating three-dimensional entities and to adapt spatial information to the characteristics of space, spatial information acquisition primarily involves recognizing continuous images and establishing the spatial information structure of the real three-dimensional world.

[0027] Step S103 involves simulating and running the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data. Specifically, simulating and running the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data includes: simulating the actions of three-dimensional entities based on the three-dimensional visualization sequence information and three-dimensional spatial information, synchronizing the actions of the three-dimensional entities through pixel spatial positioning to obtain three-dimensional entity action data; simulating the physical collisions generated by the three-dimensional entities in the scene space based on the three-dimensional interaction information to obtain three-dimensional entity collision data; identifying the material of the three-dimensional entities based on the three-dimensional light field information to obtain three-dimensional entity material data; and simulating the changes in the three-dimensional sound field based on the three-dimensional sound field information to obtain three-dimensional sound field data. The three-dimensional change data includes the three-dimensional entity action data, three-dimensional entity collision data, three-dimensional entity material data, and three-dimensional sound field data.

[0028] Step S104 involves dynamically deducing the virtual three-dimensional space based on the three-dimensional change data to obtain a dynamic virtual three-dimensional space. Specifically, this step includes: identifying continuous images of the real space, establishing a three-dimensional spatial information structure, and inputting the three-dimensional change data into the three-dimensional spatial information structure to obtain the dynamic virtual three-dimensional space.

[0029] Step S105 involves rendering the dynamic virtual 3D space into 2D material for real-time display according to the user's requirements. Specifically, the user can choose to bind a 3D simulation entity to achieve synchronization with a real person, with the 2D display effect changing in real time. Alternatively, the user can choose to operate the computer to control the movement of the 3D simulation entity, thereby achieving real-time changes in the 2D display effect.

[0030] This invention allows real people to interact with a virtual world, creating game-like interactive activities. Users can edit 3D spatial assets and models as needed, and edit and change the 3D space in real time. Users can also choose the interaction mode according to their needs. The interaction modes include a first display mode, a second display mode, a third display mode, a fourth display mode, and a fifth display mode. Spatial positioning technology, such as pixel spatial positioning, is incorporated. Pixel spatial positioning is a virtual reality / mixed reality video shooting application, applicable to various fields such as virtual production, VR, motion capture, and virtual pre-visualization.

[0031] The invention further includes: setting a three-dimensional simulation entity and projecting the three-dimensional simulation entity into the dynamic virtual three-dimensional space, wherein the dynamic virtual three-dimensional space changes as the three-dimensional simulation entity changes. This method enables real-time communication and interaction between single or multiple users in a virtual world, and allows for the generation of cinematic-effect videos based on user creativity during gameplay. These cinematic-effect videos can be generated in real-time, achieving a live streaming effect.

[0032] Figure 2 This is a schematic diagram of an embodiment of the multi-complex visual interactive method of the present invention, as shown below. Figure 2 As shown, this embodiment first acquires a two-dimensional image, recognizes the two-dimensional image, and obtains two-dimensional information. For example, there are two ways to acquire a two-dimensional image. When the user selects a first display mode, a second display mode, a fourth display mode, or a fifth display mode: the two-dimensional image is acquired by inputting the two-dimensional image into a digital engine based on the image content selected by the user, recognizing the input two-dimensional image, and classifying and transmitting the spatial information, sound field information, visual sequence information, light field information, and interactive information in the two-dimensional image in real time, and transmitting all information to the serialized neural network processing unit. When the user selects the third display mode (game-bound live-action mode), the two-dimensional image is acquired by capturing continuous two-dimensional images through a shooting system. Specifically, this method includes classifying and transmitting the spatial information, sound field information, visual sequence information, light field information, and interactive information in the two-dimensional image in real time, and transmitting all information to the serialized neural network processing unit.

[0033] The serialization neural network processes all the two-dimensional information transmitted to the serialization neural network processing unit, transforming the two-dimensional information into three-dimensional information. This invention employs the neuron structure of RNN (Recurrent Neural Networks), which is highly flexible in its use. An RNN can generate an output sequence by simulating an input sequence, thus enabling it to predict time-series sequences. Since recurrent neurons consider all previous inputs when processing the output, they can be considered to possess memory. Recurrent Neural Networks are a type of neural network capable of prediction. Unlike many other neural networks, RNNs do not require a specific input sequence length; parameter sequences of varying lengths can be fed into the RNN, such as any article or any audio / video sample. The predictive power of RNNs based on time-series data allows them to perform creative tasks such as automatic music composition and text writing. RNNs are a special type of neural network structure, proposed based on the idea that "human cognition is based on past experience and memory." Unlike DNNs and CNNs, they not only consider the input from the previous moment but also endow the network with a memory function for previous content. RNNs are called recurrent neural networks because the current output of a sequence is related to the outputs of previous sequences. Specifically, the network memorizes previous information and applies it to the calculation of the current output. That is, nodes in the hidden layers are no longer disconnected but connected, and the input to the hidden layer includes not only the output of the input layer but also the output of the hidden layer from the previous time step. Specifically, as shown in the equation: Among them, recurrent neurons receive input. And the output of this neuron at the previous moment. Each recurrent neuron has two sets of weights, namely the input and output weights. weight and the output of the previous moment Weight Output at any given time , is the activation function, and b is the bias peak. A serial neural network processing unit is used to transform two-dimensional information into three-dimensional information using an RNN processing method.

[0034] Figure 3 This is a schematic diagram of the visual sequence simulation prediction of the present invention, as shown below. Figure 3As shown, the AI ​​module transmits the different 3D information to different simulation units to obtain different 3D information, and then transmits it to the visualization sequence information simulation unit, interactive information simulation unit, light field information simulation unit, sound field information simulation unit, and spatial information simulation system respectively, so as to perform simulation calculations and operations on different types of 3D information in a unified 3D space.

[0035] In this invention, the cloud-interactive visualization system refers to the system that allows users to access different operational content and select different content based on their operational permissions. Specifically, it refers to the system that allows users to modify content in the real-time asset management system and select different playback and display modes. When the user selects the third display mode, the simulated 3D entity is generated in real time based on images of the real user captured by the camera system, creating a one-to-one replica of the simulated 3D entity. The visualization sequence information simulation unit simulates the actions of the real user in real time, achieving synchronization between the actions of the real user and the simulated 3D entity. The method for achieving action synchronization is pixel spatial positioning.

[0036] The third display mode is an interactive game mode. Specifically, it projects the user's desired character into a three-dimensional space, allowing the user to project themselves into a virtual scene. The camera system captures the user's movements, placing the real user into the three-dimensional space. Based on the user's movements, the simulated three-dimensional entity in the three-dimensional space changes in real time. For example, when a real user plays a game, controlling the virtual character's movements, the scene dynamically changes with the virtual character, as the virtual character manipulates the three-dimensional space. In this invention, when the user's movements change, a simulated three-dimensional entity of the user is generated in real time in the three-dimensional space; changes in the user's movements alter the three-dimensional scene. The interactive game display converts the dynamically changing three-dimensional space into a two-dimensional image and displays it in real time. Users can choose to bind the real user to a simulated three-dimensional entity, with the three-dimensional scene dynamically changing as the user's movements change. Alternatively, they can choose not to bind a simulated three-dimensional entity, allowing them to manipulate the simulated entity, and the three-dimensional scene will change accordingly.

[0037] like Figure 4As shown, a simulated 3D entity model is created in real time based on an image. Specifically, images, videos, or camera systems can be used to capture images and generate 3D simulated entities in real time, projecting the user's real-life image into a virtual 3D space. Once the user system receives the user's selected display mode, the cloud interactive visualization system controls the operation of each unit in the digital interface, real-time rendering, and the display effects of the interactive playback system. When the game mode is selected and the simulated 3D entity is bound, the camera system needs to be activated to capture images of the real user in real time. The serialization neural processing unit acquires the user's visualization sequence and predicts the user's upcoming actions, converting 2D information into 3D information. The serialization neural network processing unit transmits the obtained 3D information to the visualization sequence simulation unit, which reconstructs the simulated 3D entity proportionally, binds the simulated 3D entity to the user, and projects the simulated 3D entity into a virtual scene space or a desired replacement 3D scene space.

[0038] like Figure 5 As shown, the simulated 3D entity in the 3D space will synchronize its movements in real time with the changes in the actions of a real user; this process is called gait synchronization. A visual sequence of the real 3D space is acquired, and spatial information, sound field information, light field information, interaction information, and image data information from the real 3D space are recorded into the 3D digital asset world. This essentially converts people or objects in the real 3D world into 3D digital assets. Its principle mainly utilizes various technologies such as spatial recognition, spatial reconstruction, and spatial positioning. This achieves the projection of a real person into virtual space, realizing the combination of virtual and reality. Digital humans possess human appearance, with specific facial features, gender, and personality characteristics; they possess human behavior, the ability to express themselves through language, facial expressions, and body movements; they possess human thought, the ability to recognize the external environment, and the ability to communicate and interact with others. Each simulated 3D entity projected from a real person into virtual space is a unique entity.

[0039] Pixel spatial positioning is a virtual reality / mixed reality video shooting application, applicable to various fields such as virtual production, VR, motion capture, and virtual pre-visualization. Addressing the challenges of pixel spatial positioning technology, the camera system first constrains the mapping relationship between the target pixels of the target object and the spatial environment, based on a digital surface model and a target positioning algorithm based on planar constraints. The camera system first calibrates the target pixels, then extracts the corrected target pixels, and finally achieves camera positioning and tracking of the target pixels by locating the target pixels and calculating the target's three-dimensional spatial coordinates using the algorithm. The spatial mapping relationship between the three-dimensional space and the simulated three-dimensional entity is consistent with the mapping in the real world.

[0040] The visualization sequence information simulation unit acquires and stores the visualization sequence information captured by the camera. Visualization sequence information refers to the two-dimensional information of a continuous two-dimensional image. The serialization neural network processing unit first converts the two-dimensional information into three-dimensional information in real time. The visualization sequence information simulation unit acquires three-dimensional information such as the depth, material, and texture of the simulated three-dimensional entity. It allows for the selection, creation, sending, forwarding, receiving, viewing, and destruction of three-dimensional entities. Viewing the visualization information enables movement of entities between cloud-interactive visualization modules while simulating the three-dimensional entity model. All visualization sequence information that can store the simulated three-dimensional entity model is viewed in the visualization sequence information simulation unit. The visualization sequence information is viewed by displaying the simulated three-dimensional entity according to a timecode. Multiple simulated three-dimensional entity display events can occur simultaneously. The visualization sequence can display time linearly and non-linearly. The time grid displays time in a non-linear manner, with each time grid containing events occurring at the same simulation time. The timecode provides the time of the simulated three-dimensional entity display event.

[0041] The serialized neural network processing unit in this invention refers to the process of receiving and sending three-dimensional interactive information. Specifically, based on the dynamically changing movement between simulated three-dimensional entities, it simulates the physical collisions they generate in three-dimensional space, obtaining the interactive information between the simulated three-dimensional entities and each three-dimensional model in the three-dimensional space. The three-dimensional information interaction process typically consists of six parts: a three-dimensional information source, three-dimensional information, a channel or network for transmitting three-dimensional information, receiving three-dimensional information, feedback three-dimensional information from the receiving end, and generated noise. The feedback from the receiving end is used to verify the quality of the three-dimensional information interaction result.

[0042] In three-dimensional space, the simulation of three-dimensional entities and all three-dimensional digital models in the space generates physical interactions. At this time, the digital engine stores existing three-dimensional information and performs operations such as storing, sending, transmitting, and receiving newly generated three-dimensional interaction information. For example... Figure 5 As shown, the interaction information generated by the Actor simulation 3D entity and the digital human 3D model; the interaction information generated by the light field and the Actor simulation 3D entity and the digital human 3D model; the interaction information generated by the sound field changes in 3D space; the interaction information generated by the Actor simulation 3D entity and the digital human 3D model with the entire 3D space; the generation of interaction information is inseparable from spatial positioning, which requires pixel spatial positioning technology.

[0043] The light field information simulation unit acquires the light image information of identified continuous images. This unit is a spatial light field reconstruction technology based on material recognition of simulated 3D entities. The visualization sequence information simulation unit acquires the depth, material, and texture of the simulated 3D entity to obtain 3D light field information. Specifically, the light field laws of the real world are extended to the virtual world. Using deep learning methods, a precise and fusion network is proposed to solve the problem of extracting and fusing light field structural features. By utilizing the new feature extraction of light field information, simulated 3D entities from the real world are placed into the virtual world. When the simulated 3D entities dynamically change, the virtual light field space simulates the light field response of the real world.

[0044] The light field information simulation unit dynamically simulates the spatial information of the light field proportionally to the passage of time in the real world. For example, the light field space in the morning, the light field space at sunset, and the light field space on sunny or rainy days change accordingly with changes in time and weather in three-dimensional space. Physical reactions such as light wave reflection, refraction, interference, and diffraction generate interactive information that dynamically changes, causing the light field in the entire three-dimensional space to change accordingly. The light field information simulation unit mainly realizes the functions of ray tracing positioning and simulation.

[0045] In the real world, sounds from a distance are perceived as softer when they reach the ear, while sounds from a close distance at the same volume are perceived differently. Therefore, the sound field information simulation unit acquires sound field information, and the sequential neural network processing unit first converts the two-dimensional information into three-dimensional information in real time. Then, the sound field information simulation unit simulates the changes in the three-dimensional sound field. For example, in the real world, sounds closer to the sound source are perceived as louder, while sounds farther away are perceived as softer.

[0046] This invention also discloses a method for reconstructing changes in a three-dimensional sound field. For example... Figure 5 As shown, the footsteps of the Actor and the digital human gradually change as they approach each other in 3D space, with the Actor hearing the digital human's footsteps grow louder. When the game mode captures changes in the voice of a real user, the simulated 3D entity in the game's virtual world emits the same voice as the real user, and the actions and expressions of both the real user and the simulated 3D entity are identical. The real user interacts with the user in real-time via video from the camera system, and simultaneously, through a visual sequence simulation unit motion capture system, the real user's expressions and movements are projected onto the simulated 3D entity (virtual digital human), thus interacting with the user. An intelligent system automatically reads, analyzes, and recognizes external input information, and based on the analysis results, determines the digital human's subsequent output data, driving the simulated 3D entity model to generate corresponding voice and actions to enable the digital human to interact with the user. The resulting audio and video are then synthesized and presented to the user.

[0047] The spatial information simulation system places all 3D information change data into a virtual 3D space to determine a dynamically evolving 3D space. The spatial information simulation system refers to acquiring spatial information from both the real world and the virtual world. This spatial information includes spatial positioning information, spatial structure information, information to ensure spatial stability, spatial change information, and physical feedback information. Acquiring this spatial information is a crucial component of modeling and simulating 3D entities. To cope with the complexity and spatial information involved in modeling and simulating 3D entities, the spatial information simulation system possesses strong adaptability to space. Acquiring spatial information mainly refers to recognizing continuous images and establishing the spatial information structure of the real 3D world. The spatial information generated by the movement of the simulated 3D entity model is simulated by the four components of the spatial information simulation system: a sequential neural network processing unit, a light field information simulation unit, a sound field information simulation unit, a visualization sequence information acquisition unit, an interactive information simulation unit, and a multi-complexity processing unit. Therefore, using these six units as the basis for dynamically changing information, and simultaneously changing the information within these six units and simulating the spatial information processing within the spatial information simulation system, the main framework of the 3D spatial model is determined, resulting in a dynamically evolving 3D space. Reconstructing all 3D information in the space yields the dynamic 3D space.

[0048] The multi-complexity processing unit is used for engineering calculations, such as calculating accuracy using the multi-complex Gauss-Legend integral formula. It utilizes three integrand forms: Fresnel integral, two-dimensional surface integral (surface integrals can be further divided into first-class and second-class, both in three-dimensional space), and the magnetic vector potential calculation form in linear antenna oscillator radiation problems for practical computer programming. Line integrals are restricted to two- and three-dimensional spaces, while surface integrals are restricted to three-dimensional space. Analysis is performed in three-dimensional space, not extended to n-dimensional space. Line integrals are divided into first- and second-class, and in application fields, they can be further divided into two-dimensional and three-dimensional. Surface integrals can also be divided into first- and second-class, both in three-dimensional space. Even though both are second-class integrals, there are differences. Line integrals deal with the dot product of a vector field with respect to the tangent vector, while surface integrals deal with the dot product with respect to the normal vector. Therefore, it can be predicted that there will always be some differences in their processing.

[0049] The relevant calculation principles of this invention include: 1) Fresnel Integral: Diffraction is arguably the most important research subject in the field of wave optics, represented by the Fresnel-Huygens principle. In optical research, the Fresnel diffraction integral can be used to calculate the propagation of light waves in the near-field region. Assuming a light wave is incident on an opaque baffle with an aperture, a diffraction pattern will appear on the observation screen. According to Fresnel's principle, the wave disturbance at point P on the observation screen caused by a spherical secondary wave emitted from a secondary wave source Q at any point inside the aperture... for Where λ is the wavelength and S is the integration plane (aperture). R is the wave disturbance located at point Q, R is the displacement vector from point Q to point P, R is the numerical value, K(χ) is the tilt factor, and χ is the angle between the normal vector perpendicular to the aperture plane and R.

[0050] 2) Area integral: The area integral, also called the double integral, is an extension of the definite integral of a single-variable function. Geometrically, if the definite integral of a single-variable function calculates the area between the integrand curve and the coordinate axes within a certain interval, then the double integral calculates the volume between the integrand surface and the coordinate plane within a certain two-dimensional region.

[0051] 3) Vector fields in three-dimensional space: such as Figure 6 As shown, there are three-dimensional vector fields (Figure a), three-dimensional vector fields on sliced ​​surfaces (Figure b), gradient fields, etc. Vector fields can be used to vividly depict the flow of liquids, liquids flowing according to specific rules, and the motion of particles (using arrows). They also provide an intuitive method for plotting multivariate functions. The direction of the vector represents the direction of the object's movement. In three-dimensional space, they can also simulate airflow, light fields, wind, water flow, etc.

[0052] In this invention, the digital engine is an intelligent control system, empowered by digital technology to achieve intelligent control of the system. It generates a dynamic 3D space based on the main framework of the 3D spatial model and records the 3D space into 3D materials. The cloud interactive system mode is intelligently implemented based on the real-time asset management system and the user system. A flexible and reasonable allocation method is adopted to achieve the final display effect. The data engine is a digital infrastructure used in binary data calculation tables to solve data transmission and data value information for different data center scenarios. The digital engine consists of a computer and software operating system; its principles and calculation methods are completely different from general data engines, but similar to traditional data-driven computers that integrate multiple systems such as datafication, production and processing, storage and application. In this invention, the user system can log in to the cloud interactive visualization system, and after verification by the verification system, can select the display mode and edit the displayed content through the real-time asset management system. In this invention, the real-time asset management system refers to the user system's addition, replacement, deletion, and other operations and management of all materials displayed in the video image. For example, if there is a building in the background of the video image, and the user wants to add an advertisement to the building, the user can operate the video after logging in and being approved by the verification system. Figure 7As shown, the real-time asset management system adds new elements and advertisements to a 3D space, rendering them in real time after completion. Then, the dynamically changing 3D information is converted into 2D information. Finally, the cloud-based interactive visualization system controls the digital engine to render and generate the dynamically changing 3D space in real time.

[0053] The cloud-based interactive visualization system facilitates interactive communication based on the main framework of the 3D spatial model, user needs, and real-time asset changes. In this invention, the cloud-based interactive visualization system refers to the ability to access and display different content based on the user's access permissions. Specifically, it refers to different playback and display modes selected based on the user's changes to content in the real-time asset management system and the user's login interaction system. The interactive playback system in this invention allows users to select from a first, second, third, fourth, and fifth display modes according to their needs.

[0054] Specifically, the first display mode refers to VR (Virtual Reality). VR panorama, as a new type of visual display technology, has strong immersion and integration, which can greatly improve the communication effect between products and users. The final effect can be placed into the scene for display, and customers wearing VR devices can experience it firsthand, making it more interesting and thus improving service quality and effect. In this invention, the second display mode refers to video viewing mode, which presents the video in a cinematic way. This method improves the picture quality and achieves a picture quality roughly the same as the special effects of a movie. In this invention, the third display mode refers to interactive game mode. Interactive game mode means that after the third mode display is activated through verification by the verification system, the user's desired character is projected into a three-dimensional space. The user wants to project themselves into a virtual scene. The shooting system shoots the user, captures the changes in the real user's movements, and places the real person into the three-dimensional space. According to the changes in the real user's movements, the simulated three-dimensional entity in the three-dimensional space also changes in real time. For example, when the real user plays a game, he controls the movements of the virtual person in the computer, and the scene changes dynamically with the changes of the virtual person, as if the virtual person is operating this three-dimensional space. In this invention, when the user's actions change, a simulated 3D entity of the user is generated in real-time in 3D space. As the user's actions change, the 3D scene changes accordingly. In interactive games, the game display converts the dynamically changing 3D space into a 2D image and displays it in real-time. Users can choose to bind the simulated 3D entity to the user, causing the 3D scene to change dynamically with the user's actions. Alternatively, they can choose not to bind the simulated 3D entity, but by manipulating the simulated 3D entity, the 3D scene will also change accordingly. The fourth display mode in this invention refers to a live broadcast. Users edit the dynamic changes in 3D space, and the final effect of the dynamically changing 3D space being converted into a 2D image is displayed in real-time. The fifth display mode in this invention refers to digital assets, data and assets that users can control or create themselves.

[0055] Based on the cloud-based interactive visualization system and the user's selection of different display modes, real-time rendering transforms dynamic 3D spatial changes into 2D displays and their effects. When the user selects different modes, the cloud-based interactive visualization system sends real-time commands to control the digital engine to produce the final effect and records the 3D spatial information onto the 3D materials. The cloud-based interactive visualization system receives the 3D materials and sends them to the interactive playback system in real-time, selecting the display mode according to the user's needs, approaching real-time performance. During display, 3D materials are converted into 2D materials for real-time display.

[0056] This invention employs virtual reality technology to obtain simulated 3D entities and render virtual 3D spatial scenes in real time based on display spatial positioning information, displaying them on multiple display terminals. By determining spatial information, interaction information, light field information, visualization sequence information, and sound field information, it displays these elements on different display terminals and renders virtual images on the virtual screen. This enables interaction between multiple simulated 3D entities. This invention offers high realism and strong interactivity, and through virtual reality / mixed reality video recording, it can be applied to fields such as virtual production, VR production, motion capture, and virtual pre-visualization.

[0057] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention.

[0058] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not describe the various possible combinations separately.

[0059] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0060] Furthermore, various different implementations of the present invention can be combined arbitrarily, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed in the present invention.

Claims

1. A method for multi-dimensional visual interactive communication, characterized in that, The method includes: According to user requirements, acquire two-dimensional images and identify the two-dimensional information of the two-dimensional images; The two-dimensional information is processed by a serialized neural network to obtain three-dimensional information. The serialized neural network is a recurrent neural network used to predict time-series sequences. The three-dimensional information includes three-dimensional visualization sequence information, three-dimensional interactive information, three-dimensional light field information, three-dimensional sound field information, and three-dimensional spatial information. The three-dimensional visualization sequence information includes the depth, material, and texture of a simulated three-dimensional entity. The three-dimensional information is then simulated and processed in the same virtual three-dimensional space to obtain three-dimensional change data. A dynamic virtual three-dimensional space is obtained by dynamically interpreting the virtual three-dimensional space based on the aforementioned three-dimensional change data. Based on the user's requirements, the dynamic virtual 3D space is rendered into 2D materials for real-time display, including: The user can select the display mode according to their needs. The first display mode is virtual reality, the second display mode is video viewing mode, the third display mode is interactive game mode, the fourth display mode is live broadcast, and the fifth display mode is digital assets. It obtains a visualization sequence of real three-dimensional space and converts the spatial information, sound field information, light field information, interactive information, and image data information in real three-dimensional space into three-dimensional digital assets. Users can add, replace, delete, and edit scene content in virtual three-dimensional space, and the edited content is rendered and takes effect in real time. The process of simulating and running the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data includes: The movement of a three-dimensional entity is simulated based on the three-dimensional visualization sequence information and three-dimensional spatial information. The movement of the three-dimensional entity is synchronized through pixel spatial positioning to obtain three-dimensional entity movement data. The three-dimensional change data includes the three-dimensional entity motion data, three-dimensional entity collision data, three-dimensional entity material data, and three-dimensional sound field data; The step of simulating and running the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data also includes: simulating the physical collision of three-dimensional entities in the scene space based on the three-dimensional interaction information to obtain three-dimensional entity collision data. The material of the three-dimensional entity is identified based on the three-dimensional light field information, and the material data of the three-dimensional entity is obtained. Based on the aforementioned three-dimensional sound field information, the changes in the three-dimensional sound field are simulated to obtain three-dimensional sound field data; The process of dynamically interpreting the virtual three-dimensional space based on the three-dimensional change data to obtain a dynamic virtual three-dimensional space includes: Identify continuous images of real space, establish a three-dimensional spatial information structure, and input the three-dimensional change data into the three-dimensional spatial information structure to obtain a dynamic virtual three-dimensional space.

2. The method according to claim 1, characterized in that, The neurons in the recurrent neural network have memory.

3. The method according to claim 1, characterized in that, The method also includes: A three-dimensional simulation entity is set up and projected into the dynamic virtual three-dimensional space, which changes as the three-dimensional simulation entity changes.

4. The method according to claim 1, characterized in that, The three-dimensional spatial information is at least one of spatial positioning information, spatial structure information, spatial stability information, spatial change information, and physical feedback information.

5. A multi-functional visual interactive device, characterized in that, The device includes: The acquisition unit is used to acquire a two-dimensional image and identify the two-dimensional information of the two-dimensional image according to user requirements; The first processing unit is used to perform serialized neural network processing on the two-dimensional information to obtain three-dimensional information. The serialized neural network is a recurrent neural network, which is used to predict time-series sequences. The three-dimensional information includes three-dimensional visualization sequence information, three-dimensional interactive information, three-dimensional light field information, three-dimensional sound field information, and three-dimensional spatial information. The three-dimensional visualization sequence information includes the depth, material, and texture of the simulated three-dimensional entity. The second processing unit is used to simulate and run the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data. The third processing unit is used to dynamically interpret the virtual three-dimensional space based on the three-dimensional change data to obtain a dynamic virtual three-dimensional space. The fourth processing unit is used to render the dynamic virtual three-dimensional space into two-dimensional material for real-time display according to the user's requirements, including: The user can select the display mode according to their needs. The first display mode is virtual reality, the second display mode is video viewing mode, the third display mode is interactive game mode, the fourth display mode is live broadcast, and the fifth display mode is digital assets. It obtains a visualization sequence of real three-dimensional space and converts the spatial information, sound field information, light field information, interactive information, and image data information in real three-dimensional space into three-dimensional digital assets. Users can add, replace, delete, and edit scene content in virtual three-dimensional space, and the edited content is rendered and takes effect in real time. The process of simulating and running the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data includes: The movement of a three-dimensional entity is simulated based on the three-dimensional visualization sequence information and three-dimensional spatial information. The movement of the three-dimensional entity is synchronized through pixel spatial positioning to obtain three-dimensional entity movement data. The three-dimensional change data includes the three-dimensional entity motion data, three-dimensional entity collision data, three-dimensional entity material data, and three-dimensional sound field data; The step of simulating and running the three-dimensional information in the same virtual three-dimensional space to obtain three-dimensional change data also includes: simulating the physical collision of three-dimensional entities in the scene space based on the three-dimensional interaction information to obtain three-dimensional entity collision data. The material of the three-dimensional entity is identified based on the three-dimensional light field information, and the material data of the three-dimensional entity is obtained. Based on the aforementioned three-dimensional sound field information, the changes in the three-dimensional sound field are simulated to obtain three-dimensional sound field data; The process of dynamically interpreting the virtual three-dimensional space based on the three-dimensional change data to obtain a dynamic virtual three-dimensional space includes: Identify continuous images of real space, establish a three-dimensional spatial information structure, and input the three-dimensional change data into the three-dimensional spatial information structure to obtain a dynamic virtual three-dimensional space.

6. The apparatus according to claim 5, characterized in that, The neurons in the recurrent neural network have memory.

7. The apparatus according to claim 5, characterized in that, The device also includes: The setting unit is used to set a three-dimensional simulation entity and project the three-dimensional simulation entity into the dynamic virtual three-dimensional space, wherein the dynamic virtual three-dimensional space changes as the three-dimensional simulation entity changes.

Images