Method and device for generating space visual interaction medium based on space calculation

Abstract

The embodiment of the invention provides a method and device for generating a space visual interaction medium based on space calculation, and relates to the field of space intelligence and space audiovisual, and the method comprises the steps: carrying out the preprocessing of an original image, and obtaining first information; performing focal plane simulation processing on the first information to obtain second information; extracting complex data of the first information, and performing wavelet transform on the complex data to obtain a feature set; performing depth estimation and view synthesis on the second information to obtain third information; performing spatial visual interaction on the complex data and the third information to generate a new viewpoint depth map; and determining a spatial video according to the first information, the feature set and the new viewpoint depth map. According to the method, the two-dimensional information is converted into the three-dimensional interaction space video, and the interaction experience of the user is improved.

Description

Technical Field

[0001] The present invention relates to the field of spatial intelligence and spatial audio-visual, and in particular to a method and device for generating spatial visual interactive media based on spatial calculation. Background Art

[0002] With the rapid development of science and technology, the field of AI is experiencing unprecedented prosperity, with many remarkable innovative achievements emerging, such as the powerful natural language processing capabilities demonstrated by ChatGPT, the widespread application of AIGC (AI Generated Content) technology, and the rise of creative industries such as AI comics.

[0003] In existing technologies, 3D reconstruction mainly relies on advanced image processing and computer vision technology to extract rich depth information and spatial structure from a single 2D image. Combined with 3D modeling, rendering and animation technology, AI transforms 2D information into a realistic 3D interactive space, allowing users to freely explore and interact in it. However, due to the complex technical characteristics of video synthesis, motion planning, light and shadow effect simulation, existing technologies cannot provide people with a rich and immersive interactive experience. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method and device for generating a spatial visual interactive medium based on spatial calculation, which converts two-dimensional information into three-dimensional interactive spatial video, thereby improving the user's interactive experience.

[0005] In order to achieve the above object, an embodiment of the present invention provides a method for generating a spatial visual interactive medium based on spatial calculation, the method comprising:

[0006] Preprocessing the original image to obtain first information;

[0007] Performing focal plane simulation processing on the first information to obtain second information;

[0008] Extracting complex data of the first information, and performing wavelet transform on the complex data to obtain a feature set;

[0009] performing depth estimation and view synthesis on the second information to obtain third information;

[0010] Performing spatial visual interaction on the complex data and the third information to generate a new viewpoint depth map;

[0011] A spatial video is determined according to the first information, the feature set and the new viewpoint depth map.

[0012] Optionally, the performing focal plane simulation processing on the first information to obtain the second information includes:

[0013] The first information is captured by simulating a camera at different focal lengths and / or focus positions to decompose depth information of the scene.

[0014] Optionally, extracting complex data of the first information includes:

[0015] Data in which at least one of structure, content and dimension in the first information exceeds a threshold is extracted as complex data.

[0016] Optionally, performing depth estimation and view synthesis on the second information to obtain third information includes:

[0017] Analyzing the texture gradient and perspective distortion of the second information to obtain a depth detail map;

[0018] The spatial data of the depth detail map is subjected to classification analysis and processing to obtain third information, wherein the classification analysis and processing includes time series sorting, video encoding and data synchronization.

[0019] Optionally, the performing texture gradient and perspective distortion analysis on the second information to obtain a depth detail map includes:

[0020] Analyzing the gradient direction and speed of the texture in the second information to obtain the image depth, and determining the depth level according to the occlusion between objects in the image;

[0021] Determine the convergence point and / or vanishing point of parallel lines in the second information according to the principle of perspective;

[0022] According to the image depth and depth level, statistical learning and / or deep learning are performed on the convergence points and / or vanishing points to obtain a depth detail map.

[0023] Optionally, the generating a new viewpoint depth map by performing spatial visual interaction on the complex data and the third information includes:

[0024] Performing data set analysis on the complex data and the third information to obtain disparity and depth information, and determining a new viewpoint position according to the disparity and depth information;

[0025] Generate pixel values ​​at the new viewpoint position according to at least one of image interpolation technology, bilinear interpolation, cubic interpolation, and deep learning interpolation;

[0026] A new viewpoint depth map is determined according to the new viewpoint position and the pixel value of the new viewpoint position.

[0027] Optionally, determining the spatial video according to the first information, the feature set and the new viewpoint depth map includes:

[0028] Constructing a three-dimensional scene according to the first information, the feature set and the new viewpoint depth map, and coloring the spatial region of the three-dimensional scene;

[0029] Determine the spatial data according to the spatial visual interactive medium of the three-dimensional scene, wherein the spatial data includes spatial position, spatial relationship, spatial query, spatial analysis, spatial editing and spatial area coloring;

[0030] Determine a three-dimensional interactive space according to the spatial data;

[0031] The data attribute medium in the three-dimensional interactive space is visually and interactively converted to obtain a spatial video.

[0032] Optionally, the original image is at least one monocular image captured by an external source;

[0033] The preprocessing includes at least denoising, distortion correction, color adjustment, contrast enhancement and depth processing, and the depth processing is used to obtain the spatial position, spatial relationship, spatial query, spatial analysis, spatial editing and spatial area coloring of the original image.

[0034] On the other hand, the present invention also provides a device for generating a spatial visual interactive medium based on spatial calculation, the device comprising:

[0035] A first processing module, used for preprocessing the original image to obtain first information;

[0036] A second processing module, configured to perform focal plane simulation processing on the first information to obtain second information;

[0037] A third processing module, used for extracting complex data from the first information, and performing wavelet transform on the complex data to obtain a feature set;

[0038] a fourth processing module, configured to perform depth estimation and view synthesis on the second information to obtain third information;

[0039] A fifth processing module, configured to perform spatial visual interaction on the complex data and the third information to generate a new viewpoint depth map;

[0040] The sixth processing module is used to determine the spatial video according to the first information, the feature set and the new viewpoint depth map.

[0041] Optionally, the performing focal plane simulation processing on the first information to obtain the second information includes:

[0042] The first information is captured by simulating a camera at different focal lengths and / or focus positions to decompose depth information of the scene.

[0043] Optionally, performing depth estimation and view synthesis on the second information to obtain third information includes:

[0044] Analyzing the texture gradient and perspective distortion of the second information to obtain a depth detail map;

[0045] The spatial data of the depth detail map is subjected to classification analysis and processing to obtain third information, wherein the classification analysis and processing includes time series sorting, video encoding and data synchronization.

[0046] Optionally, determining the spatial video according to the first information, the feature set and the new viewpoint depth map includes:

[0047] Constructing a three-dimensional scene according to the first information, the feature set and the new viewpoint depth map, and coloring the spatial region of the three-dimensional scene;

[0048] Determine the spatial data according to the spatial visual interactive medium of the three-dimensional scene, wherein the spatial data includes spatial position, spatial relationship, spatial query, spatial analysis, spatial editing and spatial area coloring;

[0049] Determine a three-dimensional interactive space according to the spatial data;

[0050] The data attribute medium in the three-dimensional interactive space is visually and interactively converted to obtain a spatial video.

[0051] The present invention provides a method for generating a spatial visual interactive medium based on spatial calculation, the method comprising: preprocessing the original image to obtain first information; performing focal plane simulation processing on the first information to obtain second information; extracting complex data of the first information, performing wavelet transform on the complex data to obtain a feature set; performing depth estimation and view synthesis on the second information to obtain third information; performing spatial visual interaction on the complex data and the third information to generate a new viewpoint depth map; determining a spatial video based on the first information, the feature set and the new viewpoint depth map. The method restores a three-dimensional interactive space through a 2D image of a single viewpoint, and automatically generates video content related thereto, converting these two-dimensional information into a realistic three-dimensional interactive space, allowing users to freely explore and interact therein, thereby improving the user's interactive experience.

[0052] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0054] Figure 1 It is a flow chart of a method for generating a spatial visual interactive medium based on spatial calculation according to the present invention;

[0055] Figure 2 It is a flow chart of a specific embodiment of the present invention;

[0056] Figure 3 is a schematic diagram of new viewpoint images at different focal planes in the present invention;

[0057] Figure 4 is a schematic diagram of focal plane simulation of the present invention;

[0058] Figure 5 It is a schematic diagram of spatial calculation of focal plane transformation of the present invention.

[0059] Description of Reference Numerals

[0060] 1-Original viewpoint;

[0061] 2-First simulation viewpoint;

[0062] 3- Second simulation viewpoint;

[0063] 4- Third simulation viewpoint;

[0064] 5- first simulated focal plane;

[0065] 6- Second simulated focal plane;

[0066] 7-third simulated focal plane;

[0067] A-first simulated viewpoint depth space;

[0068] B-second simulated viewpoint depth space;

[0069] C-third simulated viewpoint depth space;

[0070] D-the fourth simulated viewpoint depth space;

[0071] E-Fifth simulated viewpoint depth space. DETAILED DESCRIPTION

[0072] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0073] like Figure 1 As shown, a method for generating a spatial visual interactive medium based on spatial calculation of the present invention includes: step S101 is to pre-process the original image to obtain first information.

[0074] The original image is at least one monocular image captured by an external source. Specifically, the original image (also called the original viewpoint image) is an image captured by the acquisition module from an external source of a viewpoint, and the original image is the basis for generating the spatial domain color and spatial video in the present invention. The acquisition module can be regarded as a component or functional module responsible for collecting, receiving or capturing raw data (such as images, video frames, etc.). It is the first step of the entire processing flow and provides necessary input for subsequent data processing, analysis, modeling and other steps.

[0075] The preprocessing includes at least denoising, distortion correction, color adjustment, contrast enhancement and depth processing, and the depth processing is used to obtain the spatial position, spatial relationship, spatial query, spatial analysis, spatial editing and spatial area coloring of the original image. Specifically, the acquisition module performs preliminary processing on the captured image, such as image depth denoising, size adjustment, color correction, etc. A depth model of the original viewpoint image is constructed to obtain the original viewpoint image space model, and the model parameters are optimized by back propagation algorithm to restore clean data from noise data and restore distortion data to non-distorted data. The preprocessing includes depth processing to obtain the spatial properties of the original image, such as spatial position, spatial relationship, spatial query, spatial analysis, spatial editing, spatial area coloring, etc. The spatial position refers to the relative position of the pixel atoms in the original viewpoint image in three or four dimensions. The spatial position is described by coordinates to locate the pixel atoms. The relative position, direction, distance and topological relationship between pixel atoms are the spatial relationship of pixel atoms. Spatial query is to query a pixel atom according to certain conditions such as spatial position and shape to find and locate other pixel atoms related to specific spatial elements (spatial elements include points, lines, surfaces, etc.). Spatial analysis refers to the ability to divide and classify the pixel atomic space formed by the original viewpoint image into many different spatial types of points, lines, and surfaces, which are used to subsequently combine with the new viewpoint image space to generate a new pixel atomic space. Spatial editing and spatial domain coloring are editing properties that can improve the quality of the original viewpoint image space and convert erroneous colors, so that the pixel atoms in the original image space can be accurately restored. The acquisition module also performs preliminary screening of the captured images to exclude blurred, damaged or non-compliant images to ensure the quality and consistency of the input data. If the format of the original data is incompatible with the subsequent processing module, the acquisition module may also be responsible for converting it to an appropriate format.

[0076] Step S102 is to perform focal plane simulation processing on the first information to obtain second information.

[0077] According to a specific implementation, the performing focal plane simulation processing on the first information to obtain the second information includes: capturing images of the first information at different focal lengths and / or focus positions by a simulated camera to decompose depth information of the scene.

[0078] Specifically, the content and space of the original viewpoint image are analyzed, including but not limited to objects in the space, light conditions, spatial complexity, etc., the goal of focal plane simulation processing is clarified, the depth information of the spatial scene is decomposed, and the relative distances of different objects or regional spaces composed of different pixel atoms are automatically calculated. The parameter settings of the simulated camera, the camera focal length can be the default focal length of the simulated camera or the focal length calculated based on the refraction line and the incident line. When the original viewpoint image is cropped, scaled, blurred, etc., the imaging effects under different focal lengths can be simulated. After continuously determining that the focal length is consistent with the original viewpoint image, the image focal length is calculated. Compare the images simulated at different focal lengths.

[0079] The focal plane simulation refers to the process of decomposing the depth information of a scene by simulating images captured by a camera at different focal lengths or focus positions in computer vision or image processing. This method does not directly perform mathematical transformations on the image, but instead changes the parameters of the simulated camera to observe the performance of the scene at different depth levels. This is the most critical step for depth estimation and 3D / 4D reconstruction.

[0080] Specifically, firstly, multiple images with different depths of field are simulated by the original viewpoint image, and the foreground depth of field and the back depth of field of the original viewpoint image are transformed to obtain multiple images with different depths of field. In the spatial visual interaction system, an acquisition module model is established, which can simulate various parameters of a real camera, such as focal length and focus. According to the simulated focal length and focus position, the corresponding parameters of the acquisition module model are adjusted, and the scene is rendered according to the adjusted acquisition module model parameters to generate images captured by the simulated camera at different focal lengths, focal points, and focal plane positions. Repeat the above simulation steps to generate a series of images at different focal lengths, focal points, and focal plane positions. These images will show the detail changes at different depth levels and generate image sequences. Different three-dimensional spaces are generated according to different viewpoints, and a four-dimensional spatial architecture is constructed. The generated three-dimensional spatial image sequences are compared and analyzed to observe the changes in the clarity (in-focus and out-of-focus) of objects in the scene as the focal length, focus, and focal plane position change. Depth map generation uses image processing (such as deep learning, stereo matching, optical flow method, etc.) to analyze these changes in the image sequence, thereby inferring the depth information corresponding to each pixel point and generating a depth map. Depth information optimization denoises and smoothes the generated depth map to reduce image noise or artifacts introduced by algorithm errors. Through methods such as spatial consistency checks, the logical rationality of the depth values ​​of adjacent pixels in the depth map is ensured. The extracted depth information is combined with image feature data to reconstruct 3D / 4D scenes, such as Figure 3As shown, one enters one space from the original viewpoint image (monocular), and enters another space from another viewpoint. Viewpoint A1 is the viewpoint simulation camera that performs three-dimensional restoration of the original viewpoint image through multi-level decomposition, focal plane simulation and depth processing. The camera calculates the simulated viewpoint S1 and the simulated viewpoint S2 by continuously simulating the focal plane to obtain the focal length. The viewpoint space K1, viewpoint space K2 and viewpoint space K3 are analyzed at the same level to obtain a visual and interactive mobile space. The viewpoint space K1, viewpoint space K2 and viewpoint space K3 are all three-dimensional interactive spaces. You can walk into the viewpoint space K2 from the viewpoint space K1. The viewpoint space is a three-dimensional interactive space that can be freely entered and exited by the person in need. In the present invention, n simulated viewpoints can be simulated and then the right viewpoints of the n simulated viewpoints can be restored to restore n three-dimensional interactive spaces to form a four-dimensional field. You can freely shuttle through any three-dimensional interactive space. Thereby, a four-dimensional scene effect is achieved. As shown Figure 4 As shown, the first simulated viewpoint depth space A, the second simulated viewpoint depth space B, the third simulated viewpoint depth space C, the fourth simulated viewpoint depth space D, and the fifth simulated viewpoint depth space E are five three-dimensional spaces, that is, the three-dimensional space of the simulated binocular is restored. In the present invention, the original viewpoint image is an image taken by a monocular camera for a series of subsequent operations. Specifically, another viewpoint is restored by a monocular (single viewpoint). For example, a right viewpoint is restored by selecting a left viewpoint. The restoration of this right viewpoint is the right viewpoint calculated by the focal length obtained by the simulated camera through the continuous simulation of the focal plane. After obtaining the left and right viewpoints, the three-dimensional space is restored so that it is possible to walk into this space and walk out of this space. The first simulated viewpoint depth space A, the second simulated viewpoint depth space B, the third simulated viewpoint depth space C, the fourth simulated viewpoint depth space D, and the fifth simulated viewpoint depth space E are five different three-dimensional spaces respectively. Different three-dimensional spaces are obtained by continuously adding different original viewpoints to construct a four-dimensional world. The video generated by walking from this space to another space is a spatial video, and the spatial production video is connected through a spatial visual interactive medium. By comparing the depth 3D data with the original viewpoint image, the accuracy and reliability of the focal plane simulation method in extracting depth information are verified. Based on the verification results, the camera model, rendering parameters, and depth extraction algorithm are adjusted and optimized to improve the performance and accuracy of the overall spatial visual interaction system.

[0081] Feature data is data information obtained through complex data processing and wavelet transform processing. The feature data is data information used to describe data characteristics. In image processing, feature data can be corners, edges, textures, color histograms, etc. of an image, which are used for image recognition, classification, retrieval, etc. By extracting the dimension and complexity of feature data, key information is retained by classification. Feature extraction requires corner detection, edge detection, texture analysis, and color histogram analysis. First, the high-frequency information after wavelet transform is used to detect the corners in the image. Corners are usually the points where local features are most obvious in the image, which is very important for image matching and recognition. By analyzing the deep detail layer of wavelet transform, the edges in the image can be identified. Edges are areas in the image where pixel values ​​change dramatically. The statistical characteristics of the wavelet coefficients of wavelet transform (such as energy and entropy) are used to describe the texture features of the image for texture analysis. If the extracted feature dimension is too high, dimensionality reduction processing is required, such as principal component analysis and linear discriminant analysis, to reduce the amount of calculation and improve classification efficiency, and select the most representative features according to the subject's spatial visual interaction system. The spatial visual interaction system uses a classifier trained by extracting feature data to classify and identify multiple images processed in depth, and based on feature extraction, simulates multiple images of a new viewpoint range to obtain a multi-view set image.

[0082] Step S103 is to extract complex data from the first information and perform wavelet transform on the complex data to obtain a feature set.

[0083] According to a specific implementation, extracting complex data from the first information includes: extracting data in the first information whose structure, content and dimension exceed a threshold as complex data, and the threshold of the dimension is four dimensions.

[0084] Specifically, the complex data generally refers to a data set that exceeds conventional and / or simple data types in structure, content, dimension or processing. The complexity of the data structure of complex data includes complex data such as unstructured data and semi-structured data. Compared with structured data (such as tables in a database after multi-level decomposition), unstructured data (such as text, images, audio, video) usually has no fixed format and mode. Semi-structured data refers to data with a certain format, but this structure is not as strict and standardized as a relational database, so it also has a certain complexity in processing. The complexity of the data content of complex data includes diversity, high dimensionality, and dynamics. Complex data contains various types of information, such as text, numerical values, images, videos, etc., which may come from different sources and have different representations and semantics. The high dimensionality of complex data means that when the original viewpoint image is converted into a high-dimensional three-dimensional space video, the data has multiple dimensions or attributes, and there may be complex correlations and dependencies between these dimensions, making data analysis and understanding difficult. In the present invention, the data analysis becomes structured. The dynamic nature of complex data means that in the process of dynamic changes of complex data, new data points may be generated at any time and old data points may become obsolete or invalid, which requires the data processing system in the spatial interaction system to be able to process these data in real time or near real time.

[0085] The wavelet transform is used to decompose complex data into components of different frequencies. Unlike the Fourier transform, the wavelet transform has good localization characteristics in both the time domain and the frequency domain, and can simultaneously reveal the characteristics of the signal in time and frequency. In image processing, wavelet transform is often used for tasks such as image compression, denoising, and feature extraction. Specifically, first read the image file to be processed, which usually includes a two-dimensional array and / or a three-dimensional array (color image, including three RGB channels and constructing a four-dimensional scene), and convert the image file into grayscale (if the color image is processed and the task does not require color information, the image can be converted into a grayscale image first for subsequent processing).

[0086] The effect of wavelet transform depends largely on the selected wavelet basis (such as Haar wavelet, Daubechies wavelet, Symlets wavelet, etc.). Different wavelet bases have different localization characteristics in the time domain and frequency domain, and are suitable for different application scenarios. Determine the decomposition level, and determine the number of layers of wavelet decomposition according to the multi-level decomposition processing requirements in the present invention. The more layers there are, the finer the frequency domain division, but the higher the computational complexity. To perform wavelet transform, a two-dimensional wavelet transform is first performed, and a two-dimensional wavelet transform is performed on the image after preprocessing the original viewpoint image. The two-dimensional wavelet transform refers to a one-dimensional wavelet transform for each row of data extracted, and a one-dimensional wavelet transform is performed on each column to obtain the wavelet coefficient. After the transformation, the image is decomposed into multiple coefficient matrices, including a low-frequency coefficient matrix and multiple high-frequency coefficient matrices. The processing of wavelet coefficients can reduce the amount of data by quantization (reducing the precision of the coefficients) and threshold processing (setting the coefficients less than a certain threshold to zero). In denoising applications, high-frequency noise can be removed by setting a threshold, retaining only important edge and texture information. In feature extraction, specific wavelet coefficients can be selected as features according to task requirements. When reconstructing new viewpoint images, for compressed, denoised, and multi-level decomposed images, inverse wavelet transforms need to reconstruct wavelet coefficients to restore the image. This usually involves inverse two-dimensional wavelet transforms of processed low-frequency and high-frequency coefficient matrices. After obtaining the processed image, feature extraction is performed to obtain a feature set for further analysis. This method optimizes the process by adjusting the wavelet basis, decomposition level, quantization parameters, thresholds, etc.

[0087] Multi-level decomposition refers to the process of gradually decomposing a complex data set or image into simpler, easier to process or analyze parts through a series of steps or levels. The multi-level decomposition is divided into focal plane simulation, depth information acquisition, information integration and analysis. These decomposition levels are usually organized according to frequency, scale, space, resolution, etc., so as to extract information at different abstract levels. Multi-level decomposition includes three steps. First, focal plane simulation processing is required in the present invention. Multi-level decomposition is no longer a direct mathematical transformation of the image, but decomposes the depth information of the scene by simulating different focal planes. That is, the spatial visual interaction system does not directly analyze the complexity of the image content, but captures the information of the scene at different depth levels by changing the focal length or focus position of the simulated camera. Secondly, for the acquisition of depth information, by simulating multiple focal planes, the spatial visual interaction system can obtain clear images (or at least relatively clearer representations) of objects at different depths in the scene. These images of different focal planes can be regarded as multi-level representations of scene depth information, and each level corresponds to the clarity of objects within a specific depth range.

[0088] Step S104 is to perform depth estimation and view synthesis on the second information to obtain third information.

[0089] According to a specific implementation, the depth estimation and view synthesis of the second information to obtain the third information includes: analyzing the texture gradient and perspective distortion analysis of the second information to obtain a depth detail map; and performing time series sorting, video encoding and data synchronization on the depth map corresponding to the depth detail map to obtain the third information.

[0090] The analyzing of the texture gradient and perspective distortion of the second information to obtain a depth detail map includes: analyzing the gradient direction and speed of the texture in the second information to obtain the image depth, and determining the depth level according to the occlusion between objects in the image; determining the convergence point and / or vanishing point of parallel lines in the second information according to the perspective principle; and performing statistical learning and / or deep learning on the convergence point and / or vanishing point according to the image depth and depth level to obtain a depth detail map.

[0091] Specifically, the depth estimation refers to inferring the three-dimensional depth information of objects in the scene from a two-dimensional image or image sequence. This can be achieved through a variety of methods, such as stereoscopic vision, structured light, monocular vision, etc. In monocular vision, depth estimation usually relies on clues in the image, such as texture gradients, occlusion relationships, perspective distortion, etc. Depth cues analyze texture gradients, and infer the depth changes of the surface by analyzing the gradient direction and speed of the texture in the image. Areas with dense textures and fast changes usually represent nearby objects, while areas with sparse textures or slow changes represent distant objects. Based on the occlusion relationship obtained by depth estimation, the spatial visual interaction system observes the mutual occlusion between objects, and the occluded objects are usually located behind the occluders (i.e., deeper levels). Depth estimation performs perspective distortion, and uses the principle of perspective to analyze the convergence point or vanishing point of parallel lines in the image (such as roads, building edges, etc.), which can help the depth estimation scene to have a depth hierarchy. Depth detail map generation: a training model is trained based on the combination of statistical learning and deep learning of the spatial visual interaction system. The three-dimensional structure is restored from the complex mapping relationship between the spatial structure and depth level information in the training model by simulating the multi-view geometry obtained by the focal plane. By repeating the above steps, a four-dimensional structure is obtained to achieve interactive connection between spaces and can be entered and exited according to user needs, thereby obtaining accurate depth level information.

[0092] The view synthesis refers to generating a video of a new view and / or viewpoint based on a known original viewpoint image. In this process, the present invention preferably uses technologies such as image interpolation, view deformation, and illumination model. The view synthesis realizes the functions of virtual reality, augmented reality, video simulation, etc., and observes the same scene from different perspectives or viewpoints through view synthesis to obtain a more immersive and rich experience. The data set of feature data analyzes the parallax and depth information of the data set, uses this information to determine the exact position of the new viewpoint, and uses practical image interpolation technology and bilinear interpolation, cubic interpolation, and more advanced deep learning interpolation to generate pixel values ​​at the new viewpoint position. The graphics transformation and fusion system transforms and deforms the interpolated image according to the position of the new viewpoint to simulate the perspective effect observed from the new viewpoint. The image parts after different interpolation and deformation are fused to ensure a natural transition. When simulating the perspective effect of the new viewpoint, the illumination and shadow are adjusted, and the illumination model is applied to adjust the illumination and shadow effects of the new viewpoint image according to the lighting conditions of the scene. Considering light factors such as ambient light, diffuse reflection, and specular reflection, the perspective space of the new viewpoint is made more accurate. After a series of processing, the synthesized multiple depth level images are sharpened, contrasted, etc. to improve the quality of the video space.

[0093] Step S105 is to generate a new viewpoint depth map by performing spatial visual interaction on the complex data and the third information.

[0094] According to a specific implementation manner, the spatial visual interaction of the complex data and the third information to generate a new viewpoint depth map includes: performing data set analysis on the complex data and the third information to obtain disparity and depth information, and determining a new viewpoint position based on the disparity and depth information; generating pixel values ​​of the new viewpoint position based on at least one method selected from the group consisting of image interpolation technology, bilinear interpolation, cubic interpolation, and deep learning interpolation; and determining a new viewpoint depth map based on the new viewpoint position and the pixel values ​​of the new viewpoint position.

[0095] Step S106 is to determine the spatial video according to the first information, the feature set and the new viewpoint depth map.

[0096] According to a specific implementation, the determining of the spatial video according to the first information, feature set and new viewpoint depth map includes: constructing a three-dimensional scene according to the first information, feature set and new viewpoint depth map, and coloring the spatial region of the three-dimensional scene; determining the spatial data according to the spatial visual interactive medium of the three-dimensional scene, the spatial data including spatial position, spatial relationship, spatial query, spatial analysis, spatial editing and spatial region coloring; determining a three-dimensional interactive space according to the spatial data; performing visual interactive conversion on the data attribute medium in the three-dimensional interactive space to obtain a spatial video. The three-dimensional interactive space is composed of multiple three-dimensional fields, also called a four-dimensional field.

[0097] Specifically, the spatial video generally refers to video content that can express spatial position, direction, and depth information. Unlike traditional two-dimensional videos, spatial videos may contain image data from multiple perspectives or viewpoints, as well as depth maps, light field information, etc. associated with these images. It is a process of combining multiple frames of synthesized new viewpoint image space into a spatial video in a time series. Spatial video is used to provide users with a more immersive and interactive video experience, allowing users to observe video content from different perspectives, and even enter the three-dimensional space presented by the video to explore. With the development of virtual reality and augmented reality technologies, the application prospects of spatial videos are becoming more and more broad.

[0098] In the present invention, AI technology can also automatically generate video content based on this technology. This involves multiple complex links such as video synthesis, motion planning, and light and shadow effect simulation. The spatial visual interaction system has a high degree of creativity and a deep understanding of the physical world. Through the comprehensive application of these technologies, AI can generate coherent and expressive videos, and scalably display dynamic changes and interactive processes in three-dimensional space.

[0099] In short, with the continuous advancement of AI technology, it has become possible to restore three-dimensional interactive space through 2D images from a single viewpoint and automatically generate video content. This not only brings new creative methods and unlimited possibilities to film and television production, game development and other fields, but also provides people with richer, more immersive entertainment and interactive experiences.

[0100] Figure 2 It is a flow chart of a specific embodiment of the present invention, such as Figure 2 As shown, this embodiment includes: acquiring an original viewpoint image (ie, original image) through an acquisition module, connecting the acquisition module and the spatial visual interaction system to transmit the original viewpoint image to the spatial visual interaction system.

[0101] Define the acquisition module to identify and select the viewpoint. First, it is necessary to clarify which viewpoint (or angle) to capture the image from. This viewpoint can be fixed or movable (the spatial visual interaction system controls the acquisition angle, acquisition viewpoint, and acquisition distance of the acquisition module). Configure the acquisition device: According to the selected viewpoint, configure the corresponding image capture device, including adjusting the resolution, frame rate, exposure and other parameters of the acquisition module to ensure that the quality of the captured image meets the needs of subsequent processing. Secondly, capture the original viewpoint image to start the capture: through software or hardware control, start the image capture device and start capturing images from the selected viewpoint. Real-time or timed capture: According to actual needs, you can choose to capture images in real time (such as video streaming) or capture images at specific time intervals (such as one frame per second). Then perform data storage and store the captured images in a certain location of the spatial visual interaction system for subsequent processing (such as local storage device, remote server or cloud storage). Next, connect the acquisition module to the spatial visual interaction system. According to the actual situation of the acquisition module and the spatial visual interaction system, select a suitable data transmission method. According to the selected transmission method, establish the connection between the acquisition module and the spatial visual interaction system. Once the connection is established, the captured raw viewpoint images are transmitted to the spatial visual interaction system in real time or on demand. Then, the spatial visual interaction system receives the raw viewpoint images transmitted from the acquisition module. : The received images are preprocessed, such as denoising, correcting distortion, adjusting colors, etc., to improve image quality. The preprocessed images are 3D reconstructed (if the goal is to build a 3D model) or spatial videos are generated (if the goal is to create a video with depth perception). Finally, the 3D model or spatial video is displayed on the visualization interface, and interactive functions are provided so that users can observe, explore or operate these visual contents from different angles.

[0102] Before the acquisition module transmits the original viewpoint image data, the acquisition module is responsible for performing some processing operations, such as image denoising, resizing, color correction, etc., to preliminarily screen the captured images and exclude blurred, damaged or non-compliant images. To ensure the quality and consistency of the input data. If the format of the original data is incompatible with the subsequent processing module, the acquisition module may also be responsible for converting it to an appropriate format. In the process of preprocessing the original image, the original viewpoint image will first be denoised. The spatial visual interaction system will perform mean filtering, median filtering, and Gaussian filtering on the identified original viewpoint image to obtain the first information. The spatial visual interaction system performs contrast enhancement processing on the original viewpoint image, including histogram equalization, adaptive histogram equalization, contrast stretching, nonlinear enhancement, etc., to obtain the second information.

[0103] After the original viewpoint image is preprocessed, the data is transferred to the multi-level decomposition module for data analysis. Preprocessing of the original viewpoint image, image acquisition and preliminary screening. Capture the original viewpoint image data and perform preliminary screening. Through image recognition algorithms or manual inspection, eliminate blurry, damaged or non-compliant images, perform color correction to ensure the consistency of image color, and adjust the image size to meet the requirements of subsequent processing modules. If the format of the original data is incompatible, the spatial visual interaction system performs format conversion. Reduce the noise in the image and improve the image quality.

[0104] According to the filtering process, contrast enhancement processing is performed after the filtering process. In order to improve the visual effect of the image and make the details in the image clearer, the spatial visual interaction system needs to perform histogram equalization, adaptive histogram equalization, and nonlinear enhancement processing. Histogram equalization: By stretching the histogram of the image to make its distribution more uniform, the contrast of the image is enhanced. Adaptive histogram equalization (such as CLAHE): On the basis of histogram equalization, local areas are restricted to avoid noise amplification caused by excessive enhancement. Contrast stretching: Directly adjust the brightness range of the image to make the dark part darker and the bright part brighter. Nonlinear enhancement: Use nonlinear functions (such as logarithmic functions, exponential functions) to map image pixel values ​​to change the contrast of the image. The first information includes denoised image data and contrast-enhanced image data.

[0105] Data transmission to the multi-level decomposition module The pre-processed first information is transmitted to the multi-level decomposition module. The pre-processing module packages or separately transmits the denoised image data and the contrast-enhanced image data in the first information. The data is transmitted to the multi-level decomposition module wirelessly or by wire. In the multi-level decomposition module, the information is used for further data analysis, such as feature extraction, target detection, three-dimensional reconstruction, etc. for classification and decomposition. Figure 2 As shown in the figure, the original viewpoint image first passes through the preprocessing module (including denoising and contrast enhancement), and then the processed data (first information and second information) is transmitted to the multi-level decomposition module for subsequent data analysis. This process ensures the quality and consistency of the input data and provides a solid foundation for subsequent advanced visual processing tasks.

[0106] The feature set is obtained by extracting complex data from the first information and performing wavelet transform, and the second information is obtained by performing focal plane simulation processing on the first information; the first information is processed by a multi-level grading module, and the multi-level decomposition module will identify, classify, pick up, and process the data information with an attention mechanism based on the received first information. In the present invention, multi-level decomposition refers to being creatively applied to simulate scene depth information of different focal planes. Multi-level decomposition is divided into focal plane simulation, depth information acquisition, information integration and analysis. Multi-level decomposition refers to gradually decomposing a complex data set or image into simpler, easier to process or analyze parts through a series of steps or levels. These decomposition levels are usually organized according to frequency, scale, space, resolution, etc., so as to extract information at different levels of abstraction.

[0107] According to a specific implementation, the multi-level decomposition has three steps. The multi-level decomposition is no longer a direct mathematical transformation of the image, but decomposes the depth information of the scene by simulating different focal planes. This means that the spatial visual interaction system does not directly analyze the complexity of the image content, but captures the information of the scene at different depth levels by changing the focal length or focus position of the simulated camera. Secondly, for the acquisition of depth information, by simulating multiple focal planes, the spatial visual interaction system can obtain clear images (or at least relatively clearer representations) of objects at different depths in the scene. These images of different focal planes can be regarded as multi-level representations of the scene depth information, and each level corresponds to the clarity of objects within a specific depth range. The focal plane simulation processing is performed according to the first information to obtain the second information. In the present invention, the focal plane simulation refers to the decomposition of the depth information of the scene by simulating the images captured by the camera at different focal lengths or focus positions in computer vision or image processing. This method does not directly perform mathematical transformation on the image, but observes the performance of the scene at different depth levels by changing the parameters of the simulated camera. This is the most critical step for the four-dimensional construction in the present invention.

[0108] like Figure 5 As shown, the intersection planes are distributed on different depth planes, and the simulated camera in the scene has an adjustable focal length. The simulated camera calculates the focal point through the incident light and refracted light, and the reflection and refraction intersection of the parallel lines of the left viewpoint, which is the original viewpoint, and the restored right viewpoint, which is the new viewpoint. The focal length is calculated by changing different focal planes of the simulated camera.

[0109] Multiple images with different depths of field are simulated by the original viewpoint image, and the foreground depth of field and the back depth of field of the original viewpoint image are transformed to obtain multiple images with different depths of field. An acquisition module model is established in the spatial visual interaction system, which can simulate various parameters of a real camera, such as focal length and focus. According to the simulated focal length and focus position, the corresponding parameters of the acquisition module model are adjusted, and the scene is rendered according to the adjusted acquisition module model parameters to generate images captured by the simulated camera at different focal lengths, focal points, and focal plane positions. Repeat the above simulation steps to generate a series of images with different focal lengths, focal points, and focal plane positions. These images will show the changes in details at different depth levels and generate image sequences. The generated image sequences are compared and analyzed to observe the changes in the clarity (in-focus and out-of-focus) of objects in the scene as the focal length, focus, and focal plane position change. Depth map generation uses image processing technology (such as deep learning, stereo matching, optical flow method, etc.) to analyze these changes in the image sequence, thereby inferring the depth information corresponding to each pixel and generating a depth map. Depth information optimization denoises and smoothes the generated depth map to reduce image noise or artifacts introduced by algorithm errors. The depth values ​​of adjacent pixels in the depth map are logically reasonable through methods such as spatial consistency checks. The extracted depth information is combined with image feature data to reconstruct the three-dimensional scene. By comparing it with the depth three-dimensional data of the original viewpoint image, the accuracy and reliability of the depth information extracted by the focal plane simulation method are verified. According to the verification results, the camera model, rendering parameters, and depth extraction algorithm are adjusted and optimized to improve the performance and accuracy of the overall spatial visual interaction system.

[0110] Finally, the information from different focal planes can be integrated to reconstruct the three-dimensional structure of the scene, perform depth estimation, or implement image processing tasks that rely on depth information. Although this decomposition and reorganization process is not a mathematical transformation in the traditional sense, it still logically embodies the idea of ​​multi-level decomposition. The advantage of using multi-level decomposition in the present invention is to enhance depth perception, improve processing efficiency and flexibility. By simulating multiple focal planes, the spatial visual interaction system can more accurately capture and represent the depth information of the scene, which is also of great significance to the fields of computer vision, augmented reality, and robot navigation. Compared with directly processing the entire complex scene, the multi-level decomposition method can gradually focus on key information, thereby improving computational efficiency while maintaining processing accuracy. This method allows the number and position of focal planes to be adjusted according to different application requirements, so as to flexibly adapt to different scenes and tasks. Multi-level decomposition in the present invention specifically refers to an innovative method of "decomposing" and obtaining the depth information of the scene by simulating different focal planes. It embodies a framework for hierarchical and structured processing of complex problems. The third information and feature set are obtained through the multi-level decomposition module;

[0111] The feature set refers to the data set obtained by wavelet transform through the multi-level decomposition module. In image processing, wavelet transform is often used for tasks such as image compression, denoising, and feature extraction. In the present invention, it is first necessary to read the image file to be processed, which is usually a two-dimensional array or a three-dimensional array (color image, including three RGB channels) and convert it into grayscale. If the image being processed is a color image and the task does not require color information, the image can be converted into a grayscale image first for subsequent processing. Determine the decomposition level, and determine the number of layers of wavelet decomposition according to the multi-level decomposition processing requirements in the present invention. The more layers there are, the finer the frequency domain division, but the higher the computational complexity.

[0112] After obtaining the pixel value of the new viewpoint position, the graphics transformation and fusion system needs to transform and deform the interpolated image according to the position of the new viewpoint to simulate the perspective effect observed from the new viewpoint. Fusion of different interpolated and deformed image parts to ensure a natural transition. When simulating the perspective effect of the new viewpoint, the lighting and shadows are adjusted, and the lighting model is applied to adjust the lighting and shadow effects of the new viewpoint image according to the lighting conditions of the scene. Shadow effect: In view synthesis, shadow effects are usually generated by calculating the relative position relationship between the light source and the object, and the interaction of light on the surface of the object and other objects in the scene. This involves the generation and rendering of shadow maps, such as shadow mapping and screen space shadows. In view synthesis, the effect of ambient light on the surface of the object can be simulated by adding an ambient light map or using an ambient light model. This helps to increase the brightness and details of the scene and make the image more realistic. By considering light factors such as ambient light, diffuse reflection, and specular reflection, the perspective space of the new viewpoint is made more accurate. After a series of processing, the synthesized multiple depth level images are sharpened, contrasted, etc. to improve the quality of the video space.

[0113] During the training process, the neural network learns how to predict the missing pixel values ​​based on the features of the input image (such as edges, textures, etc.). Due to the powerful representation ability of neural networks, these methods are able to capture more complex image statistical characteristics than traditional interpolation methods and produce higher quality interpolation results. The quality of the interpolation is quantified by evaluating the performance of the network on a specific data set (such as mean square error, peak signal-to-noise ratio, etc.).

[0114] According to the position of the new viewpoint, the interpolated image is transformed in perspective to simulate the perspective effect observed from the new viewpoint. The perspective transformation matrix M can be calculated using the internal and external parameters of the camera.

[0115] Applying the illumination model adjusts the lighting and shading effects of the new viewpoint image, taking into account factors such as ambient light, diffuse reflection, and specular reflection. Ambient light is usually modeled as a global illumination factor that evenly illuminates all objects in the scene. Mathematically, this can be expressed as a constant term that is added to the brightness value of each pixel.

[0116] The mathematical description of the depth estimation process includes: depth clue analysis, texture gradient, occlusion relationship, perspective distortion, depth detail map generation and perspective transformation matrix. Specifically, the original viewpoint image and possible auxiliary information (such as depth sensor data) are input; the depth is inferred by analyzing the gradient direction and speed of the texture in the image; the mutual occlusion between objects is observed to infer the depth level; the perspective principle is used to analyze the convergence point or vanishing point of parallel lines; combined with statistical learning and deep learning, accurate depth level information is obtained from the training model; it is used to transform the image from one viewpoint to another. According to the steps described above, the depth map of the new viewpoint is generated.

[0117] In the present invention, the construction of three-dimensional interactive space refers to the use of computer graphics and virtual reality technology to create a three-dimensional environment with which a user number can interact. This environment can be virtual or a three-dimensional reconstruction based on the real world. The three-dimensional interactive space is used for three-dimensional modeling, rendering, physical simulation, interactive design and other aspects of spatial design. Users can immerse themselves in the virtual world and interact with virtual objects to obtain a more realistic and rich experience. The spatial visual interaction system constructs details such as scenes, objects, materials, reconstructs lighting, shadows, texture mapping, etc., and determines the balance between rendering quality and performance. It is the physical properties of the interactive space, such as mass, matcha, elasticity, etc.

[0118] The mathematical and physical expressions of the space construction process are as follows: new viewpoint depth map, first information of the original viewpoint image (such as color, texture, material, etc.), feature set (such as object recognition, scene segmentation results); construct a three-dimensional scene model based on the depth map and image information; determine lighting, shadows, texture mapping, etc., and render a three-dimensional scene; add physical properties to the interactive space, such as mass, friction, elasticity, etc.; design the interaction method between users and virtual objects. The spatial visual interaction system is based on the creation and transformation of the geometric body, such as translation, rotation, scaling, etc. These transformations can achieve three-dimensional reconstruction through matrix operations. According to the mechanical equations, such as Newton's second law F=ma, it is used to calculate the movement of objects. According to the above steps, a constructed three-dimensional interactive space is obtained. According to the above steps, a constructed three-dimensional interactive space is obtained.

[0119] Obtain the constructed 3D interactive space from the spatial visual interaction system, including rendering information such as scene model, material, lighting, and shadow. Obtain multiple frames of new viewpoint images and their corresponding depth maps from the view synthesis module. These images have been rendered and interpolated according to different viewpoints set by the user or the system. Time series organization: Sort and organize multiple frames of new viewpoint images in time series. This usually means that the images are taken in time sequence or generated according to time steps in simulation. Video encoding: Perform video encoding on each frame of the image, including compression and format conversion to reduce file size and improve transmission efficiency. The encoding process may include steps such as color space conversion, motion estimation, transform coding, quantization, and entropy coding. At the same time, the depth map and light field information also need to be encoded accordingly to ensure that they can be played synchronously with the image data. Data synchronization: Ensure that the image data, depth map, and light field information are synchronized in time. This is crucial to provide accurate depth perception and interactive experience. Video output: Encapsulate the encoded video data, depth map, and light field information into video file formats, such as MP4, AVI, etc. Output video files for playback on virtual reality devices, augmented reality platforms, or traditional video players. Software implementation of spatial visual interaction system: Use 3D rendering engines (such as Unity, Unreal Engine) to build and render 3D scenes. Use video processing libraries (such as FFmpeg) to encode, decode, and convert images and videos. Develop specialized scripts or applications to organize time series, synchronize data, and encapsulate video files. Use high-performance graphics processing units (GPUs) to accelerate the 3D rendering and video encoding process. The storage device needs enough space to store the generated video files and data such as depth maps. View transformation: Use the perspective transformation matrix to project the 3D scene onto the 2D image plane. This involves mathematical operations such as matrix multiplication and coordinate transformation. Image coding: Transform coding (such as DCT transform) converts the image from the spatial domain to the frequency domain to facilitate subsequent quantization and entropy coding. During the quantization process, continuous pixel values ​​or transform coefficients are mapped to a finite set of discrete values ​​to reduce the amount of data. Entropy coding (such as Huffman coding, arithmetic coding) uses the statistical properties of data for lossless or lossy compression. Optical simulation of physical processes: During the 3D rendering process, the propagation and interaction of light in the scene is simulated, including physical phenomena such as reflection, refraction, and shadow. Use illumination models (such as the Phong model) to calculate the lighting effects on the surface of an object to produce realistic visual effects. Motion simulation: If the spatial video contains dynamic scenes or animation effects, the motion trajectory and dynamic characteristics of the object are simulated. Use physics equations (such as Newton's second law) to calculate parameters such as the acceleration, velocity, and position of the object. Perception simulation: Use depth maps and light field information to simulate the human eye's perception of three-dimensional space, including depth perception, motion parallax, etc.

[0120] The present invention discloses a method for restoring a three-dimensional interactive space through a 2D image from a single viewpoint. Specifically, the method includes restoring a three-dimensional interactive space through a 2D image from a single viewpoint, automatically generating video content related thereto, and converting these two-dimensional information into a realistic three-dimensional interactive space, so that users can freely explore and interact therein. Through a 2D image from one of the left viewpoint or the right viewpoint, the view images of the two viewpoints are simulated and restored by continuously adjusting the viewpoint distance, and rich depth information and spatial structure are extracted from a single 2D image.

[0121] Through the reasoning and correction of the focal plane, the focal planes at different depths are simulated, and the image features under these different focal planes are captured or calculated. The depth of the original image is estimated, and the depth map of the scene is estimated by stereo matching, deep learning and other methods using the original image and focal plane information of a viewpoint. Based on the estimated depth map and known camera parameters (such as focal length, baseline distance, etc.), the view transformation and projection technology in computer graphics, that is, deep view synthesis, is used to generate the right viewpoint image.

[0122] The present invention does not directly perform multi-level decomposition of the image (such as wavelet transform, etc.), but refers to decomposing the depth information of the scene through simulation of different focal planes. Based on the depth map of the 2D image, the present invention utilizes the psychological stereoscopic vision characteristics of the human eye, extracts the depth information from the plane image of a viewpoint, and combines it with the original viewpoint image to synthesize and reconstruct the three-dimensional interactive space.

[0123] Through the present invention, users can achieve spatial roaming, that is, free movement and rotation in the three-dimensional interactive space, controlled by input devices such as a mouse, keyboard or other devices, and can move freely in the three-dimensional interactive space and generate interactive feedback, providing visual (such as visual effects during collision) and auditory (such as collision sounds) feedback.

[0124] On the other hand, the present invention also proposes a device for generating a spatial visual interactive medium based on spatial computing, the device comprising: a first processing module, used to preprocess the original image to obtain first information; a second processing module, used to perform focal plane simulation processing on the first information to obtain second information; a third processing module, used to extract complex data of the first information, and perform wavelet transform on the complex data to obtain a feature set; a fourth processing module, used to perform depth estimation and view synthesis on the second information to obtain third information; a fifth processing module, used to perform spatial visual interaction on the complex data and the third information to generate a new viewpoint depth map; and a sixth processing module, used to determine the spatial video based on the first information, the feature set and the new viewpoint depth map.

[0125] The step of performing focal plane simulation processing on the first information to obtain the second information includes: capturing images of the first information at different focal lengths and / or focal positions by simulating a camera to decompose the depth information of the scene. The step of performing depth estimation and view synthesis on the second information to obtain the third information includes: analyzing the texture gradient and perspective distortion analysis of the second information to obtain a depth detail map; performing classification analysis processing on the spatial data of the depth detail map to obtain the third information, wherein the classification analysis processing includes time series sorting, video encoding and data synchronization. The step of determining the spatial video according to the first information, the feature set and the new viewpoint depth map includes: constructing a three-dimensional scene according to the first information, the feature set and the new viewpoint depth map, and coloring the spatial region of the three-dimensional scene; determining the spatial data according to the spatial visual interactive medium of the three-dimensional scene, wherein the spatial data includes spatial position, spatial relationship, spatial query, spatial analysis, spatial editing and spatial region coloring; determining a three-dimensional interactive space according to the spatial data; and performing visual interactive conversion on the data attribute medium in the three-dimensional interactive space to obtain the spatial video.

[0126] The present invention provides a method for generating a spatial visual interactive medium based on spatial calculation, the method comprising: preprocessing the original image to obtain first information; performing focal plane simulation processing on the first information to obtain second information; extracting complex data of the first information, performing wavelet transform on the complex data to obtain a feature set; performing depth estimation and view synthesis on the second information to obtain third information; performing spatial visual interaction on the complex data and the third information to generate a new viewpoint depth map; determining a spatial video based on the first information, the feature set and the new viewpoint depth map. The method restores a three-dimensional interactive space through a 2D image of a single viewpoint, and automatically generates video content related thereto, converting these two-dimensional information into a realistic three-dimensional interactive space, allowing users to freely explore and interact therein, thereby improving the user's interactive experience.

[0127] The optional implementation modes of the embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical scheme of the embodiments of the present invention, and these simple modifications all belong to the protection scope of the embodiments of the present invention.

[0128] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not further describe various possible combinations.

[0129] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a single-chip microcomputer, a chip or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0130] In addition, various implementation modes of the embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed by the embodiments of the present invention.

Claims

1. A method for generating a spatial visual interactive medium based on spatial calculation, characterized in that: The method includes: Preprocessing the original image to obtain first information; Performing focal plane simulation processing on the first information to obtain second information; Extracting complex data of the first information, and performing wavelet transform on the complex data to obtain a feature set; performing depth estimation and view synthesis on the second information to obtain third information; Performing spatial visual interaction on the complex data and the third information to generate a new viewpoint depth map; A spatial video is determined according to the first information, the feature set and the new viewpoint depth map.

2. The method according to claim 1, characterized in that The performing focal plane simulation processing on the first information to obtain second information includes: The first information is captured by simulating a camera at different focal lengths and / or focus positions to decompose depth information of the scene.

3. The method according to claim 1, characterized in that The extracting of complex data of the first information includes: Data in which at least one of structure, content and dimension in the first information exceeds a threshold is extracted as complex data.

4. The method according to claim 1, characterized in that: The performing depth estimation and view synthesis on the second information to obtain third information includes: Analyzing the texture gradient and perspective distortion of the second information to obtain a depth detail map; The spatial data of the depth detail map is subjected to classification analysis and processing to obtain third information, wherein the classification analysis and processing includes time series sorting, video encoding and data synchronization.

5. The method according to claim 4, characterized in that The step of analyzing the texture gradient and perspective distortion of the second information to obtain a depth detail map includes: Analyzing the gradient direction and speed of the texture in the second information to obtain the image depth, and determining the depth level according to the occlusion between objects in the image; Determine the convergence point and / or vanishing point of parallel lines in the second information according to the principle of perspective; According to the image depth and depth level, statistical learning and / or deep learning are performed on the convergence points and / or vanishing points to obtain a depth detail map.

6. The method according to claim 1, characterized in that The generating a new viewpoint depth map by performing spatial visual interaction on the complex data and the third information includes: Performing data set analysis on the complex data and the third information to obtain disparity and depth information, and determining a new viewpoint position according to the disparity and depth information; Generate pixel values ​​at the new viewpoint position according to at least one of image interpolation technology, bilinear interpolation, cubic interpolation, and deep learning interpolation; A new viewpoint depth map is determined according to the new viewpoint position and the pixel value of the new viewpoint position.

7. The method according to claim 1, characterized in that The determining of the spatial video according to the first information, the feature set and the new viewpoint depth map includes: Constructing a three-dimensional scene according to the first information, the feature set and the new viewpoint depth map, and coloring the spatial region of the three-dimensional scene; Determine spatial data according to the spatial visual interactive medium of the three-dimensional scene, wherein the spatial data includes spatial position, spatial relationship, spatial query, spatial analysis, spatial editing and spatial area coloring; Determine a three-dimensional interactive space according to the spatial data; The data attribute medium in the three-dimensional interactive space is visually and interactively converted to obtain a spatial video.

8. The method according to claim 1, characterized in that The original image is at least one monocular image captured by an external source; The preprocessing includes at least denoising, distortion correction, color adjustment, contrast enhancement and depth processing, and the depth processing is used to obtain the spatial position, spatial relationship, spatial query, spatial analysis, spatial editing and spatial area coloring of the original image.

9. A device for generating spatial visual interactive media based on spatial calculation, characterized in that: The device includes: A first processing module, used for preprocessing the original image to obtain first information; A second processing module, configured to perform focal plane simulation processing on the first information to obtain second information; A third processing module, used for extracting complex data from the first information, and performing wavelet transform on the complex data to obtain a feature set; a fourth processing module, configured to perform depth estimation and view synthesis on the second information to obtain third information; A fifth processing module, configured to perform spatial visual interaction on the complex data and the third information to generate a new viewpoint depth map; The sixth processing module is used to determine the spatial video according to the first information, the feature set and the new viewpoint depth map.

10. The device according to claim 9, characterized in that The performing focal plane simulation processing on the first information to obtain second information includes: The first information is captured by simulating a camera at different focal lengths and / or focus positions to decompose depth information of the scene.

11. The device according to claim 9, characterized in that The performing depth estimation and view synthesis on the second information to obtain third information includes: Analyzing the texture gradient and perspective distortion of the second information to obtain a depth detail map; The spatial data of the depth detail map is subjected to classification analysis and processing to obtain third information, wherein the classification analysis and processing includes time series sorting, video encoding and data synchronization.

12. The device according to claim 9, characterized in that The determining of the spatial video according to the first information, the feature set and the new viewpoint depth map includes: Constructing a three-dimensional scene according to the first information, the feature set and the new viewpoint depth map, and coloring the spatial region of the three-dimensional scene; Determine spatial data according to the spatial visual interactive medium of the three-dimensional scene, wherein the spatial data includes spatial position, spatial relationship, spatial query, spatial analysis, spatial editing and spatial area coloring; Determine a three-dimensional interactive space according to the spatial data; The data attribute medium in the three-dimensional interactive space is visually and interactively converted to obtain a spatial video.

Images