Real-time 3D model and real scene fusion system based on smartphone camera

Through the smartphone camera combined with reverse light projection and energy propagation simulation technology, the problems of high equipment cost and computational complexity are solved, and the 3D model and real scene fusion in diverse scenarios are realized, providing high-quality, smooth virtual and reality The experience improves user interaction and the authenticity of lighting effects.

CN117456135BActive Publication Date: 2024-12-03JINAN YITE NETWORK INFORMATION CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202311380518.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-24
Publication Date
2024-12-03
Estimated Expiration
2043-10-24

AI Technical Summary

Technical Problem

When the prior art realizes the integration of 3D models and real scenes, there are problems such as high equipment cost, poor mobility, high calculation complexity, poor real-time performance and unreal lighting effects, and it is difficult to achieve accurate identification and fusion in diverse scenarios.

Method used

Using a real-time 3D model based on smartphone camera and a real-life fusion system, the object feature matching and lighting effects can be realized through reverse light projection, energy propagation simulation and high-dimensional data structure, combining multi-scale filters and conditional random field technology. Perform precise object recognition and position matching.

Benefits of technology

It realizes the integration of efficient 3D models and real scenes, provides smooth and instant augmented reality and virtual reality experience, improves the authenticity of lighting effects and user interaction, and ensures high-quality virtual and realistic interaction in mobile environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117456135B_ABST
    Figure CN117456135B_ABST
Patent Text Reader

Abstract

The present invention discloses a real-time 3D model and real scene fusion system based on a smartphone camera, and relates to the field of augmented reality technology. The system includes: a camera configured to obtain a real scene image; a real scene image modeling unit configured to model the real scene image into a high-dimensional data structure, and each object in the real scene image is represented as a high-dimensional feature vector; a light processing unit configured to perform reverse light projection for each pixel in the real scene image to determine the light path of the pixel, and interpolate the light path in a high-dimensional space to obtain the sampling point position; a fusion unit configured to calculate the energy distribution of each sampling point according to the sampling point position obtained by interpolation, and display the fused color value obtained by reverse projection in a 3D model to achieve fusion of the 3D model and the real scene. The present invention achieves high-quality, real-time and accurate fusion of 3D models and real scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a real-time 3D model and real scene fusion system based on a smartphone camera. Background Art

[0002] With the rapid development of smartphones and mobile computing, more and more application scenarios need to integrate virtual content with the real world to create a richer and more immersive user experience. Among them, the integration of real-time generated 3D models with real-life images has brought unlimited possibilities to fields such as augmented reality (AR) and virtual reality (VR). However, there are still some challenges in achieving high-quality fusion of 3D models and real scenes, including accurate object recognition, precise position matching, and realistic lighting effects.

[0003] In traditional methods of fusing 3D models with real scenes, it is usually necessary to rely on dedicated equipment or complex camera equipment to obtain scene information, and use complex algorithms for object recognition and positioning. However, these methods have many problems in practical applications. First, the high cost of dedicated equipment limits its popularity in the mass market. Second, due to its strong dependence on equipment and poor mobility, it cannot meet the needs of users in various scenarios. In addition, existing object recognition methods are often sensitive to the lighting and viewing angle of the scene, making it difficult to achieve accurate recognition and fusion in a variety of scenarios.

[0004] On the other hand, in the prior art, although methods such as ray tracing and projection have been widely used in the fusion of 3D models and real scenes, they still have problems such as high computational complexity, poor real-time performance, and insufficiently realistic results. This limits the scope of application of these methods in real-time applications. At the same time, due to the influence of factors such as lighting conditions and the reflective properties of the object surface, traditional 3D model and real scene fusion methods often find it difficult to obtain satisfactory fusion effects. Summary of the invention

[0005] The purpose of the present invention is to provide a real-time 3D model and real scene fusion system based on a smartphone camera.

[0006] In order to solve the above technical problems, the present invention provides a real-time 3D model and real scene fusion system based on a smartphone camera, the system comprising: a camera, configured to acquire a real scene image; a real scene image modeling unit, configured to model the real scene image into a high-dimensional data structure, each object in the real scene image is represented as a high-dimensional feature vector; a light processing unit, configured to perform reverse light projection on each pixel in the real scene image to determine the light path of the pixel, map the light path to a high-dimensional space, and form a high-dimensional coordinate system; each light path is mapped to a high-dimensional point, and for each high-dimensional point of the light path, the highest point is mapped to the highest point by matching the highest point. The feature vectors of objects in similar scenes are used to determine the objects to which they belong, and the light paths are interpolated in high-dimensional space through an interpolation method to obtain the sampling point positions; a fusion unit is configured to calculate the energy distribution of each sampling point according to the sampling point positions obtained by interpolation, and the propagation of light is simulated by moving the high-dimensional points of the light paths in the high-dimensional space, and the energy convergence value is calculated in the high-dimensional space for each high-dimensional point of the light path, and the fused color value of each pixel position is obtained by back-projecting the energy convergence value back to the two-dimensional image space, and the fused color value obtained by back-projection is displayed in the 3D model to achieve the fusion of the 3D model and the real scene.

[0007] Furthermore, the method in which the light processing unit performs reverse light projection for each pixel in the real scene image to determine the light path of the pixel includes: obtaining the internal parameters and external parameters of the camera, converting the pixel position (x, y) into coordinates (u, v, w) in the camera coordinate system; using the following formula to calculate the reverse light direction vector d from the camera position to the pixel position (x, y) xy :

[0008]

[0009] The starting point of the reverse ray is (u,v,w).

[0010] Furthermore, the method in which the light processing unit maps the light path to a high-dimensional space to form a high-dimensional coordinate system includes: mapping the light path of each pixel to a high-dimensional space to form a high-dimensional coordinate system, mapping the direction vector d of each light path to a high-dimensional space to form a high-dimensional coordinate system, xy Map it into a part of the high-dimensional space, map the pixel position (x, y) into another part of the high-dimensional space, and get the representation p of each light path in the high-dimensional space. xy , where p xy =[d xy ,x,y].

[0011] Furthermore, the method in which the light processing unit determines the object to which each high-dimensional point of the light path belongs by matching the feature vector of the object in the most similar scene includes: for each high-dimensional point p of the light pathxy , use the following formula to calculate the feature vector f of each object i The similarity between:

[0012]

[0013] where · represents the dot product of the vectors; for each high-dimensional point p of the light path xy , calculate its similarity with all objects, and select the object corresponding to the maximum similarity as the object to which it belongs.

[0014] Furthermore, the method in which the light processing unit interpolates the light path in the high-dimensional space by an interpolation method to obtain sampling points includes:

[0015] Suppose there are two high-dimensional points p in the high-dimensional space a and p b , and the interpolation parameter l, use the following formula to obtain the sampling point p between two high-dimensional points l :

[0016] p l =0.25*(1-l)·p a +0.75*l·p b ;

[0017] Among them, p l is the sampling point location.

[0018] Furthermore, the fusion unit calculates the energy distribution of each sampling point according to the sampling points obtained by interpolation, and the method of simulating the propagation of light by moving the high-dimensional points of the light path in the high-dimensional space includes:

[0019] For the sampling point position p l , calculate its energy distribution, let E(p l ) represents the energy distribution in high-dimensional space, which is calculated by the following formula:

[0020]

[0021] Among them, f i is the sampling point position p l The corresponding high-dimensional point p a Or a high-dimensional point p b The feature vector of the object; is the adjustment parameter, a positive number less than 0.01.

[0022] After obtaining the energy distribution, the propagation of light is simulated in high-dimensional space. In each time step dt, the energy is transferred to the next sampling point according to the energy distribution and the direction of the light:

[0023]

[0024] Among them, p' l (t+dt) is the position of the next sampling point after the time step dt, t is the time variable; let p l (t), is the time t, two high-dimensional points p in the high-dimensional space a and p b , the sampling point location under the interpolation parameter l.

[0025] Furthermore, the method for the fusion unit to calculate the energy convergence value in the high-dimensional space for each high-dimensional point of the light path includes: for each high-dimensional point of the light path, calculating the total energy of all corresponding sampling points as the energy convergence value of the high-dimensional point.

[0026] Furthermore, the feature vector f of the object i It is calculated by the following method:

[0027] Step 1: Detect all object regions in the real scene image, specifically including: preprocessing the real scene image using a feature conversion model to convert the real scene image into a frequency domain representation; identifying the region of interest in the frequency domain representation as the object region by applying a multi-scale filter group;

[0028] Step 2: Calculate the feature vector of each object region as the feature vector f of each object i .

[0029] Furthermore, the feature conversion model is expressed using the following formula:

[0030]

[0031] Among them, F(z,c) is the pixel value of the pixel with pixel coordinates (z,c) of the input image, (b,n) is the frequency domain coordinate, N is the image size; QFT(b,n) is the frequency domain representation.

[0032] The real-time 3D model and real scene fusion system based on the smartphone camera of the present invention has the following beneficial effects: the present invention uses the smartphone camera to obtain the real scene image, combines the reverse light projection, energy propagation simulation and other technologies, and realizes the fusion of 3D model and real scene under high efficiency. Compared with the traditional ray tracing and other methods, the algorithm of the present invention is more lightweight and efficient, making it possible to merge the real-time 3D model with the real scene. Users can get smooth and instant augmented reality and virtual reality experience in a mobile environment, and this real-time performance will bring new possibilities for various application scenarios. The present invention introduces technologies such as multi-scale filters and conditional random fields, and realizes more refined and accurate fusion of 3D models and real scenes through precise object recognition and position matching. Methods such as reverse light projection and energy propagation simulation make the lighting effect more realistic, so that the fusion between virtual models and real scenes is more natural. Users can obtain a more realistic and immersive virtual and real interaction experience, which brings higher quality visual effects and sensory experience to users. The present invention optimizes the calculation of light propagation and energy convergence, and realizes the calculation of energy distribution and color value by simulating light propagation in high-dimensional space, thereby realizing a real-time fusion process. This highly efficient algorithm design enables the system to maintain smooth operation in scenarios with high real-time requirements, and users can obtain an instant virtual and real-world fusion experience, greatly enhancing the user's sense of participation and interactivity. The present invention maps the high-dimensional points of each light path to the most similar object feature vector through similarity calculation, thereby achieving accurate object recognition. At the same time, the interpolation calculation in the high-dimensional space enables the light path to obtain a more accurate position, thereby further improving the matching accuracy of the object position. This precise recognition and matching provides a solid foundation for the authenticity and realism of the fusion results. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0034] Figure 1 A schematic diagram of the system structure of a real-time 3D model and real scene fusion system based on a smartphone camera provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The core of the present invention is to provide a real-time 3D model and real-scene fusion system based on a smartphone camera. By innovatively combining advanced technologies such as reverse ray projection and energy propagation simulation, high-quality, real-time and accurate 3D model and real-scene fusion is achieved. The system realizes the real interaction between virtual and reality by mapping the real-scene image to a high-dimensional data structure, representing the objects in the real-scene image as high-dimensional feature vectors, and simulating the propagation of light and energy distribution in high-dimensional space. Compared with the prior art, the present invention brings significant advantages in real-time, fusion effect, mobility and cost-effectiveness. Users can get higher quality, more realistic and smoother augmented reality and virtual reality experience, and it also brings new opportunities for the expansion and innovation of application fields.

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Example 1: Reference Figure 1 A real-time 3D model and real scene fusion system based on a smartphone camera, the system comprising: a camera configured to acquire a real scene image; a real scene image modeling unit configured to model the real scene image into a high-dimensional data structure, wherein each object in the real scene image is represented as a high-dimensional feature vector; a light processing unit configured to perform reverse light projection on each pixel in the real scene image to determine the light path of the pixel, and map the light path to a high-dimensional space to form a high-dimensional coordinate system; each light path is mapped to a high-dimensional point, and for each high-dimensional point of the light path, by matching the most similar object in the scene The method comprises the following steps: using a feature vector of a body to determine the object to which it belongs, interpolating the light path in a high-dimensional space by an interpolation method to obtain the sampling point position; a fusion unit is configured to calculate the energy distribution of each sampling point according to the sampling point position obtained by interpolation, simulating the propagation of light by moving the high-dimensional points of the light path in the high-dimensional space, calculating the energy convergence value in the high-dimensional space for each high-dimensional point of the light path, obtaining the fused color value of each pixel position by back-projecting the energy convergence value back to the two-dimensional image space, and displaying the back-projected fused color value in the 3D model to achieve the fusion of the 3D model and the real scene.

[0038] Specifically, traditional image processing and computer vision techniques usually perform dimensionality reduction on images, that is, converting images from high-dimensional pixel space to low-dimensional feature vectors. This dimensionality reduction method can reduce computational complexity, but it may also lose some useful information. In contrast, mapping real-life images to high-dimensional data structures is a "hyperdimensional" representation method that attempts to retain more image information in a higher-dimensional space.

[0039] Specifically, the high-dimensional data structure may adopt a vector space model in which each dimension represents a different feature or attribute of the image. These features can be color, texture, shape, lighting, etc., and traditional image processing methods may not be able to capture the complex relationship of these features. By mapping the image to a high-dimensional space, the system can more comprehensively express the characteristics of each object, thereby improving the accuracy of subsequent processing steps. Real-life images contain rich information, including color, texture, shape, etc. Mapping the image to a high-dimensional data structure can capture more details of this information, so that each object can be described by features of more dimensions, thereby more accurately distinguishing different objects. Each dimension in the high-dimensional data structure can represent a specific aspect of the object, such as color distribution, shape change, etc. In this way, by comparing high-dimensional feature vectors, the system can more accurately identify and distinguish different objects in the real-life image, thereby improving the accuracy of object recognition. For an object, different dimensions of the feature vector can represent different color distributions, texture features, shape descriptions, etc. By combining these features into a high-dimensional feature vector, the characteristics of the object in the real-life image can be more comprehensively expressed, including its information in different aspects.

[0040] The core principle of the light processing unit is reverse ray casting. When a camera captures a real-life image, it actually sends a beam of light from a point on the surface of an object to the camera. Reverse ray casting means that for each pixel in the real-life image, we trace back the ray path corresponding to that pixel to determine how this ray path is reflected from the object to the camera. Such ray path casting can reflect the depth and geometry of the object.

[0041] By mapping the light path to a high-dimensional space, the system can create a high-dimensional point for each light path, which represents the information of the light in a high-dimensional coordinate system, such as the angle of incidence, angle of reflection, etc. Through this mapping, the system can describe the propagation and reflection of light in the real scene in the form of high-dimensional coordinates, providing more information for subsequent processing steps. The primary function of the light processing unit is to establish a light path for each pixel in the real scene image. This step is the key to achieving depth perception of objects and scenes. By reverse ray casting, the system can infer how the light path corresponding to each pixel propagates in the scene. In high-dimensional space, the system can use interpolation methods to calculate the sampling point positions of the light path. These sampling point positions play a key role in the subsequent fusion and energy propagation process, affecting the fusion effect and the accuracy of light propagation simulation.

[0042] In photography, the imaging process of a camera is to form an image by reflecting or refracting light from the surface of an object onto the camera lens. In a camera, light is reflected or refracted from objects in the scene and eventually converges to pixels on the imaging plane. The concept of reverse ray casting is to trace the path of light rays from the pixels on the camera imaging plane back to the scene. This means that for each pixel, we find a point on the camera imaging plane and then trace the ray path corresponding to that point in reverse to determine the position and direction of the light rays from the surface of the object. The key to reverse ray casting is to determine the position of objects in the scene that intersect with a particular ray path. By detecting the points where the ray intersects with objects in the scene, the position of the object in 3D space can be determined. These intersection points are usually calculated by intersection with the geometry of the objects in the scene. The result of reverse ray casting is a ray path for each pixel that intersects with the objects in the scene. By calculating the distance from the camera to the intersection point, the depth information can be obtained. This depth information can be used to construct a depth map to restore the 2D image to a 3D scene.

[0043] In the light processing unit, each ray path is assigned an energy distribution, which represents the distribution of energy in the process of light reflecting or refracting from the object to the camera. This energy distribution is calculated based on the propagation and interaction of light in the scene, which tells us how different parts of the light on the surface of the object affect the final imaging. The fusion unit uses the energy distribution to simulate the propagation of light in the real-life image. This means that by tracing each ray path, considering the interaction, reflection and refraction of light with the surface of the object, the propagation process of light from the object to the camera is simulated. In the process of simulating light propagation, energy may converge from different paths to the same point. The fusion unit calculates the energy convergence value, which represents the contribution of the energy of each ray path at that point in the high-dimensional space.

[0044] Embodiment 2: Based on the previous embodiment, the method in which the light processing unit performs reverse light projection for each pixel in the real scene image to determine the light path of the pixel includes: obtaining the internal and external parameters of the camera, converting the pixel position (x, y) into coordinates (u, v, w) in the camera coordinate system; using the following formula to calculate the reverse light direction vector d from the camera position to the pixel position (x, y) xy :

[0045]

[0046] The starting point of the reverse ray is (u,v,w).

[0047] Specifically, the intrinsic and extrinsic parameters of the camera are parameters that describe the geometric relationship of camera imaging. Intrinsic parameters include focal length, principal point coordinates, etc., and extrinsic parameters include the position and direction of the camera in the world coordinate system. These parameters are used to convert pixel positions into coordinates in the camera coordinate system. The pixel position (x, y) can be converted into coordinates in the camera coordinate system (u, v, w) through the intrinsic parameters of the camera. This conversion takes into account parameters such as focal length and principal point coordinates, so that the points in the camera coordinate system correspond to points in the real world. Using the given formula, calculate the reverse light direction vector d from the camera position (u, v, w) to the pixel position (x, y) xy . This vector represents the direction of the light from the camera position to the pixel position, which is the direction from the scene to the camera. The denominator in the formula is to normalize the vector to ensure that the direction of the light is a unit vector. The starting point of the reverse ray is the camera position (u,v,w). This is because we want to trace the reverse path of the light from the camera position to find the point where the light interacts with the object. The principle of this method is based on the basic principle of ray tracing, that is, starting from the camera position, tracing the path of the light and finding its intersection with the surface of the object. In this case, we are tracing in the reverse direction of the light, from the pixel position back to the camera position to determine the path of the light.

[0048] Embodiment 3: Based on the previous embodiment, the method in which the light processing unit maps the light path to a high-dimensional space to form a high-dimensional coordinate system includes: mapping the light path of each pixel to a high-dimensional space to form a high-dimensional coordinate system, and mapping the direction vector d of each light path to a high-dimensional space to form a high-dimensional coordinate system. xy Map it into a part of the high-dimensional space, map the pixel position (x, y) into another part of the high-dimensional space, and get the representation p of each light path in the high-dimensional space. xy , where p xy =[d xy ,x,y].

[0049] Specifically, in this embodiment, for each pixel in the real scene image, the light processing unit maps the corresponding light path to a high-dimensional space. The high-dimensional space is constructed as a coordinate system, in which each light path can be represented by a high-dimensional vector. In the high-dimensional space, the direction vector d of each light path xy is mapped into one part of the high-dimensional space, and the pixel position (x, y) is also mapped into another part of the high-dimensional space. This process combines these two parts of information to construct a high-dimensional vector that represents the direction of the light and its position in the image. The representation of each light path in high-dimensional space is p xy It is defined as a high-dimensional vector containing a direction vector and a pixel position. xy =[d xy ,x,y]. The dimension of this high-dimensional vector depends on the dimension of the pixels in the original image and the dimension of the light direction. The principle of this method is to map the light path from the two-dimensional image plane to a higher-dimensional coordinate space. By combining the direction vector of the light path and the pixel position, a vector in a high-dimensional space is constructed, which contains more information about the light. In high-dimensional space, the light path can be described more richly because it includes not only the direction information but also the position information in the image. This high-dimensional representation will be used in subsequent processing steps such as interpolation, energy propagation simulation, etc. to achieve more accurate light propagation and fusion effects.

[0050] Embodiment 4: Based on the above embodiment, the method in which the light processing unit determines the object to which each high-dimensional point of the light path belongs by matching the feature vector of the object in the most similar scene includes: for each high-dimensional point p of the light path xy , use the following formula to calculate the feature vector f of each object i The similarity between:

[0051]

[0052] where · represents the dot product of the vectors; for each high-dimensional point p of the light path xy , calculate its similarity with all objects, and select the object corresponding to the maximum similarity as the object to which it belongs.

[0053] Specifically, for each high-dimensional point p on the light path xy, the light processing unit uses a similarity calculation formula to evaluate the similarity between it and the feature vectors of each object in the scene. The similarity is calculated in the form of cosine similarity, which measures the directional similarity of two vectors by calculating the ratio of the dot product and the modulus of the two vectors. The light processing unit uses similarity calculation to associate light path points with objects in the scene. This process enables each light path to be mapped to the object that is most similar to it, so that the light path can obtain accurate position and attributes in the scene. By associating light path points with the most similar objects, the system can fuse real-life images with 3D models. This process allows the system to overlay real-life information on the 3D model, thereby achieving a fusion effect of real scenes and virtual models.

[0054] Embodiment 5: Based on the previous embodiment, the light processing unit interpolates the light path in a high-dimensional space by an interpolation method to obtain a sampling point, including:

[0055] Suppose there are two high-dimensional points p in the high-dimensional space a and p b , and the interpolation parameter l, use the following formula to obtain the sampling point p between two high-dimensional points l :

[0056] p l =0.25*(1-l)·p a +0.75*l·p b ;

[0057] Among them, p l is the sampling point location.

[0058] Specifically, interpolation method: interpolation is a method of estimating or predicting an unknown point between two known points. In this embodiment, the light processing unit uses an interpolation method to interpolate two high-dimensional points p in a high-dimensional space. a and p b Interpolate between them to obtain the location of the sampling points.

[0059] Interpolation parameter: The interpolation parameter l is used to control the degree of interpolation and is usually in the range of [0,1]. When l = 0, the sampling point p l Located in p a When l = 1, the sampling point p l Located in p b Place.

[0060] The principle of the interpolation method is to use two known high-dimensional points p a and p b , and the interpolation parameter l, calculate the position p between them l This process is actually to convert p a and pb The interpolated point is estimated so that it can be appropriately between these two points in the high-dimensional space.

[0061] In practical applications, interpolation methods can be used to generate a series of sampling points between two high-dimensional points. These points can be used to simulate the changing trend of the light path in high-dimensional space, so as to more accurately calculate the propagation and fusion effects of light in steps such as energy propagation simulation.

[0062] In summary, the method for interpolating the light path in high-dimensional space is to use the interpolation parameter l to calculate the sampling point position between two high-dimensional points by weighted average. This method is used in the real-time 3D model and real scene fusion system to continuously sample the light path in high-dimensional space.

[0063] Embodiment 6: Based on the previous embodiment, the fusion unit calculates the energy distribution of each sampling point according to the sampling points obtained by interpolation, and the method of simulating the propagation of light by moving the high-dimensional points of the light path in the high-dimensional space includes:

[0064] For the sampling point position p l , calculate its energy distribution, let E(p l ) represents the energy distribution in high-dimensional space, which is calculated by the following formula:

[0065]

[0066] Among them, f i is the sampling point position p l The corresponding high-dimensional point p a Or a high-dimensional point p b The feature vector of the object; is the adjustment parameter, a positive number less than 0.01.

[0067] Specifically, ||p a -f i || 2 and ||p b -f i || 2 :These two parts represent the sampling point p l To two high-dimensional points p a and p b The feature vector f of the object i The square of the distance represents the distance between the sampling point and the object. The smaller the distance in the denominator, the greater the energy distribution value.

[0068] ||f i ||: This part represents the sampling point position p l The corresponding high-dimensional point pa or p b The feature vector f of the object i The modulus represents the size of the object's feature vector, which can also be understood as the brightness or energy of the object.

[0069] l+ : This part involves the interpolation parameter l, which controls the degree of interpolation. Plus This is to avoid the denominator being zero. When the interpolation parameter l is small, the denominator is small and the energy distribution value is large.

[0070] The goal of the formula is to calculate the energy distribution of the sampling point. This energy distribution can be understood as the intensity distribution of the sampling point in a high-dimensional space, which is used to simulate the propagation and interaction of light in space.

[0071] Relationship between energy distribution and distance: The denominator of the formula involves the distance from the sampling point to the object's feature vector. The smaller the distance, the smaller the denominator and the larger the energy distribution value. This shows that the closer the light path is to the object, the more concentrated its energy distribution is.

[0072] Energy distribution and object characteristics: The numerator of the formula involves the modulus of the object's feature vector, which represents the brightness or energy of the object. This reflects the degree to which the object's characteristics affect the propagation of light.

[0073] Influence of interpolation parameters: The interpolation parameter l controls the degree of interpolation. A smaller l will make the denominator smaller, resulting in a more concentrated energy distribution.

[0074] By calculating the energy distribution, the fusion unit simulates the propagation of light. The energy distribution reflects the energy intensity distribution of light in high-dimensional space. By moving the high-dimensional points of the light path in high-dimensional space, the propagation effect of light is simulated. The method of calculating the energy distribution based on the sampling points obtained by interpolation and simulating the propagation of light is to calculate the energy distribution for each sampling point by calculating the energy distribution formula, and then move the high-dimensional points of the light path in high-dimensional space to simulate the propagation effect of light. This method provides a basis for energy propagation simulation for the fusion unit in the real-time 3D model and real scene fusion system.

[0075] After obtaining the energy distribution, the propagation of light is simulated in high-dimensional space. In each time step dt, the energy is transferred to the next sampling point according to the energy distribution and the direction of the light:

[0076]

[0077] Among them, p' l (t+dt) is the position of the next sampling point after the time step dt, t is the time variable; let p l(t), is the time t, two high-dimensional points p in the high-dimensional space a and p b , the sampling point location under the interpolation parameter l.

[0078] Specifically, p' l (t+dt): This is the next sampling point position after time step dt. It indicates the new position that the light propagates from the current sampling point position within a time step dt after time t.

[0079] p l (t): This is the current sampling point position at time t, that is, the current position of the ray path.

[0080] E(P l ,t): This is the energy distribution of the current sampling point at time t. This energy distribution affects the propagation intensity of the light.

[0081] d xy : This is the direction vector of the light. According to the previous description, it is the inverse light direction vector calculated by the light processing unit.

[0082] ||d xy ||: This is the magnitude of the light's direction vector and is used to normalize the light's direction.

[0083] dt: This is the time step, which represents the time interval of the simulation.

[0084] The principle of this formula is to simulate the propagation process of light in high-dimensional space, where the energy distribution and light direction affect the intensity and direction of light propagation. l (t) and time t, obtain the energy distribution E(P l ,t). Energy distribution represents the energy intensity distribution at that location, that is, the distribution of energy in different directions. Using the direction vector d xy , calculate the direction and distance that the light should propagate. l ,t) is multiplied by the light direction vector to obtain the energy propagation intensity in the light direction. Multiply the propagation intensity calculated in step 2 by the time step dt, and then divide it by the modulus of the light direction vector || d xy ||. Add the obtained value to the current position p l (t), get the new sampling point position p' l (t+dt).

[0085] In summary, the principle of this formula is to combine energy distribution and light direction to simulate the propagation process of light in each time step dt in high-dimensional space. Energy distribution determines the propagation intensity of energy in different directions, and light direction determines the direction of light propagation. This process is used to simulate the propagation and interaction of light in space, providing a basis for the integration of real-time 3D models and real scenes.

[0086] Example 7: Based on the previous example, the method by which the fusion unit calculates the energy convergence value for each high-dimensional point of the light path in the high-dimensional space includes: for each high-dimensional point of the light path, calculating the total energy of all corresponding sampling points as the energy convergence value of the high-dimensional point.

[0087] Specifically, by calculating the energy convergence value, the system can better simulate the energy distribution and propagation effect of light in high-dimensional space. The energy convergence value can affect the calculation of the fusion color, thereby affecting the fusion effect of the 3D model and the real scene. A light path with a larger energy convergence value may have a greater impact on the final fusion color.

[0088] In summary, the method for calculating the energy convergence value of each high-dimensional point on the light path in the high-dimensional space is to accumulate the energy of all sampling points on the light path as the energy convergence value of the high-dimensional point. This step plays an important role in simulating light propagation and calculating fusion color.

[0089] Embodiment 8: Based on the previous embodiment, the feature vector f of the object i It is calculated by the following method:

[0090] Step 1: Detect all object regions in the real scene image, specifically including: preprocessing the real scene image using a feature conversion model to convert the real scene image into a frequency domain representation; identifying the region of interest in the frequency domain representation as the object region by applying a multi-scale filter group;

[0091] Specifically, the principle of multiscale filters is based on the feature changes of images at different scales. Since features such as objects, edges, and textures in images may show different appearances at different scales, multiscale filters capture features at different scales by using filter kernels of different sizes. In the frequency domain, multiscale filters usually use Gaussian functions to construct filter kernels of different scales. The Gaussian function is a bell-shaped curve with smooth characteristics, which is suitable for capturing features at larger scales. By adjusting the standard deviation of the Gaussian function, the scale of the filter kernel can be controlled, thereby achieving multiscale filtering.

[0092] Step 2: Calculate the feature vector of each object region as the feature vector f of each object i .

[0093] Embodiment 9: Based on the previous embodiment, the feature conversion model is expressed using the following formula:

[0094]

[0095] Among them, F(z,c) is the pixel value of the pixel with pixel coordinates (z,c) of the input image, (b,k) is the frequency domain coordinate, N is the image size; QFT(b,k) is the frequency domain representation.

[0096] Specifically, QFT(b,k): This represents the frequency domain representation value at position (b,k) in the frequency domain. It is obtained by performing a specific calculation on the pixels in the input image.

[0097] F(z,c): This is the pixel value at pixel coordinate (z,c) of the input image. It represents the intensity of the pixel at a certain location in the image.

[0098] (b,k): This is the frequency domain coordinate, used to indicate the location in the frequency domain.

[0099] N: This is the size of the image, usually the height or width of the image.

[0100] This is a complex exponential term whose angle is determined by the frequency domain coordinates (b,k) and the image coordinates (z,c). It corresponds to a transformation in the frequency domain.

[0101] The quantum Fourier transform (QFT) is a mathematical transform similar to the classical Fourier transform, but has important applications in quantum computing and quantum information. In this embodiment, the QFT is used to transform the input image from the spatial domain to the frequency domain to extract the frequency information of the image.

[0102] The principle of QFT is to convert the pixel values ​​(spatial domain) in the image into complex values ​​on the frequency domain coordinates. The advantage of this is that the frequency domain representation can better highlight the frequency components in the image, such as edges, textures, etc. In the formula, for each frequency domain coordinate (b, k), all pixel values ​​in the input image are weighted summed and phase modulated using complex exponential terms. In this way, the value of each frequency domain position (b, k) is calculated to form the representation of the image in the frequency domain.

[0103] The function of this formula is to convert the image from the spatial domain to the frequency domain representation, and to represent the features of the image by extracting the frequency information of the image. In an embodiment, the feature conversion model uses quantum Fourier transform to process the real-scene image, thereby obtaining the frequency domain representation for the subsequent application of multi-scale filters and object area detection. This frequency domain representation is widely used in image processing to analyze the frequency characteristics of images, and is of great significance for realizing the fusion of real-time 3D models and real scenes.

[0104] Specifically, the multi-scale filter bank identifies the region of interest Q-ROI in the frequency domain representation, and its formula is as follows:

[0105]

[0106] Among them, H n (z,c) is the frequency domain representation of the nth filter, and s is the scale parameter. After obtaining the object region, the object region will also be processed as follows:

[0107] Spatial transformation clustering uses spatial transformation (QST) and cluster analysis to group Q-ROIs to obtain potential object candidate sets. The QST formula is as follows:

[0108]

[0109] in, represents element-wise multiplication, F s (z,c) is the scale space function. Introduction of Virtual Convolutional Neural Network (VQ-CNN) A virtual convolutional neural network (VQ-CNN) is proposed, which has a structure similar to the traditional CNN, but introduces virtual convolution operation in the convolution layer. The convolution operation formula of VQ-CNN is as follows:

[0110]

[0111] Among them, Qconv i (b) represents the operation of the i-th virtual convolution kernel, b i is the bias term, and σ is the activation function. This series of formulas is used to identify the region of interest in the frequency domain representation and further process the object region. The role of the multi-scale filter bank to identify the region of interest (Q-ROI) is to capture image features from the frequency domain information and provide a basis for the generation of object candidate sets. Spatial transform clustering (QST) and virtual convolutional neural network (VQ-CNN) are used to perform more detailed analysis and feature extraction of object regions to identify potential object candidate sets. The whole process helps to extract and filter out information that may be related to actual objects, providing more accurate data for the results of the fusion of real-time 3D models and real scenes.

[0112] Object region proposal generation Using VQ-CNN, object region proposals can be generated. The formula is as follows:

[0113]

[0114] in, represents element-wise multiplication, F s(z,c) is a scale space function. Object region fusion can model the spatial consistency of object regions by introducing conditional random fields (Q-CRF). The formula is as follows:

[0115]

[0116] Among them, x represents the state of all object regions, s i (x) and s ij (x) is the feature of the corresponding unit and the adjacent unit, θ i and θ ij is the learned parameter and Z is the normalization term.

[0117] These formulas are used to generate and fuse object region proposals, thereby providing more accurate and consistent object location information. VQ-CNN combined with QST can generate preliminary candidate object region proposals, utilizing frequency domain information and spatial transformation. Q-CRF introduces spatial consistency modeling, which allows the relationship between object regions to be considered, thereby improving the accuracy and stability of object region fusion. This is of great significance for realizing object recognition and fusion in real-time 3D model and real scene fusion systems.

[0118] Perceptual non-maximum suppression (Q-NMS) In order to further reduce proposals, the perceptual non-maximum suppression method is introduced to ensure that the final detection results will not have too many repetitions. The formula is as follows:

[0119]

[0120] Among them, b represents the proposal after the fusion of all object regions. Step 8: Image reconstruction Through the inverse feature transform (IQFT), the processed object region can be remapped back to the original image space to obtain the image representation of the detection result:

[0121]

[0122] Among them, VQ-NMS(b,k) is the pixel value of the object area after Q-NMS processing.

[0123] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0124] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0125] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0126] The present invention is described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A real-time 3D model and real scene fusion system based on a smartphone camera, characterized in that: The system comprises: a camera configured to acquire a real-scene image; a real-scene image modeling unit configured to model the real-scene image into a high-dimensional data structure, wherein each object in the real-scene image is represented as a high-dimensional feature vector; a light processing unit configured to perform reverse light projection on each pixel in the real-scene image to determine the light path of the pixel, and map the light path to a high-dimensional space to form a high-dimensional coordinate system; each light path is mapped to a high-dimensional point, and for each high-dimensional point of the light path, the object to which it belongs is determined by matching the feature vector of an object in the most similar scene, and the light path is interpolated in the high-dimensional space by an interpolation method to obtain a sampling point position; and a fusion unit configured to calculate each sampling point position according to the interpolated sampling point position. energy distribution of the sample points, by moving the high-dimensional points of the light path in the high-dimensional space to simulate the propagation of light, calculating the energy convergence value in the high-dimensional space for each high-dimensional point of the light path, and obtaining the fused color value of each pixel position by back-projecting the energy convergence value back to the two-dimensional image space, and displaying the back-projected fused color value in the 3D model to achieve the fusion of the 3D model and the real scene; the method in which the light processing unit performs reverse light projection on each pixel in the real scene image to determine the light path of the pixel includes: obtaining the internal parameters and external parameters of the camera, converting the pixel position (x, y) into the coordinates (u, v, w) in the camera coordinate system; using the following formula to calculate the reverse light direction vector d from the camera position to the pixel position (x, y) xy : The starting point of the reverse light is (u, v, w); the light processing unit maps the light path to a high-dimensional space to form a high-dimensional coordinate system, including: mapping the light path of each pixel to a high-dimensional space to form a high-dimensional coordinate system, and mapping the direction vector d of each light path to a high-dimensional space to form a high-dimensional coordinate system. xy Map it into a part of the high-dimensional space, map the pixel position (x, y) into another part of the high-dimensional space, and get the representation p of each light path in the high-dimensional space. xy , where p xy =[d xy ,x,y]; The method for the light processing unit to determine the object to which the high-dimensional point of each light path belongs by matching the feature vector of the object in the most similar scene includes: for each high-dimensional point p of the light path xy , use the following formula to calculate the feature vector f of each object i The similarity between: where · represents the dot product of the vectors; for each high-dimensional point p of the light path xy , calculate its similarity with all objects, and select the object corresponding to the maximum similarity as the object to which it belongs; the light processing unit interpolates the light path in the high-dimensional space through the interpolation method to obtain the sampling point position method includes: Suppose there are two high-dimensional points p in the high-dimensional space a and p b , and the interpolation parameter l, use the following formula to obtain the sampling point p between two high-dimensional points l : p l =0.25*(1-l)·p a +0.75*l·p b ; Among them, p l is the sampling point position; the fusion unit calculates the energy distribution of each sampling point according to the sampling point position obtained by interpolation, and the method of simulating the propagation of light by moving the high-dimensional point of the light path in the high-dimensional space includes: For the sampling point position p l , calculate its energy distribution, let E(p l ) represents the energy distribution in high-dimensional space, which is calculated by the following formula: Among them, f i is the sampling point position p l The corresponding high-dimensional point p a Or a high-dimensional point p b The feature vector of the object; is the adjustment parameter, a positive number less than 0.01; After obtaining the energy distribution, the propagation of light is simulated in high-dimensional space. In each time step dt, the energy is transferred to the next sampling point according to the energy distribution and the direction of light: Among them, p' l (t+dt) is the position of the next sampling point after the time step dt, t is the time variable; let p l (t), is the time t, two high-dimensional points p in the high-dimensional space a and p b , the sampling point position under the interpolation parameter l; the method for the fusion unit to calculate the energy convergence value in the high-dimensional space for each high-dimensional point of the light path includes: for each high-dimensional point of the light path, calculating the total energy of all corresponding sampling points as the energy convergence value of the high-dimensional point; The feature vector f of the object i It is calculated by the following method: Step 1: Detect all object regions in the real scene image, specifically including: preprocessing the real scene image using a feature conversion model to convert the real scene image into a frequency domain representation; identifying the region of interest in the frequency domain representation as the object region by applying a multi-scale filter group; Step 2: Calculate the feature vector of each object region as the feature vector f of each object i ; The feature conversion model is expressed using the following formula: Among them, F(z,c) is the pixel value of the pixel with pixel coordinates (z,c) of the input image, (b,n) is the frequency domain coordinate, N is the image size; QFT(b,n) is the frequency domain representation.

Citation Information

Patent Citations

  • Image projection method and device, equipment and storage medium

    CN109961522A

  • Bimodal volume data region-of-interest fusion method based on ray casting

    CN113421323A

  • Virtual rock core imaging equipment and method based on foresight panoramic drilling camera shooting

    CN116804359A

  • Method and system for fusing image data

    US20200005520A1

  • Image rendering system with local, adaptive estimation of incident diffuse energy

    US5488700A