A method and apparatus for generating a mixed reality scene

Abstract

The application relates to the technical field of virtual reality, and discloses a method and device for generating a fusion reality scene, which comprises the following steps: acquiring video image data of a multi-angle reality scene; selecting a three-dimensional model matched with the reality scene, and automatically fusing the reality scene with the three-dimensional model; performing seamless processing on the fusion boundary of the reality scene; and generating and displaying the fusion reality scene. The application has the beneficial effects of improving the fusion speed of the reality scene, improving the display effect of the fusion reality scene, effectively guaranteeing the authenticity and restoration of the fusion reality scene.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and more specifically to a method and apparatus for generating integrated real-world scenes. Background Technology

[0002] Virtual reality (VR), also known as virtual environment, virtual world, or artificial environment, refers to a technology that uses computers to generate a virtual world that can directly exert visual, auditory, and tactile sensations on participants and allow them to interactively observe and operate it.

[0003] Augmented Reality (AR) is a technology that calculates the position and angle of camera images in real time and adds corresponding images. It's a new technology that seamlessly integrates real-world and virtual-world information, aiming to overlay the virtual world onto the real world on a screen and allow for interaction. Currently, the application of these two technologies involves displaying virtual information within real-world information and enhancing the display, with the two types of information complementing and overlaying each other. For example, there are now head-mounted displays that composite the displayed world with computer graphics, achieving virtual reality display fusion and creating a three-dimensional display of the real world surrounding the headset.

[0004] While it is possible to achieve virtual and fusion displays using equipment such as head-mounted displays, there are too many limitations to the viewing experience. Specific devices are required to perceive the effects directly. For certain scenarios, such as directly integrating computer graphics into a 3D display space, this method cannot be achieved. Even if it could be achieved, the lack of breadth in the integration would result in a low level of realism in the presentation. Summary of the Invention

[0005] The present invention aims to provide a method and apparatus for generating integrated real-world scenes, so as to improve the realism of the results of virtual-real scene fusion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for generating integrated real-world scenes, comprising the following steps:

[0007] Step S1: Acquire video image data of the real scene from multiple angles;

[0008] Step S2: Select a 3D model that matches the real-world scene and automatically merge the real-world scene with the 3D model;

[0009] Step S3: Seamlessly process the fusion boundary of the real-world scene to generate and display the fused real-world scene.

[0010] The principle and advantages of this solution are as follows: In practical applications, video image data of the real-world scene is first acquired from multiple angles using a camera. Then, a corresponding 3D model is selected based on the type of the real-world scene, and the video image data of the real-world scene is imported into the 3D model, allowing the two to be automatically fused. Simultaneously, during the fusion process, the corresponding boundaries between the video and the model are seamlessly processed, resulting in a better overall fusion effect. This ensures that there are no gaps or imperfections at the connection points, avoiding visual discomfort and maximizing the reproduction of the actual real-world scene. Compared to existing technologies, this solution's advantages lie in its ability to automatically fuse the real-world scene and the 3D model, achieving faster fusion efficiency and seamless fusion, greatly improving the fused display effect and effectively guaranteeing the realism and fidelity of the fused real-world scene.

[0011] Preferably, as an improvement, in step S1, multiple cameras at different angles are used to acquire real-world scenes from multiple different angles, and the acquired real-world scene data is sorted according to the position order of the cameras.

[0012] Preferably, as an improvement, in step S2, a 3D model that is compatible with the current real scene is automatically matched from the 3D model library, and feature points of the real scene are identified, and the model is improved by targeted edge-setting based on the feature points.

[0013] Preferably, as an improvement, when acquiring real-world scenes, the camera's shooting direction is dynamically adjusted according to the changing patterns of human eye gaze, and the corners of the images acquired by the camera are processed according to the characteristics of images acquired by the human eye.

[0014] Preferably, as an improvement, in step S3, when seamlessly processing the fusion boundary, video overlay and cropping technology is used to unify the fusion boundary.

[0015] Preferably, as an improvement, when acquiring video images of real-world scenes, each pixel in the image is assigned an initial velocity vector to form image motion, and the image is dynamically analyzed based on the velocity vector characteristics of each pixel.

[0016] Preferably, as an improvement, it is determined whether there are moving objects in the image. If there are no moving objects in the image, the optical flow vector changes continuously throughout the entire image area. If there are moving objects in the image, there is relative motion between the real target scene and the image background. The velocity vector formed by the moving object is different from the velocity vector of the neighboring background, thereby detecting the moving object and its position. Finally, the scaling ratio of the 3D model is adjusted according to the new position change to complete the fusion of the real scene.

[0017] Preferably, as an improvement, during the process of merging the real scene and the 3D model, the size of each object in the real scene is first compared with the size of the corresponding objects in the 3D model, and the size ratio of the objects in the real scene is adjusted to be consistent with that in the 3D model. Then, the real scene is overlaid on the 3D model to merge the two.

[0018] Preferably, as an improvement, after the real scene and the 3D model are merged, the shooting angle of the camera is dynamically adjusted according to the user's display needs, and the real-time adjusted image is correspondingly changed into the merged display image.

[0019] The present invention also provides an apparatus for generating a fused real-world scene, comprising a microprocessor, and a camera module, a fusion module, and a display module respectively connected to the microprocessor; the camera module includes multiple 3D cameras; the microprocessor includes a storage unit, a 3D model library, and a data processing unit; the fusion module is used to fuse the acquired real-world scene with the corresponding 3D model; and the display module is used to display the fused real-world scene in real time.

[0020] Using this device and the aforementioned method for generating and integrating real-world scenes, the integration of virtual reality scenes can be completed automatically and quickly, and the accuracy and realism of the integrated scene can be effectively improved. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a method for generating integrated real-world scenes according to the present invention.

[0022] Figure 2 This is a schematic diagram of a first embodiment of the device for generating integrated real-world scenes according to the present invention. Detailed Implementation

[0023] The following detailed description illustrates the specific implementation method:

[0024] The labels in the accompanying drawings include: microprocessor 1, camera module 2, fusion module 3, display module 4, storage unit 5, 3D model library 6, and data processing unit 7.

[0025] Example 1:

[0026] This embodiment is basically as shown in the appendix. Figure 1 As shown: A method for generating integrated real-world scenes includes the following steps:

[0027] Step S1: Acquire video image data of the real scene from multiple angles;

[0028] Step S2: Select a 3D model that matches the real-world scene and automatically merge the real-world scene with the 3D model;

[0029] Step S3: Seamlessly process the fusion boundary of the real-world scene to generate and display the fused real-world scene.

[0030] Specifically, in step S1, multiple cameras installed at different angles in the real scene are used to acquire video image data of the real scene from different perspectives. The acquired real scene is then sorted according to the position order of the cameras, which facilitates the subsequent fusion of the sorted real scene with the pre-set 3D model, thereby improving the fusion efficiency.

[0031] Specifically, in step S2, the system automatically matches a 3D model from the 3D model library 6 that is compatible with the current real-world scene. Simultaneously, it identifies feature points of the real-world scene and makes targeted edge-correction improvements to the model based on these feature points. Furthermore, the aforementioned 3D model is automatically generated by the system and can automatically adapt to the perspective of the currently captured video footage of the real-world scene, thereby effectively reducing the difficulty of video fusion and improving fusion efficiency.

[0032] Specifically, when acquiring video data of real-world scenes, the camera's shooting direction is dynamically adjusted according to the changing patterns of human eye gaze, and the edges and corners of the video images captured by the camera are processed according to the characteristics of images acquired from the human eye's perspective. For example, the edges and corners of the video images are flattened, and the perspective is adjusted to a wide-angle mode for acquiring images of real-world scenes.

[0033] Specifically, in step S3, when seamlessly processing the fusion boundary, video overlay and cropping technology is used to unify the fusion boundary. By processing the boundary, the fusion effect of the real scene can be improved, bringing an ultimate visual experience to the human eye.

[0034] Specifically, each pixel in the acquired real-world scene image is assigned an initial velocity vector to form image motion. The image is then dynamically analyzed based on the velocity vector characteristics of each pixel, generating a modulator for real-world scene fusion. This not only improves the fusion efficiency but also ensures the realism and fidelity of the fused scene. Based on this, this method can quickly acquire target real-world scene video data while ensuring the validity of the acquired data, maximizing its consistency with the actual scene and guaranteeing a high degree of matching with the fusion model. Furthermore, during basic video data acquisition, adjusting the camera angle and acquisition order allows for direct matching with the corresponding 3D model. During fusion, seamless processing is performed on video boundaries to ensure the fused scene has no abrupt boundaries, preventing unpleasant visual effects.

[0035] When acquiring video images of real-world scenes, each pixel in the image is assigned an initial velocity vector to form image motion, and the image is dynamically analyzed based on the velocity vector characteristics of each pixel. By setting the velocity vector, the image change points in the real-world scene image can be accurately analyzed, thus enabling precise acquisition of the real-world scene image. This facilitates subsequent processing of the real-world scene, resulting in a better fit with the selected 3D model, improving the fusion effect, and to a certain extent enhancing the realism and fidelity of the fused scene. This step also improves the efficiency of virtual reality scene fusion to a certain extent, maximizing the realism of the fused product while ensuring the fusion effect, and faithfully reproducing the real-world scene from the current perspective in the 3D model.

[0036] Specifically, after image motion is generated, the system then determines whether there are moving objects in the image. If there are no moving objects, the light loss is continuously varying across the entire image area. If there are moving objects, there is relative motion between the real target scene and the image background. The velocity vector formed by the moving object will inevitably differ from the velocity vector of the neighboring background, thus detecting the moving object and its position. Based on this, this system can pre-plan the video acquisition angle and the processing mode of the real-world scene video image boundaries from the very beginning of the video data acquisition stage, thereby controlling the quality of the video data from the outset and facilitating subsequent fusion with the 3D model.

[0037] Specifically, based on the user's specific display needs, the camera's shooting angle will be dynamically adjusted during the fusion process of the real scene and the 3D model, and the real-time image will be changed accordingly in the fused display screen. This will meet the fusion needs of different users and different display requirements, and produce a more complete and adaptable fusion product.

[0038] Specifically, in the process of generating and integrating real-world scenes, the selection of real-world scenes and the setting of integration parameters are dynamically adjusted according to the degree of matching between the 3D model and the scene. This is to improve the integration effect, enhance the model integration speed and the display effect after integration, and facilitate the adjustment of its display content and display rhythm. Furthermore, when importing video data, the video is first compressed to half the original data size, and then the parameters of the 3D model are adjusted. By integrating the model with the real-world scene data, the display effect of the overall integration product is improved.

[0039] As attached Figure 2 As shown, this solution also provides an apparatus for applying the above-described method for generating fused reality scenes, including a microprocessor 1, and a camera module 2, a fusion module 3, and a display module 4 respectively connected to the microprocessor 1;

[0040] The camera module 2 includes multiple 3D cameras;

[0041] The microprocessor 1 includes a storage unit 5, a 3D model library 6, and a data processing unit 7. The storage unit 5 is used to store the acquired data and system cache data; the 3D model library 6 has multiple 3D modules for different scenes pre-set; the data processing unit 7 is used to initially process the acquired real-world scene video data.

[0042] The fusion module 3 is used to fuse the collected real-world scene with the corresponding 3D model;

[0043] The display module 4 is used to display the merged real-world scene in real time.

[0044] The specific implementation process of this embodiment is as follows:

[0045] The first step is to acquire 3D images of the real-world scene to be fused. This is done by using multiple cameras installed in a distributed manner at different angles to capture video images of the actual scene. The captured images are then preprocessed, with the clarity of the video images being improved and noise reduction being performed. Finally, the video images captured within the same time period are sorted according to the camera sequence number.

[0046] The second step is to retrieve a 3D model adapted to the current real-world scene from the system for fusion, and then import the processed video data of the real-world scene into the 3D model so that the 3D model and the real-world scene data can be automatically fused. During the fusion process, the fusion boundary is observed in real time, and the feature points of the real-world scene are identified. Based on the feature points, the model is improved in a targeted manner to provide the effect of fusion between the model and the display scene.

[0047] The third step involves seamlessly processing the fusion boundary of the real-world scene. Video overlay and cropping techniques are used to unify the fusion boundary, and each pixel in the image is assigned an initial velocity vector to form image motion. Based on the velocity vector characteristics of each pixel, the image is dynamically analyzed to determine whether there are moving objects in the image. If there are no moving objects in the image, the optical flow vector changes continuously throughout the entire image area. If there are moving objects in the image, there is relative motion between the real target scene and the image background, and the velocity vector formed by the moving object will necessarily be different from the velocity vector of the neighboring background, thereby detecting the moving object and its position.

[0048] The fourth step involves dynamically adjusting the camera's shooting angle based on the user's display needs during the fusion process of the real-world scene and the corresponding changes to the fused display screen. Then, the 3D model is re-fused with the new display scene video to obtain a new fused scene display screen.

[0049] With the development of VR and AR technologies, virtual reality integration technology has gradually become commonplace in daily life, such as 3D screens in shopping malls and AR games. The widespread application of this technology has brought many conveniences to people's work and life. However, there are still many shortcomings, such as insufficient integration with real-world scenes, low real-world reproduction, and the need for specific equipment to experience it intuitively. Therefore, to truly achieve the realistic reproduction of virtual reality integrated scenes, more detailed processing of the scene integration process is needed.

[0050] In this solution, multiple distributed 3D cameras are used to collect video image data of the target real-world scene from multiple angles. Then, based on the current real-world scene, the most suitable 3D model is automatically matched from the pre-stored 3D model library 6, and the real-world scene and 3D model are automatically merged to obtain a merged real-world scene image. On the other hand, during the automatic fusion process, the boundary between the real-world scene and the 3D model is seamlessly processed, thereby improving the fusion effect and making the boundary look more natural. In addition, the collected real-world scene can be compressed in a targeted manner during the fusion process, so as to better adapt it to the 3D model, further improving the fusion effect and effectively ensuring the authenticity and restoration of the merged real-world scene.

[0051] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A method for generating integrated real-world scenes, characterized in that: Includes the following steps: Step S1: Using distributed 3D cameras, acquire video image data of the real scene from multiple angles. When acquiring video images of the real scene, assign an initial velocity vector to each pixel in the image to form image motion, and perform dynamic analysis on the image based on the velocity vector characteristics of each pixel. After forming image motion, determine whether there are moving objects in the image. If there are no moving objects in the image, the optical flow vector changes continuously throughout the entire image area. If there are moving objects in the image, there is relative motion between the real target scene and the image background. The velocity vector formed by the moving object is different from the velocity vector of the neighboring background, thereby detecting the moving object and its position. This allows for adjusting the scaling ratio of the 3D model based on the new position changes when automatically fusing the real scene with the 3D model. When acquiring the real scene, dynamically adjust the shooting direction of the camera according to the changing pattern of human eye vision, and process the corners of the camera-acquired image according to the characteristics of human eye image acquisition. Step S2 involves selecting a 3D model that matches the real-world scene and automatically fusing the real-world scene with the 3D model. During this fusion process, the sizes of objects in the real-world scene and their corresponding objects in the 3D model are compared, and the size ratio of the objects in the real-world scene is adjusted to match that in the 3D model. Then, the real-world scene is overlaid onto the 3D model for fusion. In Step S2, a 3D model matching the current real-world scene is automatically matched from the 3D model library. The processed video data of the real-world scene is then imported into the 3D model, enabling automatic fusion of the 3D model and the real-world scene data. Simultaneously, feature points of the real-world scene are identified, and the model is improved with targeted edge-setting based on these feature points to enhance the fusion effect between the model and the displayed scene. The 3D model is automatically generated and automatically adapted to the perspective of the currently captured video footage of the real-world scene. Step S3: Seamlessly process the fusion boundary of the real scene to generate and display the fused real scene; in step S3, when seamlessly processing the fusion boundary, video overlay and cropping technology is used to unify the fusion boundary; the fusion boundary is the connection point between the corresponding boundary of the video and the model.

2. The method for generating a real-world scene according to claim 1, characterized in that: In step S1, multiple cameras at different angles are used to acquire real-world scenes from multiple different angles, and the acquired real-world scene data is sorted according to the position order of the cameras.

3. The method for generating a real-world scene according to claim 1, characterized in that: After the real-world scene is merged with the 3D model, the camera's shooting angle is dynamically adjusted according to the user's display needs, and the real-time adjusted image is correspondingly changed into the merged display.

4. A device for generating and integrating real-world scenes, characterized in that: A method for generating a fused real-world scene according to claim 1; the apparatus includes a microprocessor, and a camera module, a fusion module, and a display module respectively connected to the microprocessor; the camera module includes multiple 3D cameras; the microprocessor includes a storage unit, a 3D model library, and a data processing unit; the fusion module is used to fuse the acquired real-world scene with the corresponding 3D model; the display module is used to display the fused real-world scene in real time.

Images