AR content projection method, system, electronic device and storage medium
By using the rendering engine in the AR terminal to synchronize the real scene image and virtual content in the space-time and space-time synchronization of real scene images and virtual content, and using WebRTC technology to realize wireless streaming, the space-time anchoring problem in AR screen projection technology is solved, the user experience and picture quality is improved, and the additional Hardware requirements.
Patent Information
- Application Number
- CN202211652608.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-21
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-12-21
AI Technical Summary
The existing AR screen projection technology has the problem of poor space-time anchoring effect, which leads to poor user experience and requires additional hardware equipment such as microcameras, which increases cost and installation complexity.
By using the camera module to generate real scene images at the first terminal, and using the rendering engine to synchronize time and space with the virtual content, combining WebRTC technology to establish a wireless streaming channel, directly sending augmented reality screen to the second terminal, realizing synchronous screen projection of AR content .
Without additional hardware devices, the problem of space out of synchronization during the AR screen projection method is solved, the quality and user experience of the AR screen projection picture are guaranteed, and the equipment installation and cost are simplified.
Smart Images

Figure CN116193182B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of augmented reality, and in particular to a method, system, electronic device and storage medium for projecting AR content. Background Art
[0002] With the continuous development of AR (Argument Reality) technology, users generally hope to be able to record and share their own AR content. How to record AR content and project it to other terminals is an urgent need in this field.
[0003] Among the related technologies, the most widely used is the screen projection technology based on VR (Virtual Reality). Since VR content does not involve real content, there is no problem of time and space confusion caused by the superposition of virtual and real content. Compared with AR screen projection technology, it is easier to implement.
[0004] The AR screen projection content in the related technology is achieved by installing a micro camera on one side of the AR terminal device, using the micro camera to simultaneously capture the virtual image and real image in the display interface, and connecting to an external display device in a wired or wireless manner and sending the captured image, thereby realizing screen projection playback.
[0005] However, the above solution requires the addition of specific equipment (micro camera). This method not only requires additional hardware costs, but also has poor virtual image imaging effect when the micro camera shoots virtual images through the lens, and the installation angle of the micro camera makes it difficult to ensure the spatial consistency of virtual and real content, resulting in a poor user experience on the projection side. Summary of the invention
[0006] The embodiments of the present application provide a method, system, computer device and storage medium for projecting AR content, so as to at least solve the problem of poor spatiotemporal anchoring effect of the AR content projection method in the related art.
[0007] In a first aspect, an embodiment of the present application provides a method for projecting AR content, the method comprising:
[0008] The first terminal generates a real scene image through the camera module data stream, and superimposes the real scene image with virtual content through a rendering engine to generate an augmented reality image.
[0009] Wherein, if the first terminal obtains the posture data by tracking the camera, in the process of generating the augmented reality picture, the real scene picture and the virtual content are synchronized in time and space based on a preset process;
[0010] The first terminal establishes a wireless streaming channel with the second terminal through the WebRTC communication technology, directly sends the augmented reality image to the second terminal through the wireless streaming channel, and instructs the second terminal to synchronously display the augmented reality image.
[0011] In some of the embodiments, when the first terminal simultaneously obtains the posture data and captures the real scene image through the RGB camera, the real scene image is directly superimposed with the virtual content through the rendering engine to generate an augmented reality picture.
[0012] In some embodiments, the real scene image and the virtual content are synchronized in time and space based on a preset process, including:
[0013] In the process of superimposing the real scene image with the virtual content through the rendering engine, the real scene image and the virtual content are synchronized in time.
[0014] And, according to the posture data collected by the tracking camera, the real scene image and the virtual content are spatially synchronized.
[0015] In some embodiments, time synchronization is performed on the real scene image and the virtual content, including:
[0016] Performing synchronous exposure through a hardware device to synchronize the real scene image and the virtual content in time, or,
[0017] The time mark information of the real scene image and the virtual content is obtained, and based on the time mark information, the time synchronization of the real scene image and the virtual content is achieved through data stream soft synchronization.
[0018] In some embodiments, spatially synchronizing the real scene image and the virtual content according to the posture data collected by the tracking camera includes:
[0019] Acquire external parameters between the tracking camera and the RGB camera;
[0020] According to the external parameters, the posture data of the tracking camera is converted to the RGB camera, and the tracking camera and the RGB camera complete posture binding;
[0021] The virtual camera in the rendering engine is bound to the posture data collected by the tracking camera, and the virtual camera and the RGB camera complete posture binding;
[0022] After the virtual camera and the RGB camera complete posture binding, the real scene image and the virtual content are superimposed through a rendering engine, and the real scene image and the virtual content complete spatial synchronization.
[0023] In some embodiments, establishing a wireless streaming channel with a second terminal based on the WebRTC protocol includes:
[0024] The first terminal sends an SDP request to a signal server, and the signal server sends the SDP request to the second terminal after checking that the SDP request is qualified;
[0025] The second terminal generates an SDP answer in response to the SDP request, and forwards the SDP answer to the first terminal via the signal server;
[0026] After receiving the SDP answer, the first terminal sends the ICE candidate data to the signal server, and the signal server sends the ICE candidate data to the second terminal after checking that the ICE candidate data is qualified;
[0027] The second terminal generates second ICE candidate data in response to the ICE candidate data, and forwards the second ICE candidate data to the first terminal through the signal server;
[0028] The first terminal receives the second ICE candidate, and establishes the wireless streaming channel with the second terminal after checking that the second ICE candidate is qualified.
[0029] In some of the embodiments, after the first terminal establishes a wireless streaming channel with the second terminal, the first terminal directly sends the video stream and / or audio stream of the augmented reality image to the second terminal through the wireless streaming channel.
[0030] In a second aspect, an embodiment of the present application provides a screen projection system for AR content, the system comprising: a first terminal and a second terminal, wherein:
[0031] The first terminal is used to generate a real scene image through a camera module data stream, and to superimpose the real scene image with virtual content through a rendering engine to generate an augmented reality image, wherein if the first terminal obtains posture data through a tracking camera, in the process of generating the augmented reality image, the real scene image and the virtual content are synchronized in time and space based on a preset process;
[0032] The second terminal is used to establish a wireless streaming channel with the first terminal based on the WebRTC protocol, and receive the augmented reality picture sent by the first terminal through the wireless streaming channel, and display the augmented reality picture synchronously with the first terminal.
[0033] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.
[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.
[0035] Compared with the related art, the method for projecting AR content provided in the embodiment of the present application generates a real scene image through the camera module data stream at the first terminal, and superimposes the real scene image with the virtual content through the rendering engine to generate an augmented reality picture, wherein if the first terminal obtains the posture data through the tracking camera, in the process of generating the augmented reality picture, the real scene image and the virtual content are synchronized in time and space based on the preset process; further, the first terminal establishes a wireless streaming channel with the second terminal through the WebRTC technology, sends the augmented reality picture to the second terminal through the wireless streaming channel, and instructs the second terminal to display it synchronously. Through this application, the problem of time and space asynchrony of the AR projection method in the related art is solved, and the projection and playback of AR content is realized without the need for additional hardware devices, and the quality of the AR projection picture can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0037] Figure 1 is a schematic diagram of an application environment of a method for projecting AR content according to an embodiment of the present application;
[0038] Figure 2 is a flow chart of a method for projecting AR content according to an embodiment of the present application;
[0039] Figure 3 is a schematic diagram of establishing a wireless streaming channel based on WebRTC communication technology according to an embodiment of the present application;
[0040] Figure 4 is a schematic diagram of spatially synchronizing a real scene image and virtual content according to an embodiment of the present application;
[0041] Figure 5 is a schematic diagram of a process of establishing a streaming channel according to an embodiment of the present application;
[0042] Figure 6 is a structural block diagram of an AR content projection system according to an embodiment of the present application;
[0043] Figure 7 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0045] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.
[0046] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0047] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.
[0048] In this document, it is to be understood that the data involved may be a technical means for implementing a part of the present invention or other summary technical terms. For example, the terms may include:
[0049] AR (Argument Reality): Augmented reality is a technology that cleverly integrates virtual information with the real world. It simulates computer-generated text, images, three-dimensional models, videos and other virtual information and applies them to the real world. The two types of information complement each other to achieve "enhancement" of the real world.
[0050] VR (Virtual Reality): Virtual reality uses computers and other devices to create a virtual world with three-dimensional visual experience, making people in the virtual world feel as if they are in the real world.
[0051] Pose: position and attitude (orientation), for example: in two dimensions it is generally (x, y, yaw), in three dimensions it is generally (x, y, z, yaw, pitch, roll), including 6 degrees of freedom, 6of (6Degrees Of Freedom), the last three elements describe the attitude of the object, where yaw is the heading angle, rotating around the Z axis, pitch is the pitch angle, rotating around the Y axis, and roll is the roll angle, rotating around the X axis;
[0052] Tracking Camera: generally refers to the device / glasses SLAM camera, which is used to obtain posture data in real time;
[0053] Rendering Camera: generally refers to the virtual camera in the rendering engine;
[0054] WebRTC (Web Real-Time Communications): A web real-time communication technology that allows network applications or sites to establish peer-to-peer connections between browsers without the help of an intermediary to achieve the transmission of video streams and / or audio streams or any other data;
[0055] The AR content projection method provided in this embodiment can be applied to Figure 1 In the application environment shown, Figure 1 Schematic diagram of the application environment of the method for projecting AR content according to an embodiment of the present application. Figure 1 As shown, the user shoots the real scene through the first terminal 10, and the first terminal 10 uses the rendering engine to superimpose the real scene with the virtual content to generate an augmented reality picture with virtual and real superposition. Furthermore, the first terminal 10 establishes a wireless streaming channel that can directly communicate with the second terminal 12 through the message server 11, and sends the augmented reality picture directly to the second terminal 12 through the channel. Finally, the second terminal 12 and the first terminal synchronously display the augmented reality picture, and the synchronous screen projection and playback of AR content have been realized.
[0056] This application solution can be applied in a variety of scenarios. For example, when a user wears AR glasses to experience AR content, he or she can record the AR images he or she sees or creates and share them with other users. Or in medical and educational scenarios, an instructor wearing AR glasses can demonstrate to other students, and the students can see the virtual and real superimposed content displayed by the instructor in real time through their mobile phone screens or TV screens.
[0057] It should be noted that the above-mentioned first terminal 10 and the second terminal 12 can be head-mounted display devices, smart phones, tablet computers, laptops, smart TVs, etc. The second terminal in the figure is only a schematic diagram. In a specific scenario, the first terminal 10 can share the screen projection with multiple second terminals 12; the message server 11 is only used in the channel establishment phase, and the message server 11 does not need to participate in the data sending and receiving phase.
[0058] Figure 2 is a flow chart of a method for projecting AR content according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:
[0059] S201, the first terminal generates a real scene image through a camera module data stream, and superimposes the real scene image with virtual content through a rendering engine to generate an augmented reality image;
[0060] If the first terminal obtains the posture data by tracking the camera, in the process of generating the augmented reality picture, the real scene picture and the virtual content need to be synchronized in time and space based on the preset process;
[0061] It can be understood that if the first terminal obtains posture data through a tracking camera, it is most likely an AR headset device. The AR headset device obtains real-time posture data by running the SLAM algorithm through a tracking camera, and collects scene images through a common RGB camera. There are significant differences between the two cameras in terms of resolution and FOV.
[0062] For AR head-mounted display devices, since their posture data and scene images are collected by different cameras, it will cause the problem of difficult time and space synchronization. In this embodiment, the time and space synchronization of the two is performed through a preset process, specifically including synchronization on timestamps, and spatial synchronization based on the posture data obtained by the tracking camera;
[0063] In addition, if the first terminal is a smart terminal device such as a smartphone or tablet computer, this type of device is responsible for capturing scene images and acquiring posture data through the same camera. The scene images and posture data acquired by this type of device must be synchronized, so there is no need to perform time and space synchronization operations, and the real scene image and virtual content can be directly superimposed in the rendering engine.
[0064] The above rendering engine can be Unity3D or UE engine, which is not specifically limited in this application;
[0065] In addition, the specific overlay step is to set the projection matrix of the "virtual camera" according to the internal parameters of the RGB camera, and then render the texture of the real scene image on the background (virtual content) of the "virtual camera" through a shader.
[0066] S202, the first terminal establishes a wireless streaming channel with the second terminal through the WebRTC communication technology, directly sends the augmented reality picture to the second terminal through the wireless streaming channel, and instructs the second terminal to synchronously display the augmented reality picture.
[0067] Through the above step S201, a spatiotemporally aligned augmented reality picture is obtained; in this step, in order to more conveniently allow other users to experience the AR content in real time, the augmented reality picture needs to be sent to other terminals, such as smart phones, smart TVs, etc., through screen projection;
[0068] In order to make the screen projection widely applicable, in this embodiment, the screen projection is performed by wireless streaming. The specific data transmission method can be local area network WiFi communication, or 5G, external network WiFi communication, etc., which is not specifically limited in this embodiment.
[0069] Furthermore, in this embodiment, a P2P communication connection is established between multiple terminals based on WebRTC communication technology to implement a decentralized data transmission link to save bandwidth. Specifically:
[0070] Figure 3 is a schematic diagram of establishing a wireless streaming channel based on WebRTC communication technology according to an embodiment of the present application, such as Figure 3 As shown:
[0071] The signal server is responsible for establishing a communication connection between the sender and the receiver, exchanging SDP and network information between the two through WebSocket; after the connection is established, the sender can send data transmission such as video streams and audio streams directly to the receiver.
[0072] After the second terminal receives the above-mentioned augmented reality picture, it parses and plays the video stream and audio stream of the augmented reality picture in real time, thereby realizing real-time recording and screen projection of AR content; it should be noted that the web interface displays the above-mentioned augmented reality picture, and the display style and interactive control method of the web interface can be customized.
[0073] Through the above steps S201 to S202, compared with the method of shooting the augmented reality picture on the screen by a micro camera in the related art, the present application generates a real-scene background image through the camera module data stream, and generates a real-time virtual-reality superimposed video stream in combination with the virtual camera of the rendering engine. At the same time, the WebRTC technology is used to push the video stream to other clients in real time. The receiving user can experience the virtual-reality superimposed AR picture of the party in real time, which solves the problem of spatial asynchrony in the AR screen projection method in the related art, and realizes the screen projection and playback of AR content without the need for additional hardware devices.
[0074] In some of the embodiments, considering that the AR head display device collects posture data and real scene images through different cameras, which may cause the problem of spatiotemporal anchoring confusion in the subsequent overlay rendering process, therefore, in this embodiment, the real scene image and the virtual content are synchronized in time and space based on a preset process, specifically including:
[0075] first:
[0076] In the process of overlaying the real scene image with the virtual content through the rendering engine, the real scene image and the virtual content are synchronized in time. There are many ways to synchronize the time, for example:
[0077] Synchronous exposure is performed through hardware devices to achieve time synchronization between real scene texture maps and virtual content. This method usually has a delay accuracy of <1ms;
[0078] Alternatively, the data stream soft synchronization technology is used to obtain the time stamp information of the real scene image and the virtual content, and synchronize the time of the real scene image and the virtual content based on the time stamp information. This method usually has a delay accuracy of <10ms.
[0079] Specifically, the method for time synchronization can be flexibly selected according to the needs of the actual application scenario, and this is not specifically limited in this embodiment.
[0080] Secondly:
[0081] In addition to time synchronization, it is also necessary to process the spatial synchronization between the two cameras, that is, to synchronize the real scene image and the virtual content in space according to the pose data collected by the tracking camera. Figure 4 is a schematic diagram of spatially synchronizing a real scene image and virtual content according to an embodiment of the present application, such as Figure 4 As shown, the process specifically includes the following steps:
[0082] S401, obtaining external parameters between the tracking camera and the RGB camera;
[0083] S402, converting the posture data of the tracking camera to the RGB camera according to the external parameters, and completing posture binding between the tracking camera and the RGB;
[0084] The specific method of obtaining the external parameters between the two cameras is irrelevant to the invention of the present application and is a conventional technical means in the art, so it will not be described in detail in this embodiment.
[0085] In addition, according to the external parameters between the two cameras, the pose data under one camera is transformed to that under another camera, thereby completing the binding of the pose data between the two cameras. Those skilled in the art also know the specific implementation details, so they will not be repeated here.
[0086] S403, the virtual camera in the rendering engine is bound to the posture data collected by the tracking camera, and the virtual camera and the RGB camera complete the posture binding;
[0087] S404, after the virtual camera and the RGB camera complete the posture binding, the real scene image and the virtual content are superimposed through the rendering engine, and the real scene image and the virtual content are spatially synchronized.
[0088] It can be understood that after the RGB camera sends the real-scene texture map to the rendering engine, the rendering engine will set the projection matrix of the rendering camera through the internal parameters of the RGB camera. Further, according to the projection matrix, the real-scene texture is rendered on the background of the virtual camera through Shader. At this time, the rendering camera can see the AR virtual-real overlay content anchored in time and space.
[0089] In some of the embodiments, in order to make it easier for other users to experience the AR content in real time, it is necessary to display the AR content to other clients through screen projection.
[0090] Specifically, the first terminal generating the AR content and the second terminal on the receiving side can establish a wireless streaming channel through the WebRTC communication technology; after the channel is established, the first terminal directly sends the augmented reality picture to the second terminal through the wireless streaming channel, and instructs the second terminal to synchronously display the augmented reality picture;
[0091] Figure 5 is a flow chart of establishing a streaming channel according to an embodiment of the present application, such as Figure 5 As shown, establishing a wireless streaming channel with the second terminal based on the WebRTC protocol includes:
[0092] The first terminal sends an SDP request to the signal server, and the signal server sends the SDP request to the second terminal after checking that the SDP request is qualified;
[0093] The second terminal generates an SDP answer in response to the SDP (Session Description Protocol) request and forwards it to the first terminal through the signaling server;
[0094] The first terminal sends ICE candidate data to the signal server, and the signal server sends it to the second terminal after checking that the ICE (Interactive Connectivity Establishment) candidate data is qualified;
[0095] The second terminal generates a second ICE candidate in response to the ICE candidate, and forwards it to the first terminal through the signaling server;
[0096] The first terminal receives the second ICE candidate, and after checking that the second ICE candidate is qualified, establishes a wireless streaming channel with the second terminal.
[0097] This embodiment also provides a method for projecting AR content. The system is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been given will not be repeated. As used below, the terms "module", "unit", "sub-unit", etc. can implement a combination of software and / or hardware for predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0098] Figure 6 is a structural block diagram of an AR content projection system according to an embodiment of the present application, such as Figure 6 As shown, the system includes: a first terminal 60 and a second terminal 61, wherein:
[0099] The first terminal 60 is used to generate a real scene image through the camera module data stream, and to superimpose the real scene image with the virtual content through the rendering engine to generate an augmented reality image, wherein if the first terminal obtains the posture data through the tracking camera, the real scene image and the virtual content are synchronized in time and space based on a preset process during the process of generating the augmented reality image;
[0100] The second terminal 61 is used to establish a wireless streaming channel with the first terminal based on the WebRTC protocol, receive the augmented reality picture sent by the first terminal through the wireless streaming channel, and display the augmented reality picture synchronously with the first terminal.
[0101] The AR content projection system provided in this application can adapt different video codecs to the hardware of different platforms. If there is hardware support (such as GPU), hard encoding and decoding can be enabled to obtain better encoding and decoding performance.
[0102] This system can transmit video streams in real time without auxiliary devices such as micro cameras, and has the characteristics of low latency and high frame rate. Finally, through the Web front end of the terminal on the other side, the AR video stream data is displayed on the display page through the video control. Among them, the interactive control interface of the web end and the application end can be designed and developed according to the needs.
[0103] In one embodiment, Figure 7 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 7 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 7As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected by an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, and the computer program is executed by the processor to implement a screen projection method based on AR content, and the database is used to store data.
[0104] Those skilled in the art will understand that Figure 7 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0106] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A method for projecting AR content, characterized in that: The method comprises: The first terminal generates a real scene image through the camera module data stream, and superimposes the real scene image with virtual content through a rendering engine to generate an augmented reality image. Among them, if the first terminal obtains posture data through a tracking camera, in the process of generating an augmented reality picture, the real scene image and the virtual content are synchronized in time and space based on a preset process; when the first terminal simultaneously obtains posture data and shoots a real scene image through an RGB camera, the real scene image and the virtual content are directly superimposed through a rendering engine to generate an augmented reality picture; The first terminal establishes a wireless streaming channel with the second terminal through the WebRTC communication technology, directly sends the augmented reality image to the second terminal through the wireless streaming channel, and instructs the second terminal to synchronously display the augmented reality image.
2. The method according to claim 1, characterized in that: The real scene image and the virtual content are synchronized in time and space based on a preset process, including: In the process of superimposing the real scene image with the virtual content through the rendering engine, the real scene image and the virtual content are synchronized in time. And, according to the posture data collected by the tracking camera, the real scene image and the virtual content are spatially synchronized.
3. The method according to claim 2, characterized in that The real scene image and the virtual content are synchronized in time, including: Performing synchronous exposure through a hardware device to synchronize the real scene image and the virtual content in time, or, The time mark information of the real scene image and the virtual content is obtained, and based on the time mark information, the time synchronization of the real scene image and the virtual content is achieved through data stream soft synchronization.
4. The method according to claim 1, characterized in that According to the posture data collected by the tracking camera, spatially synchronizing the real scene image and the virtual content includes: Acquire external parameters between the tracking camera and the RGB camera; According to the external parameters, the posture data of the tracking camera is converted to the RGB camera, and the tracking camera and the RGB camera complete posture binding; The virtual camera in the rendering engine is bound to the posture data collected by the tracking camera, and the virtual camera and the RGB camera complete posture binding; After the virtual camera and the RGB camera complete posture binding, the real scene image and the virtual content are superimposed through a rendering engine, and the real scene image and the virtual content complete spatial synchronization.
5. The method according to claim 1, characterized in that Establishing a wireless streaming channel with the second terminal based on the WebRTC protocol includes: The first terminal sends an SDP request to a signal server, and the signal server sends the SDP request to the second terminal after checking that the SDP request is qualified; The second terminal generates an SDP answer in response to the SDP request, and forwards the SDP answer to the first terminal via the signal server; After receiving the SDP answer, the first terminal sends the ICE candidate data to the signal server, and the signal server sends the ICE candidate data to the second terminal after checking that the ICE candidate data is qualified; The second terminal generates second ICE candidate data in response to the ICE candidate data, and forwards the second ICE candidate data to the first terminal through the signal server; The first terminal receives the second ICE candidate, and establishes the wireless streaming channel with the second terminal after checking that the second ICE candidate is qualified.
6. The method according to claim 5, characterized in that After the first terminal establishes a wireless streaming channel with the second terminal, the first terminal directly sends the video stream and / or audio stream of the augmented reality image to the second terminal through the wireless streaming channel.
7. A screen projection system for AR content, characterized in that: The system comprises: a first terminal and a second terminal, wherein: The first terminal is used to generate a real scene image through a camera module data stream, and to superimpose the real scene image with virtual content through a rendering engine to generate an augmented reality image, wherein if the first terminal obtains posture data through a tracking camera, in the process of generating the augmented reality image, the real scene image and the virtual content are synchronized in time and space based on a preset process; when the first terminal simultaneously obtains posture data and shoots a real scene image through an RGB camera, the real scene image is directly superimposed with virtual content through a rendering engine to generate an augmented reality image; The second terminal is used to establish a wireless streaming channel with the first terminal based on the WebRTC protocol, and receive the augmented reality picture sent by the first terminal through the wireless streaming channel, and display the augmented reality picture synchronously with the first terminal.
8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Remote interaction method and system based on augmented reality and electronic equipment
CN113360057A
