Augmented reality virtual-real mixed rendering method, storage medium and mobile terminal

Abstract

This invention discloses an augmented reality (AR) hybrid rendering method, storage medium, and mobile terminal, relating to the field of internet-based AR technology. The method includes: acquiring real-world scene images from dual-channel camera input and scene pose data from a pose sensor; performing visual recognition on the real-world scene images; smoothing the camera view matrix based on the scene pose data to generate a smoothed camera view matrix, while keeping the camera projection matrix unchanged during the smoothing process; matching an existing virtual content model according to the physical dimensions of the target object; mixing the virtual content and real-world images, encoding the final frame, and finally streaming it to a live streaming platform. This invention achieves stable virtual material integration and zero perspective distortion in the background image by applying low-pass filtering smoothing based on inertial sensor fusion to the camera view matrix while strictly keeping the camera projection matrix unchanged.

Description

Technical Field

[0001] This invention relates to the field of Internet augmented reality technology, and in particular to augmented reality virtual-real hybrid rendering methods, storage media and mobile terminals. Background Technology

[0002] With the rapid development of mobile internet and new media platforms, mobile live streaming has become a mainstream content dissemination method. Currently, mobile live streaming applications on major platforms can typically only acquire and stream real-time footage captured by the camera, unable to integrate virtual content into the footage in real time. When broadcasters need to demonstrate interior design effects, product color changes, or other information, existing solutions rely on physical materials such as PC-based overlay layers or handheld signboards. The former is only suitable for fixed, seated broadcasting scenarios, while the latter offers a limited presentation method, neither of which can meet the enhanced expressive needs of mobile broadcasting scenarios.

[0003] To enhance the expressiveness of mobile live streaming, the industry is attempting to introduce AR technology into mobile live streaming scenarios. This involves using visual recognition algorithms to calculate camera pose and overlaying virtual content onto the real-world scene. However, existing mobile AR solutions face a key challenge: visual pose calculation is limited by factors such as floating-point arithmetic precision, changes in scene texture, and fluctuations in ambient lighting. The camera view matrix calculated for each frame exhibits slight fluctuations between frames, which, when directly used for rendering virtual content, can cause visible jitter and drift in the virtual materials.

[0004] While smoothing filters can suppress jitter in the overall rendered image, they also modify the pixels of the background, causing distortion of perspective and significantly reducing live stream quality. Therefore, eliminating jitter in virtual assets while maintaining the perspective of the real-world background remains completely unchanged is a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0005] The main objective of this invention is to provide an augmented reality hybrid rendering method, storage medium, and mobile terminal, aiming to solve the technical problems of existing mobile AR live streaming, where virtual materials jitter and drift due to inter-frame noise in visual pose calculation when integrating virtual content and real-world scenes; and the fact that directly smoothing the overall image would cause background perspective distortion.

[0006] To achieve the above objectives, this invention proposes an augmented reality hybrid rendering method, comprising: Acquire real-world scene images captured by a camera and collect scene posture data using an attitude sensor; By visually recognizing real-world scene images, the camera projection matrix, camera view matrix, and physical dimensions of the target object can be obtained. The camera view matrix is ​​smoothed based on scene pose data to generate a smoothed camera view matrix. During the smoothing process, the camera projection matrix remains unchanged so that the perspective relationship of the real scene image is not affected by the smoothing process. Based on the physical dimensions of the target object, match the virtual content data pre-generated and stored from the large video model, and generate the model matrix of the virtual content; By using a smoothed camera view matrix, camera projection matrix, and model matrix, virtual content is rendered and overlaid onto a real-world scene image to generate a hybrid virtual-real image. The mixed real and virtual images are encoded and then streamed to the live streaming platform.

[0007] Smoothing the camera view matrix includes: The original camera view matrix of the current frame visual recognition is interpolated with the smoothed camera view matrix of the previous frame at the matrix element level to filter out the small inter-frame jumps caused by floating-point noise in visual recognition.

[0008] The steps of performing element-wise weighted interpolation between the original camera view matrix of the current frame and the smoothed camera view matrix of the previous frame to filter out minute inter-frame jumps caused by floating-point noise in visual recognition specifically include: A first-order low-pass filtering method is used to perform element-wise weighted interpolation between the original camera view matrix of the current frame visual recognition and the smoothed camera view matrix of the previous frame; wherein, the filtering coefficients of the first-order low-pass filter are dynamically adjusted according to the device motion speed in the attitude data.

[0009] The device's attitude sensors include accelerometers and gyroscopes. The method also includes smoothing the camera view matrix. The device tilt angle data collected by the accelerometer and the device rotation angular velocity data collected by the gyroscope are fused to perform inter-frame attitude compensation and predictive interpolation on the camera view matrix.

[0010] The steps of matching virtual content data pre-generated and stored from a large video model with the physical dimensions of the target object, and generating the model matrix of the virtual content, specifically include: Based on the physical size and / or visual characteristics of the target object, the corresponding virtual content texture or video frame sequence is matched from the pre-generated virtual content library; using the physical size of the target object as a reference, the coordinates of the four corner vertices of the virtual plane are constructed, and the coordinates of the four corner vertices are multiplied by an independent proportional scaling factor to obtain the scaled virtual plane vertex coordinates, thereby generating the model matrix of the virtual content, wherein the virtual content texture or video frame sequence is mapped onto the virtual plane.

[0011] The rendering of virtual content overlays onto a real-world scene image is performed using the graphics rendering pipeline. This process specifically includes the following steps: Create a vertex buffer object and bind the vertex coordinate data and texture coordinate data of the virtual content to the vertex buffer object; Create and enable a programmable shader program, and pass the smoothed camera view matrix, the unchanged camera projection matrix, and the MVP matrix formed by combining the model matrix into the shader program to perform coordinate transformation on the vertices of the virtual content. The texture of the virtual content is bound, the texture is sampled by the shader program, and the pixels are colored according to the sampling results; Enable transparent blending mode based on source alpha and one minus source alpha, and enable deep testing; Execute drawing instructions to draw virtual content to the frame buffer, which is then overlaid onto the real-world scene image.

[0012] Real-world scene images are directly output as two video streams after acquisition: The first image data is sent to the graphics rendering pipeline as the underlying background image for rendering; the second image data is compressed and rotated before being sent to the visual recognition module for feature point extraction, image matching and planar tracking; when generating a virtual-real hybrid image, the first background layer is drawn first, and then a virtual content layer is drawn on top of the background layer.

[0013] After encoding the mixed real-virtual footage, it is streamed to the live streaming platform, specifically including: The hardware video encoder of the mobile terminal is invoked to encode the mixed virtual and real images in real time in H.264 or H.265 format, generating an encoded video stream; The encoded video stream is pushed to the live streaming platform server using RTMP or SRT transmission protocols.

[0014] A storage medium storing a computer program that, when executed by a processor, implements steps of an augmented reality hybrid rendering method for mobile live streaming.

[0015] A mobile terminal includes a camera, a posture sensor, a processor, and a memory. The memory stores a computer program, and the processor executes the computer program to implement the steps of an augmented reality virtual-real hybrid rendering method for mobile live streaming.

[0016] This invention achieves stable virtual material fitting and zero distortion of background perspective by applying low-pass filtering smoothing processing based on inertial sensor fusion to the camera view matrix while strictly locking the camera projection matrix unchanged. Furthermore, the scaling operation of virtual content is limited to the model matrix and completely decoupled from the camera matrix, so that scaling does not interfere with tracking accuracy. Finally, through layered hybrid rendering of the graphics pipeline, the background layer is drawn first and then the virtual layer is superimposed. Combined with hardware coding and real-time streaming, a high-quality, low-jitter AR virtual-real hybrid live broadcast effect is achieved on the mobile device. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the augmented reality hybrid rendering method of the present invention; Figure 2 This is a schematic diagram of the mobile terminal structure of the present invention.

[0020] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of the present invention and are not intended to limit the present invention.

[0022] To better understand the technical solution of the present invention, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0023] like Figure 1 As shown, Figure 1 This is a flowchart illustrating the augmented reality hybrid rendering method of the present invention.

[0024] An augmented reality hybrid rendering method includes: S10: Acquire real-world scene images captured by the camera and collect scene posture data through the attitude sensor; S20: Visual recognition of real-world scene images is used to obtain the camera projection matrix, camera view matrix, and physical dimensions of the target object. S30: Based on scene pose data, smooth the camera view matrix to generate a smoothed camera view matrix. During the smoothing process, the camera projection matrix remains unchanged so that the perspective relationship of the real scene image is not affected by the smoothing process. S40, based on the physical size of the target object, match the virtual content data pre-generated and stored by the large video model, and generate the model matrix of the virtual content; The S50 uses a smoothed camera view matrix, camera projection matrix, and model matrix to render and overlay virtual content onto real scene images, generating a hybrid virtual-real image. S60, the virtual-real mixed image is output through the camera driver layer of the mobile terminal so that the live streaming application running on the mobile terminal can capture and push it to the live streaming platform.

[0025] The implementation of the method of this invention depends on the following operating environment: System version: Android 12 and above; Graphics capability: The device must support OpenGL ES 2.0 and above (compatible with 3.0); Hardware peripherals: Built-in high-definition rear camera, accelerometer, gyroscope sensor; Supports real-time camera preview, continuous sensor data reporting, and normal creation and rendering of EGL graphics context.

[0026] In this embodiment, after the live stream starts, the mobile phone camera continuously captures images of the real-world scene at a rate of no less than 30 frames per second. Simultaneously, the accelerometer and gyroscope built into the mobile terminal report attitude data at a frequency of over 100Hz, including tilt angle changes and rotational angular velocity data of the device in space. Each captured frame of the real-world scene image outputs two video streams immediately upon acquisition: the first video stream is directly sent to the graphics rendering pipeline as the underlying background image and cached as texture data for the background layer; the second video stream is preprocessed by compression and rotation and then sent to the visual recognition module. The visual recognition module performs grayscale processing on the second video stream, extracts feature points from the image, and performs feature matching with a preset recognition target. Upon successful matching, an AR local coordinate system is established with the center of the recognition target as the world coordinate origin. Using camera intrinsic parameters and visual features for alignment, the camera projection matrix and camera view matrix are calculated, and the physical dimensions of the target object are returned. The camera projection matrix is ​​determined by the camera intrinsic parameters, near and far clipping planes, screen aspect ratio, and screen rotation angle, and its initial value remains unchanged throughout the subsequent processing.

[0027] The purpose of splitting the real-world scene image into two paths is to achieve decoupling: the background rendering path and the visual recognition path operate independently without blocking each other, ensuring that each path can process data with maximum efficiency. Locking the camera projection matrix is ​​a prerequisite for all subsequent processing. This matrix determines the perspective relationship of the camera image; once modified, the background image will suffer irreversible perspective distortion.

[0028] Smoothing the camera view matrix includes: The original camera view matrix of the current frame visual recognition is interpolated with the smoothed camera view matrix of the previous frame at the matrix element level to filter out the small inter-frame jumps caused by floating-point noise in visual recognition.

[0029] The steps of performing element-wise weighted interpolation between the original camera view matrix of the current frame and the smoothed camera view matrix of the previous frame to filter out minute inter-frame jumps caused by floating-point noise in visual recognition specifically include: A first-order low-pass filtering method is used to perform element-wise weighted interpolation between the original camera view matrix of the current frame visual recognition and the smoothed camera view matrix of the previous frame; wherein, the filtering coefficients of the first-order low-pass filter are dynamically adjusted according to the device motion speed in the attitude data.

[0030] In this embodiment, the original camera view matrix output by the visual recognition module for each frame exhibits slight numerical jumps between frames due to limitations in floating-point arithmetic precision and minor changes in scene texture. If the original matrix is ​​directly used for virtual content rendering, these jumps will manifest as visible jitter in the virtual material. Therefore, this method performs element-wise weighted interpolation between the original camera view matrix output by the visual recognition module in the current frame and the smoothed camera view matrix from the previous frame. This weighted interpolation employs a first-order low-pass filtering algorithm, with the filter coefficients dynamically adjusted based on the device's motion speed in the pose data: when the device is stationary or moving slowly, the filter coefficients are smaller, making the smoothing process more dependent on the current frame data and preventing trailing delays in the virtual material; when the device moves rapidly, the filter coefficients automatically increase, increasing the weight of historical frames to suppress pose noise amplification caused by rapid movement. The above smoothing process only applies to the camera view matrix and does not involve the camera projection matrix.

[0031] Matrix element-level weighted interpolation operates directly on each floating-point element of the camera view matrix, rather than post-processing blurring the entire rendered image. Therefore, smoothing operations do not introduce additional image quality loss. Dynamic coefficient adjustment resolves the inherent contradiction of fixed coefficient filtering: if the fixed coefficient is too large, virtual materials will respond sluggishly in static scenes; if it is too small, jitter suppression will be insufficient in dynamic scenes. Adaptive coefficients achieve the optimal smoothing effect across all scenes.

[0032] The device's attitude sensors include accelerometers and gyroscopes. The method also includes smoothing the camera view matrix. The device tilt angle data collected by the accelerometer and the device rotation angular velocity data collected by the gyroscope are fused to perform inter-frame attitude compensation and predictive interpolation on the camera view matrix.

[0033] In this embodiment, the accelerometer senses the tilt angle change of the device relative to the horizontal plane, and the gyroscope senses the rotational angular velocity of the device around each axis. The data from both are fused using complementary filtering or Kalman filtering to obtain the device's attitude estimate in three-dimensional space. When the real-world scene has a weakly textured surface, insufficient lighting, or the image is briefly occluded, the feature point tracking quality of the visual recognition module deteriorates, and the output camera view matrix may exhibit jumps or drifts. In this situation, inertial sensor fusion data is introduced to perform inter-frame attitude prediction and interpolation compensation on the camera view matrix, maintaining the pose continuity of the virtual material in the image.

[0034] Inertial sensors update data at a much higher frequency than visual recognition (typically above 100Hz compared to 30Hz), providing high-frequency attitude information to supplement visual output when visual tracking degrades. This vision-inertial fusion solution addresses the instability of pure vision AR in complex real-world scenarios and is a key means to maintain stable AR material alignment in mobile live streaming scenarios.

[0035] The specific steps for matching virtual content pre-generated from a large video model with the physical dimensions of the target object and generating a model matrix include: In this embodiment, the virtual content is not generated in real time by the graphics engine, but rather uses high-quality video clips or image textures pre-made using a large video model. These pre-generated virtual contents are stored locally on the mobile terminal or in a cloud cache, categorized by type or scene identifier. When the visual recognition module returns the physical dimensions of the target object and matching identifier information, the system selects the corresponding virtual content data from the virtual content library.

[0036] Furthermore, based on the physical dimensions of the target object, the coordinates of the four corner vertices of the virtual plane are constructed to form a rectangular plane with the same dimensions as the target object. Depending on the actual display requirements, the coordinates of the four corner vertices can be multiplied by an independent proportional scaling factor to obtain the scaled vertex coordinates of the virtual plane, thereby generating the model matrix of the virtual content. This virtual plane is used to map the pre-generated virtual content texture or update the video texture frame by frame. The scaling factor only applies to the model matrix, is completely decoupled from the camera matrix, and does not affect the background perspective.

[0037] In this embodiment, based on the physical size of the target object returned by the visual recognition module, the coordinates of the four corner vertices of a virtual plane are constructed in the AR local coordinate system, forming a rectangular plane with the same size as the recognized target. When the live streaming service needs to zoom in or out on the virtual content (for example, zooming in on interior design renderings to show details in a real estate live stream, or zooming out on car body paint to show the overall effect in a car live stream), the system applies an independent proportional scaling factor to the coordinates of the four corner vertices to obtain the scaled vertex coordinates of the virtual plane. This scaled vertex coordinates are then matched with the virtual content data pre-generated and stored from the large video model, and a model matrix of the virtual content is generated. This scaling factor only operates during the model matrix construction stage and does not couple with the camera projection matrix or camera view matrix. That is, the scaling operation only changes the size of the virtual content in the AR world, does not change the way the camera observes the world, and does not affect the perspective relationship of the background image.

[0038] The architecture design, which decouples the scaling factor from the camera matrix, makes adjusting the size of virtual content an independent, secure, and real-time responsive operation. During the live stream, the broadcaster can zoom in or out of the virtual content at any time, while the real-world background seen by the audience maintains its original perspective, avoiding image distortion or tracking shifts caused by scaling operations affecting the camera matrix in traditional solutions.

[0039] The rendering of virtual content overlays onto a real-world scene image is performed using the graphics rendering pipeline. This process specifically includes the following steps: Create a vertex buffer object and bind the vertex coordinate data and texture coordinate data of the virtual content to the vertex buffer object; Create and enable a programmable shader program, and pass the smoothed camera view matrix, the unchanged camera projection matrix, and the MVP matrix formed by combining the model matrix into the shader program to perform coordinate transformation on the vertices of the virtual content. The texture of the virtual content is bound, the texture is sampled by the shader program, and the pixels are colored according to the sampling results; Specifically, in this embodiment, the texture of the bound virtual content can be a static image pre-generated from the large video model or the current frame in the pre-generated video frame sequence. When the virtual content is a dynamic video, the rendering loop is responsible for updating the texture data according to the timestamp or frame index to ensure smooth playback of the virtual animation. The updating and mapping of the texture are completely controlled by the model matrix and the MVP matrix, and can inherit the stability brought by the smoothed camera view matrix.

[0040] Enable transparent blending mode based on source alpha and one minus source alpha, and enable deep testing; Execute drawing instructions to draw virtual content to the frame buffer, which is then overlaid onto the real-world scene image.

[0041] In this embodiment, once the smoothed camera view matrix, the unchanged camera projection matrix, and the model matrix are all prepared, the final rendering and compositing stage begins. This stage is executed using the graphics rendering pipeline, and the specific process is as follows: First, a vertex buffer object is created, and the vertex coordinate data and texture coordinate data of the virtual content are bound to this vertex buffer object for efficient GPU reading. Second, a programmable shader program is created and enabled, combining the smoothed camera view matrix, the unchanged camera projection matrix, and the model matrix in the order MVP = projection matrix × view matrix × model matrix to form the MVP matrix, which is then passed into a unified variable in the shader program. The vertex shader performs coordinate transformation on each vertex of the virtual content, completing the perspective projection from the AR 3D world coordinate system to the screen 2D coordinate system. Third, the texture image of the virtual content is bound, the texture is sampled by the fragment shader, and pixel coloring is performed based on the sampling results to obtain the color value of each pixel. The fourth step is to enable a transparent blending mode based on the source alpha and a subtraction of the source alpha. This causes the pixels in the semi-transparent areas of the virtual content to be weighted and blended with the background pixels according to their alpha values. Simultaneously, a depth test is enabled to ensure correct occlusion between the virtual content and the background image. The fifth step is to execute drawing instructions, writing the processed pixel data into the frame buffer. In the entire rendering process, the background layer corresponding to the first path of image data is drawn first, followed by the overlay drawing of the virtual content layer on top of the background layer, ultimately forming a complete frame of mixed reality and virtuality in the frame buffer.

[0042] The graphics rendering pipeline fully leverages the GPU hardware acceleration capabilities of mobile devices, ensuring that the rendering of each frame's virtual-real blending is completed within 16ms, meeting the real-time requirements of live streaming scenarios. The layered rendering order—background first, then virtual—combined with alpha blending and depth testing, ensures that virtual content is overlaid on the real-world image with correct layering and natural edge transparency, avoiding issues such as improper occlusion or harsh edges.

[0043] Furthermore, the final output of the mixed virtual and real images to the live streaming platform includes the following two forms: The first method is push streaming: by calling the hardware video encoder of the mobile terminal, the virtual and real mixed images are encoded in real time in H.264 or H.265 format to generate an encoded video stream; the encoded video stream is then pushed to the live streaming platform server through the RTMP or SRT transmission protocol.

[0044] In this embodiment, after the hybrid virtual-real image is generated in the frame buffer, it is sent to the video encoding module for real-time encoding. The encoding module calls the mobile terminal's hardware video encoder and utilizes the chip's built-in encoding acceleration unit to perform real-time compression of the image in H.264 or H.265 format, generating encoded video stream data. The encoded video stream is then pushed to the target live streaming platform's server address via the device's network module through the RTMP or SRT transmission protocol. After receiving the stream, the live streaming platform server distributes it to the viewers in the live stream room, and what the viewers see is the augmented live stream image containing AR virtual content.

[0045] By using a hardware encoder instead of software encoding, CPU load and power consumption are significantly reduced, enabling the phone to maintain stable operation, without overheating or lag, during long-term live streaming scenarios. H.264 and H.265 formats are universal encoding formats compatible with major live streaming platforms, while RTMP and SRT are industry-standard low-latency transmission protocols. The entire streaming solution is fully compatible with the existing live streaming ecosystem and can be used directly without any modifications to the platform.

[0046] The second approach involves a virtual camera: after the mixed real-virtual image is generated in the frame buffer, it is not directly encoded and streamed by this method. Instead, it is output through the camera driver framework at the bottom layer of the mobile terminal operating system. Specifically, a virtual camera device is registered in the Android system HAL layer or the Linux kernel layer, and the mixed real-virtual image frame data in the frame buffer is sent to the output pipe of the virtual device in real time in YUV or RGB format.

[0047] In this embodiment, when a live streaming application (such as Douyin or WeChat Video Channel) running on the same mobile terminal starts broadcasting, it will invoke the system's camera list. At this time, the virtual camera device can be selected by the user or invoked by default through system policies. The image captured by the live streaming app through the standard Camera API is the virtual-real hybrid image synthesized by this method. The subsequent encoding (such as H.264 / H.265) and streaming (such as RTMP) processes are completely completed within the live streaming app without the intervention of this method. This method ensures compatibility with any native live streaming app while allowing the rendering pipeline of this method to focus on image compositing and AR stabilization.

[0048] In this embodiment, the virtual camera frame output supports configurable resolution and frame rate to adapt to the requirements of different live streaming platforms; at the same time, since the output is raw frame data, the live streaming APP can adapt the bit rate according to the network conditions, avoiding the loss of image quality caused by double encoding.

[0049] The present invention also proposes a storage medium storing a computer program that, when executed by a processor, implements the steps of an augmented reality hybrid rendering method for mobile live streaming.

[0050] The computer program includes instructions for performing the following operations: acquiring real-world scene images captured by a camera and acquiring scene pose data via a pose sensor; obtaining a camera projection matrix, a camera view matrix, and the physical dimensions of a target object by performing visual recognition on the real-world scene images; smoothing the camera view matrix based on the scene pose data to generate a smoothed camera view matrix, while keeping the camera projection matrix unchanged during the smoothing process; matching virtual content data pre-generated and stored by a large video model according to the physical dimensions of the target object, and generating a model matrix for the virtual content; rendering and overlaying the virtual content onto the real-world scene image using the smoothed camera view matrix, camera projection matrix, and model matrix to generate a hybrid virtual-real image; and streaming the hybrid virtual-real image to a live streaming platform after video encoding.

[0051] When the processor loads and executes the computer program, the mobile terminal will complete the entire chain of processing from image acquisition, visual recognition, pose fusion smoothing, model matrix generation, virtual-real hybrid rendering to encoding and streaming in the above steps, thereby achieving a high-quality AR live streaming effect in the live streaming scenario with stable virtual material fitting and unchanged background perspective.

[0052] like Figure 2 As shown, Figure 2 This is a schematic diagram of the mobile terminal structure of the present invention; refer to Figure 2 The present invention also proposes a mobile terminal, including a camera 10, a posture sensor 20, a processor 30 and a memory 40. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of an augmented reality virtual-real hybrid rendering method for mobile live streaming.

[0053] Specifically, the mobile terminal can be a portable electronic device with camera capabilities and sensor configuration, such as a smartphone or tablet. The camera is a built-in high-definition rear camera, used to continuously capture images of the real-world scene at a rate of no less than 30 frames per second. One of the captured images serves as the underlying background image for direct rendering and display by the graphics rendering pipeline, while the other serves as the raw data source for visual recognition. The attitude sensors include an accelerometer and a gyroscope. The accelerometer is used to collect data on the device's tilt angle changes in space, and the gyroscope is used to collect data on the device's rotational angular velocity around each axis. Both report attitude data to the processor at a frequency of over 100Hz.

[0054] The computer program stored in the memory contains multiple functional modules. When the processor executes the computer program, the functional modules work together to complete the following processing flow: The data acquisition module controls the camera and attitude sensor to collect data; the visual recognition module performs grayscale processing, feature point extraction, and feature matching on the real scene image, and calculates the camera projection matrix, camera view matrix, and physical size of the target object after successful matching, wherein the camera projection matrix remains unchanged in subsequent processing; the smoothing module performs matrix element-level weighted interpolation smoothing on the camera view matrix based on the attitude data, and simultaneously integrates accelerometer and gyroscope data for inter-frame attitude compensation and prediction interpolation; the model matrix generation module constructs the vertex coordinates of the virtual plane according to the physical size of the target object, and generates the model matrix through independent proportional scaling factors; the rendering and compositing module uses the smoothed camera view matrix, the unchanged camera projection matrix, and the model matrix to render the virtual content layer by layer onto the real scene image through the graphics rendering pipeline, generating a virtual-real mixed image; the video encoding module calls the device's hardware video encoder to perform real-time compression encoding of the virtual-real mixed image, and pushes the encoded video stream to the live streaming platform server through the network module.

[0055] As the core of computing and control in a mobile terminal, the processor coordinates the work of various hardware components and executes program instructions stored in memory. During live streaming, the processor continuously processes each frame of image data according to the above steps, ensuring that virtual materials fit stably in the frame, the perspective of the background is not affected by smoothing, and the scaling of virtual content is independent of the camera matrix. The streamer can adjust the size of the virtual materials at any time during the live stream while the background seen by the audience remains unchanged. Finally, through hardware encoding and standard transmission protocols, high-quality AR hybrid live stream footage is pushed to major live streaming platforms in real time, achieving full compatibility with the existing live streaming ecosystem.

[0056] The aforementioned storage medium can be any medium capable of storing program code, such as the phone's internal memory, SD card, or USB flash drive. The mobile terminal can be a smartphone, tablet, smartwatch, or other live streaming device. Its built-in camera is responsible for capturing images of the real-world scene, its attitude sensor is responsible for collecting acceleration and angular velocity data, and when the processor executes the computer program in the memory, it completes the virtual-real hybrid rendering and streaming according to the steps of the various embodiments of the above method.

[0057] The above are only some embodiments of the present invention and do not limit the patent scope of the present invention. All equivalent structural transformations made under the technical concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. An augmented reality virtual-real mixed rendering method, characterized in that, include: Acquire real-world scene images captured by a camera and collect scene posture data using an attitude sensor; The attitude sensor includes an accelerometer and a gyroscope; By performing visual recognition on the real-world scene images, the camera projection matrix, camera view matrix, and physical dimensions of the target object can be obtained. The camera view matrix is ​​smoothed based on the scene pose data to generate a smoothed camera view matrix. During the smoothing process, the camera projection matrix remains unchanged so that the perspective relationship of the real scene image is not affected by the smoothing process. The smoothing process includes: performing element-wise weighted interpolation between the original camera view matrix of the current frame and the smoothed camera view matrix of the previous frame. The weighted interpolation uses a first-order low-pass filter, and the filter coefficients are dynamically adjusted according to the device motion speed in the pose data. Furthermore, the camera view matrix is ​​subjected to inter-frame pose compensation and prediction interpolation by fusing the device tilt angle data collected by the accelerometer and the device rotation angular velocity data collected by the gyroscope. Based on the physical dimensions of the target object, virtual content data pre-generated and stored from a large video model is matched, and a model matrix of the virtual content is generated; wherein the virtual plane vertex coordinates are constructed based on the physical dimensions of the target object, and multiplied by an independent proportional scaling factor to obtain scaled virtual plane vertex coordinates, and a model matrix is ​​generated, wherein the proportional scaling factor is decoupled from the camera projection matrix and the camera view matrix; The virtual content is rendered and superimposed onto the real scene image using the smoothed camera view matrix, camera projection matrix, and model matrix to generate a hybrid virtual-real image. A virtual camera device is registered in the operating system driver layer of the mobile terminal, and the frame data of the virtual-real mixed image is written to the output buffer of the virtual camera device in real time, so that the live streaming application running on the mobile terminal can collect and push the video to the live streaming platform through the standard camera interface. 2.The method of claim 1, wherein, The smoothing process for the camera view matrix includes: The original camera view matrix of the current frame visual recognition is interpolated with the smoothed camera view matrix of the previous frame at the matrix element level to filter out the small inter-frame jumps caused by floating-point noise in visual recognition. 3.The enhanced reality mixed reality rendering method of claim 1, wherein, The step of matching virtual content data pre-generated and stored from a large video model with the physical dimensions of the target object, and generating a model matrix of the virtual content, specifically includes: Based on the physical size and / or visual characteristics of the target object, the corresponding virtual content texture or video frame sequence is matched from the pre-generated virtual content library; using the physical size of the target object as a reference, the coordinates of the four corner vertices of the virtual plane are constructed, and the coordinates of the four corner vertices are multiplied by an independent proportional scaling factor to obtain the scaled virtual plane vertex coordinates, thereby generating the model matrix of the virtual content, wherein the virtual content texture or video frame sequence is mapped onto the virtual plane.

4. The augmented reality hybrid rendering method according to claim 1, characterized in that, The rendering and overlaying of virtual content onto the real-world scene image is performed using a graphics rendering pipeline. Specifically, this process includes the following steps: Create a vertex buffer object and bind the vertex coordinate data and texture coordinate data of the virtual content to the vertex buffer object; Create and enable a programmable shader program, and pass the smoothed camera view matrix, the unchanged camera projection matrix, and the MVP matrix formed by combining the model matrix into the shader program to perform coordinate transformation on the vertices of the virtual content; The texture of the virtual content is bound, the texture is sampled by the shader program, and the pixels are colored according to the sampling results; Enable transparent blending mode based on source alpha and one minus source alpha, and enable deep testing; Execute the drawing command to draw the virtual content to the frame buffer, so as to overlay it onto the real scene image.

5. The augmented reality hybrid rendering method according to claim 1, characterized in that, The real-world scene image is directly output as two image data streams after acquisition: The first image data is sent to the graphics rendering pipeline as the underlying background image for rendering; the second image data is compressed and rotated before being sent to the visual recognition module for feature point extraction, image matching and planar tracking; when generating a virtual-real hybrid image, the first background layer is drawn first, and then a virtual content layer is drawn on top of the background layer.

6. A storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as claimed in any one of claims 1 to 5.

7. A mobile terminal, comprising a camera, a posture sensor, a processor, and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 5.

Images