An image processing method and device, an AR device, and a storage medium

Abstract

The disclosure provides an image processing method and device, AR equipment and storage medium, and relates to the technical field of augmented reality. The method acquires a physical world video stream and rendering data of a virtual world rendering layer respectively, performs horizontal flip operations on the two layers of data respectively in the same rendering cycle to generate a physical mirror layer and a virtual mirror layer, excludes at least one element in a user interface (UI) layer from the horizontal flip operation so that the element maintains the original rendering direction, and finally composites the flipped physical mirror layer, virtual mirror layer and UI layer in a preset order to output a final picture. The scheme ensures the complete readability and operability of the user interface while achieving a natural mirror effect of the AR picture, and guarantees the time synchronization and spatial alignment of the virtual and real scenes after mirroring, thereby providing an intuitive and smooth AR mirror selfie experience for the user.

Description

Technical Field

[0001] This disclosure relates to the field of augmented reality technology, and in particular to an image processing method, apparatus, AR device, and storage medium. Background Technology

[0002] In augmented reality (AR) selfie scenarios, users need to capture their interaction with virtual objects. Existing technologies typically involve directly horizontally flipping the final composite image generated by the AR device, which already includes the physical background, virtual objects, and user interface (UI), to simulate a mirror effect. This method has significant drawbacks: First, the overall flip causes all UI elements (such as text and buttons) to be reversed horizontally, rendering the interface completely unreadable and inoperable, resulting in a loss of interactive functionality. Second, because physical video capture and virtual rendering are asynchronous processes, a one-time overall flip cannot guarantee the spatiotemporal consistency of the flipping processes, easily leading to problems such as misalignment of virtual and real content, mismatched action directions, and visual ghosting, resulting in a harsh and distorted mirror effect that does not conform to users' natural perception of mirror reflections. Summary of the Invention

[0003] This disclosure provides an image processing method, apparatus, AR device, and storage medium to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this disclosure, an image processing method is provided, the method comprising: Acquire video streams from the physical world and rendering data from the virtual world's rendering layer; Within the same rendering cycle, a horizontal flip operation is performed on the physical world video stream and the virtual world rendering layer respectively to generate a physical mirror layer and a virtual mirror layer; At least one element in the user interface (UI) layer should maintain its original orientation and not participate in the horizontal flipping operation. The physical mirror layer, virtual mirror layer, and UI layer are rendered and composited according to a preset layer order to output a target image containing a mirrored selfie.

[0005] In one possible implementation, performing a horizontal flip operation on the physical world video stream and the virtual world rendering layer respectively within the same rendering cycle includes: In response to receiving both the physical video frame ready signal and the virtual rendering frame ready signal for the same frame, a synchronous flip instruction is sent to the image processing module. In response to the synchronous flip instruction, the image processing module performs the horizontal flip operation on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer based on the flip parameters.

[0006] In one possible implementation, the horizontal flip operation is performed on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer based on flip parameters, including: The flipping parameters are configured based on the current AR scene; the flipping parameters include at least: flip axis coordinates, flipping direction, and horizontal flipping coordinate mapping rules; Based on the configured flip parameters, the horizontal flip operation is performed on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer.

[0007] In one possible implementation, performing the horizontal flip operation on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer based on the configured flip parameters includes: The image processing module samples the texture data of the physical world video stream and the virtual world rendering layer. Calculate the sampling coordinates according to the horizontal flip coordinate mapping rule; Based on the sampling coordinates, pixel colors are obtained from the texture data to generate flipped texture data.

[0008] In one possible implementation, prior to performing the horizontal flipping operation, the method further includes: Set a unified timestamp identifier for the frame data corresponding to the physical video frame ready signal and the virtual rendering frame ready signal; In the image processing module, it verifies whether the timestamp identifiers carried by the texture data to be processed are consistent, and performs the horizontal flipping operation only on texture data with consistent timestamp identifiers.

[0009] In one possible implementation, before performing a horizontal flip operation on the virtual world rendering layer, the method further includes: offset compensation of the rendering coordinates of the virtual world rendering layer according to pre-stored calibration parameters, wherein the calibration parameters are used to compensate for the positional deviation between the physical camera and the virtual camera.

[0010] In one possible implementation, maintaining at least one element in the user interface (UI) layer in its original orientation and not participating in the horizontal flipping operation includes: In the rendering pipeline, the physical world video stream and the texture data of the virtual world rendering layer are allocated to the first rendering queue and configured to pass through a shader flow that includes horizontal flipping. The texture data of at least one element in the UI layer is assigned to the second rendering queue and configured to skip the shader process that includes horizontal flipping. The second rendering queue has a higher rendering priority than the first rendering queue.

[0011] In one possible implementation, rendering and compositing the physical mirror layer, virtual mirror layer, and UI layer according to a preset layer order includes: Different depth values ​​are set for the physical image layer, the virtual image layer, and the UI layer, with the UI layer having the smallest depth value. During rendering and compositing, depth testing of the UI layer is disabled, so that the UI layer is overlaid on the physical mirror layer and the virtual mirror layer.

[0012] According to a second aspect of this disclosure, an image processing apparatus, the apparatus comprising: The acquisition module is used to acquire video streams from the physical world and rendering data from the virtual world rendering layer. The flip module is used to perform horizontal flipping operations on the physical world video stream and the virtual world rendering layer respectively within the same rendering cycle, generating a physical mirror layer and a virtual mirror layer; The flipping module is also used to maintain at least one element in the UI layer of the user interface in its original orientation and not participate in the horizontal flipping operation; The compositing module is used to render and composite the physical mirror layer, virtual mirror layer, and UI layer according to a preset layer order, and output a target image containing the mirrored selfie scene.

[0013] According to a third aspect of this disclosure, an AR device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described in this disclosure.

[0014] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this disclosure.

[0015] The image processing method, apparatus, AR device, and storage medium disclosed herein acquire rendering data from the physical world video stream and the virtual world rendering layer, respectively. Within the same rendering cycle, they perform horizontal flipping operations on these two layers to generate a physical mirror layer and a virtual mirror layer. Simultaneously, at least one element in the user interface (UI) layer is explicitly excluded from the horizontal flipping operation to maintain its original rendering orientation. Finally, the flipped physical mirror layer, virtual mirror layer, and UI layer are composited in a preset order to output the final image. Thus, this solution achieves a natural mirroring effect for AR images while ensuring the complete readability and operability of the user interface, and guarantees the temporal synchronization and spatial alignment of the virtual and real scenes after mirroring, providing users with an intuitive and smooth AR mirror selfie experience.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0018] Figure 1 This illustration shows the implementation flow of the image processing method according to an embodiment of the present disclosure. Figure 1 ; Figure 2 This illustration shows the implementation flow of the image processing method according to an embodiment of the present disclosure. Figure 2 ; Figure 3 This illustration shows the implementation flow of the image processing method according to an embodiment of the present disclosure. Figure 3 ; Figure 4 This illustration shows the implementation flow of the image processing method according to an embodiment of the present disclosure. Figure 4 ; Figure 5 This illustration shows the implementation flow of the image processing method according to an embodiment of the present disclosure. Figure 5 ; Figure 6 This illustration shows the implementation flow of the image processing method according to an embodiment of the present disclosure. Figure 6 ; Figure 7 This illustration shows the implementation flow of the image processing method according to an embodiment of the present disclosure. Figure 7 ; Figure 8A schematic diagram of the image processing apparatus structure according to an embodiment of the present disclosure is shown; Figure 9 A schematic diagram of the composition structure of an AR device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0019] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0020] This disclosure provides an image processing method, such as Figure 1 As shown, the method includes: Step 101: Obtain the physical world video stream and the rendering data of the virtual world rendering layer.

[0021] In this example, the physical world video stream refers to a continuous sequence of images representing a real-world scene, captured and output in real-time by the front-facing camera of the AR device. This video stream typically requires format conversion after capture (e.g., from YUV to RGB format) to form texture data that the GPU can directly process. The rendering data of the virtual world rendering layer refers to image data representing virtual objects (such as special effects and models), calculated and generated in real-time by the AR engine (such as Unity or Unreal Engine) based on 3D models, spatial positioning, and lighting information. After conversion from the world coordinate system to the screen coordinate system, this data is rendered as a 2D texture with the same resolution as the physical world video stream.

[0022] Step 102: Within the same rendering cycle, perform a horizontal flip operation on the physical world video stream and the virtual world rendering layer respectively to generate a physical mirror layer and a virtual mirror layer.

[0023] In this example, the rendering cycle refers to a fixed time interval defined by the display system (such as the GPU's vertical sync signal) for generating and outputting one frame of video (e.g., approximately 16.67 milliseconds at a 60Hz refresh rate). Within this cycle, the physical video stream and the virtual rendering layer must perform a horizontal flip operation based on their states at the same time to avoid misalignment between user actions and virtual effects in the final image due to processing delays. The horizontal flip operation is implemented through the GPU's shader post-processing, which independently performs pixel-level coordinate transformations and resampling on the physical video texture and the virtual rendering texture to generate flipped physical and virtual mirror layers.

[0024] Step 103: Maintain at least one element in the user interface (UI) layer in its original orientation and not participate in the horizontal flipping operation.

[0025] In this example, the user interface (UI) layer refers to the layer of graphical elements overlaid on the AR screen, used to provide interactive control and information display, such as buttons, menus, and text labels. Among the multiple UI elements contained in this UI layer, at least one core UI element carrying key interactive or information transmission functions, such as a button containing operation guidance text, a status bar displaying shooting parameters, or a function icon with clear directionality, is ensured to maintain its original visual presentation in a direction consistent with the user's reading and operation habits (e.g., from left to right). When executing the horizontal flip processing flow described in step 102, based on predefined element attributes or specified through configuration information, the at least one core UI element is isolated from the processing object of the horizontal flip operation through specific technical means, so that it completely skips the horizontal flip operation process.

[0026] For other non-core or decorative UI elements, their processing methods can be uniformly configured according to design requirements. However, to maintain visual logic consistency, they are usually configured to retain their original orientation. This ensures that in the final mirrored image, the isolated core operation interface elements always maintain readability and operability, solving the problem of UI elements being reversed and unrecognizable due to the overall screen flipping.

[0027] Step 104: Render and composite the physical mirror layer, virtual mirror layer, and UI layer according to the preset layer order to output a target image containing the mirrored selfie screen.

[0028] In this example, rendering compositing is the process by which the graphics processor overlays and blends three independently processed image layers based on their depth information (Z-order) and blending modes (such as alpha blending) to generate a complete image frame. For example, the semi-transparent edges of the virtual mirror layer are alpha blended with the physical mirror layer to achieve a natural blend of reality and virtuality, while the UI layer is overlaid on top of the former two. The composited result is a final target image in which the user sees themselves and the real environment as a mirror image, conforming to the visual habits of taking a selfie. At the same time, the virtual effects are mirrored synchronously with the user's movements, and all operation interfaces are clearly readable. This target image is then sent to the display buffer and finally displayed on the screen of the AR device, completing the generation of the AR mirrored selfie.

[0029] This disclosure provides an image processing method that acquires rendering data from a physical world video stream and a virtual world rendering layer, respectively. Within the same rendering cycle, it performs a horizontal flip operation on these two layers to generate a physical mirror layer and a virtual mirror layer. Simultaneously, it explicitly excludes at least one element from the user interface (UI) layer from the horizontal flip operation, maintaining its original rendering orientation. Finally, it composites the flipped physical mirror layer, virtual mirror layer, and UI layer in a preset order to output the final image. This solution achieves a natural mirror effect for AR images while ensuring the complete readability and operability of the user interface, and guarantees the temporal synchronization and spatial alignment of the virtual and real scenes after mirroring, providing users with an intuitive and smooth AR mirror selfie experience.

[0030] In one example, the horizontal flipping operation is performed on the physical world video stream and the virtual world rendering layer respectively within the same rendering cycle, such as... Figure 2 As shown, it includes: Step 201: In response to receiving the physical video frame ready signal and the virtual rendering frame ready signal for the same frame, send a synchronous flip instruction to the image processing module.

[0031] In this example, the main control module (such as the rendering controller) continuously listens for the physical video frame ready signal and the virtual rendering frame ready signal. Only when both ready signals for the same frame number are received simultaneously within the same rendering cycle does the main control module determine that the physical world video stream and the texture data of the virtual world rendering layer are aligned and ready, and generate and send a synchronous flip instruction to the image processing module.

[0032] Step 202: In response to the synchronous flip instruction, the image processing module performs the horizontal flip operation on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer based on the flip parameters.

[0033] In this example, the flip parameters refer to one or more sets of control data that encode the geometric rules and spatial references required to perform a horizontal flip operation. In response to the synchronous flip command, the image processing module reads and adjusts the flip parameters from the cache to perform a horizontal flip operation on the physical world video stream texture data and the virtual world rendering layer texture data. This ensures that a consistent flip effect can be achieved through the same set of parameterized control flows, even if the physical and virtual layers originate from different sources.

[0034] The specific implementation of the flip parameters can be executed through various image processing algorithms. For example, in a preferred embodiment implemented via shader post-processing, these parameters are compiled into constants or sampling logic in the shader program, enabling the GPU to perform coordinate mapping and resampling. Alternatively, other applicable image processing algorithms (such as CPU-based geometric transformation libraries or fixed-function pipelines on dedicated hardware accelerators) can also be configured to read the flip parameters to achieve horizontal flipping of the physical world video stream texture data and the virtual world rendering layer texture data. Regardless of the underlying algorithm used, the image processing module ensures spatial synchronization and consistency of results between the two layer transformations by parsing and applying uniform flip parameters that are adaptable to the scene and hardware.

[0035] In one example, the horizontal flip operation is performed on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer based on the flip parameters, such as... Figure 3 As shown, it includes: Step 301: Configure the flip parameters based on the current AR scene; the flip parameters include at least: flip axis coordinates, flip direction, and horizontal flip coordinate mapping rules.

[0036] In this example, the flip axis coordinates refer to the center point of symmetry used during horizontal flipping. This point is defined as the center of the screen or normalized device coordinate system, such as the center point coordinates in the screen pixel coordinate system. At a resolution of 1920×1080, the center point coordinates are (960, 540). This parameter is stored in the GPU cache as a shared reference parameter, ensuring that physical and virtual layers use the exact same geometric center for transformation. A uniform flip direction specifically refers to a mirror transformation performed only around the vertical axis in the horizontal plane; that is, only the X-axis coordinates are operated on, while the Y-axis coordinates remain unchanged. This rule excludes unexpected transformations such as upside down. The horizontal flip coordinate mapping rule is a predefined mathematical mapping relationship. Its core is the inverse mapping in the horizontal direction, that is, the horizontal coordinates of the target pixels in the output image are inverted and mapped back to the corresponding horizontal coordinates of the source pixels in the input texture.

[0037] The flipping parameters can be dynamically obtained from the cloud server via the network to adapt to the needs of multi-device collaboration or switching between different scenarios; they can also be pre-stored locally on the AR device to ensure rapid access and operation in offline environments or standardized scenarios. These parameter adjustments depend on the hardware parameters of the AR device (such as the physical installation position of the AR device's camera, screen resolution, and the relative offset between the camera and the screen), scene parameters (the spatial layout of the shooting scene (such as the distance between the user and virtual attractions in cultural and tourism scenarios, and the relative position of the equipment and the manipulated object in industrial and educational scenarios)), and interaction requirements (user operating habits, such as frequently clicked areas, the posture of holding the device, and the required accuracy of interaction between virtual objects and physical scenes) to optimize the mirroring effect.

[0038] Step 302: Based on the configured flip parameters, perform the horizontal flip operation on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer.

[0039] In this example, upon receiving the synchronous flip instruction, the image processing module reads the aforementioned flip parameters from the shared buffer. Subsequently, the module performs independent pixel coordinate transformations and resampling calculations on the physical video texture and virtual rendering texture data, based on a unified flip axis coordinate, flip direction, and horizontal flip coordinate mapping rule, either in parallel or sequentially. Because both follow an absolutely consistent transformation benchmark and rule, the processed physical and virtual mirror layers achieve pixel-level precise alignment and matching of their image content in spatial location.

[0040] In one example, the horizontal flipping operation is performed on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer based on the configured flipping parameters, such as... Figure 4 As shown, it includes: Step 401: The image processing module samples the texture data of the physical world video stream and the virtual world rendering layer.

[0041] In this example, the image processing module is specifically a shader program that executes on the graphics processor. The sampling preparation process involves the shader reading texture data from the physical world video stream and the virtual world rendering layer from the graphics processor's cache. The shader binds texture sampler states to these two types of textures respectively. These states are typically configured in linear filtering mode to ensure that during subsequent coordinate mapping and sampling, when the sampling point is located between texture pixels, a smooth color transition can be obtained through interpolation calculation, avoiding jagged visual defects in the final mirrored image.

[0042] Step 402: Calculate the sampling coordinates according to the horizontal flip coordinate mapping rule.

[0043] In this example, the calculation of sampling coordinates begins in the vertex shader stage, which positions the pixels to be processed into a unified, normalized device coordinate system (NDC). This coordinate system is a normalized space where the X-axis ranges from left to right to [-1, 1], the Y-axis ranges from bottom to top to [-1, 1], and the origin is located at the center. Subsequently, in the fragment shader, a preset horizontal flip coordinate mapping rule is applied to transform the NDC coordinates. The core definition of this rule is: for any pixel, its horizontal (X-axis) coordinates are inverted, while its vertical (Y-axis) coordinates remain unchanged. Through this mathematical operation, the coordinates originally located on the left are mapped to the right, and vice versa, thereby calculating the sampling coordinates of the current output pixel in the original input texture.

[0044] Step 403: Based on the sampling coordinates, obtain the pixel color from the texture data and generate the flipped texture data.

[0045] In this example, based on the calculated sampling coordinates, the fragment shader calls the texture sampling function to obtain the corresponding pixel color values ​​from the texture data of the physical world video stream and the virtual world rendering layer, respectively, thereby generating physically mirrored texture data and virtual mirrored texture data after horizontal flipping. This process is in reverse: in order to fill a pixel at a specific position in the mirrored image, the shader obtains color information from the corresponding position in the original texture in reverse, based on the coordinates calculated according to the flipping rules.

[0046] In addition, if the calculated sampling coordinates exceed the normal coordinate range of the original texture, an edge sampling strategy is adopted, that is, the pixel color values ​​of the texture edges are repeated to prevent unnatural black or transparent areas from appearing at the edges of the final image.

[0047] In one example, before performing the horizontal flip operation, as Figure 5 As shown, it also includes: Step 501: Set a unified timestamp identifier for the frame data corresponding to the physical video frame ready signal and the virtual rendering frame ready signal.

[0048] In this example, the physical video frame ready signal refers to the notification signal issued by the camera module after completing the acquisition and format conversion of a frame of physical world image. The virtual rendering frame ready signal refers to the notification signal issued by the AR engine after completing the rendering of a frame of virtual object and submitting it to the graphics processor cache. The unified timestamp identifier is a unique sequence mark generated by the system master clock or synchronization module, such as Frame_12345. This identifier is assigned to both the physical video frame data and the virtual rendering frame data that are ready within the current cycle.

[0049] Step 502: In the image processing module, it is verified whether the timestamp identifiers carried by the texture data to be processed are consistent, and the horizontal flipping operation is performed only on the texture data with consistent timestamp identifiers.

[0050] In this example, before the horizontal flip operation, the image processing module reads the timestamps carried in the physical video texture and the virtual rendering texture to be processed. By comparing these two timestamps, it determines whether they point to the same rendering cycle. Only when the two timestamps are completely identical is it confirmed that the two texture data belong to the same virtual and real scene at the same moment, and then the horizontal flip operation is initiated. If the timestamps are inconsistent, it indicates that the data may not be aligned due to transmission or processing delays, and the process will pause or wait for correctly matched data. This effectively avoids the visual misalignment problem caused by flipping and compositing physical and virtual images from different moments.

[0051] In one example, before performing a horizontal flip operation on the virtual world rendering layer, the method further includes: offset compensation of the rendering coordinates of the virtual world rendering layer according to pre-stored calibration parameters, wherein the calibration parameters are used to compensate for the positional deviation between the physical camera and the virtual camera.

[0052] In this example, the pre-stored calibration parameters are a set of values ​​measured and stored through a dedicated calibration procedure during the AR device's factory manufacturing or system initialization. These parameters quantify the fixed, minute spatial deviation between the optical center of the physical camera in the device and the virtual camera used by the AR rendering engine. Due to this absolute hardware difference, directly using the raw rendering coordinates of the virtual world's rendering layer for subsequent processing will result in a slight misalignment between the virtual object and the physical world background in the final image.

[0053] Therefore, before the horizontal flip operation, the coordinates of each object to be rendered in the virtual world rendering layer are mathematically translated or transformed and corrected according to the calibration parameters, so as to ensure that the generated virtual mirror layer and the physical mirror layer obtained by directly flipping the physical camera image are precisely aligned in screen space.

[0054] In one example, maintaining at least one element in the user interface (UI) layer in its original orientation and not participating in the horizontal flipping operation is described, such as... Figure 6 As shown, it includes: Step 601: In the rendering pipeline, the physical world video stream and the texture data of the virtual world rendering layer are allocated to the first rendering queue and configured to pass through a shader flow that includes horizontal flipping.

[0055] In this example, the rendering pipeline refers to the standardized process of processing and compositing image data in a graphics rendering system. The first rendering queue is the Background Queue, which is assigned a lower drawing priority. The physical world video stream and the texture data of the virtual world rendering layer are allocated to the first rendering queue and managed as scene background content. In the rendering path of the first rendering queue, a shader processing stage is inserted to perform a horizontal flip operation. All data in this queue (i.e., physical and virtual layers) is automatically directed and mirrored, thereby generating physically mirrored layers and virtual mirrored layers.

[0056] Step 602: Assign the texture data of at least one element in the UI layer to the second rendering queue and configure it to skip the shader process that includes horizontal flipping; wherein the drawing priority of the second rendering queue is higher than that of the first rendering queue.

[0057] In this example, the second rendering queue is the UI Queue, which is given a higher drawing priority. Assigning the texture data of at least one element in the UI layer to this separate queue logically isolates it from the physical and virtual layers. By setting a layer mask or similar mechanism, the data in this queue can bypass the aforementioned flip shader stage when flowing through the rendering pipeline, thereby avoiding any mirror transformation operations and maintaining the original orientation and readability of UI elements (such as text and buttons).

[0058] In one example, the physical mirror layer, virtual mirror layer, and UI layer are rendered and composited according to a preset layer order, such as... Figure 7 As shown, it includes: Step 701: Set different depth values ​​for the physical image layer, the virtual image layer, and the UI layer, wherein the depth value of the UI layer is the smallest.

[0059] In this example, the depth value is a numerical value used in the rendering system to characterize the relative distance between pixels and the camera. It is stored in the depth cache and follows the rendering occlusion rule that the smaller the depth value, the closer it is to the camera and the more visually it appears in front. The default layer stacking order is: the bottom layer is the physical mirror layer, the middle layer is the virtual mirror layer, and the top layer is the UI layer. Specifically, the depth value of the UI layer is set to the smallest (e.g., 0.1), representing that it is closest to the user; the depth value of the virtual mirror layer is set to a middle value (e.g., 0.5), representing that it is behind the UI layer; and the depth value of the physical mirror layer is set to the largest (e.g., 0.9), representing that it is the background and the furthest away. This ensures that during depth testing, the UI layer, because of its smallest depth value, is prioritized as the foreground by the system. Regardless of the movement of objects within the virtual mirror layer, its depth value is always greater than that of the UI layer, thus logically always being occluded by the UI layer and unable to appear in front of UI elements, solving the basic occlusion problem.

[0060] Step 702: During rendering and compositing, disable the depth test of the UI layer so that the UI layer covers the physical mirror layer and the virtual mirror layer.

[0061] In this example, depth testing is the process in the rendering pipeline that compares pixel depth values ​​against a depth buffer to determine whether to overwrite. Disabling depth testing for the UI layer is typically implemented in the UI layer shader using instructions such as `ZTest Off`. During final image compositing, instead of checking if the depth value of each pixel in the UI layer is less than the depth value of the pixels behind it, the UI layer is unconditionally rendered and output to the corresponding screen position in the frame buffer, directly overwriting any pixels in any physical or virtual mirror layers that might exist at that position. This prevents virtual objects from passing through the UI layer: even if a calculation error or a specific action of a virtual object (such as reaching forward) causes the depth value of a part in the virtual mirror layer to temporarily be less than the preset depth value of the UI layer, these virtual pixels cannot be displayed on the UI through depth comparison because depth testing is disabled in the UI layer.

[0062] Furthermore, the AR engine limits the depth of virtual objects to no less than the depth of the UI layer, thus completely eliminating the visual error of virtual objects abnormally penetrating the UI layer from the rendering mechanism, ensuring that UI buttons are always clearly and stably displayed on all virtual content.

[0063] This disclosure also provides an image processing apparatus, such as Figure 8 As shown, the device includes: The acquisition module 801 is used to acquire the video stream of the physical world and the rendering data of the virtual world rendering layer. The flipping module 802 is used to perform horizontal flipping operations on the physical world video stream and the virtual world rendering layer respectively within the same rendering cycle to generate a physical mirror layer and a virtual mirror layer. The flipping module 802 is also used to maintain at least one element in the UI layer of the user interface in its original orientation and not participate in the horizontal flipping operation; The compositing module 803 is used to render and composite the physical mirror layer, virtual mirror layer and UI layer according to a preset layer order, and output a target image containing the mirrored selfie screen.

[0064] In one example, the flip module 802 is specifically used for: In response to receiving both the physical video frame ready signal and the virtual rendering frame ready signal for the same frame, a synchronous flip instruction is sent to the image processing module. In response to the synchronous flip instruction, the image processing module performs the horizontal flip operation on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer based on the flip parameters.

[0065] In one example, the flip module 802 includes an execution unit for: The flipping parameters are configured based on the current AR scene; the flipping parameters include at least: flip axis coordinates, flipping direction, and horizontal flipping coordinate mapping rules; Based on the configured flip parameters, the horizontal flip operation is performed on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer.

[0066] In one example, the execution unit is specifically used for: The image processing module samples the texture data of the physical world video stream and the virtual world rendering layer. Calculate the sampling coordinates according to the horizontal flip coordinate mapping rule; Based on the sampling coordinates, pixel colors are obtained from the texture data to generate flipped texture data.

[0067] In one example, the device further includes a control module 804 for performing the horizontal flipping operation before: Set a unified timestamp identifier for the frame data corresponding to the physical video frame ready signal and the virtual rendering frame ready signal; In the image processing module, it verifies whether the timestamp identifiers carried by the texture data to be processed are consistent, and performs the horizontal flipping operation only on texture data with consistent timestamp identifiers.

[0068] In one example, the device further includes a correction module 805 for offset compensation of the rendering coordinates of the virtual world rendering layer according to pre-stored calibration parameters before performing a horizontal flip operation on the virtual world rendering layer, the calibration parameters being used to compensate for the positional deviation between the physical camera and the virtual camera.

[0069] In one example, the flipping module 802 further includes an allocation unit for: In the rendering pipeline, the physical world video stream and the texture data of the virtual world rendering layer are allocated to the first rendering queue and configured to pass through a shader flow that includes horizontal flipping. The texture data of at least one element in the UI layer is assigned to the second rendering queue and configured to skip the shader process that includes horizontal flipping. The second rendering queue has a higher rendering priority than the first rendering queue.

[0070] In one example, the synthesis module 803 is specifically used for: Different depth values ​​are set for the physical image layer, the virtual image layer, and the UI layer, with the UI layer having the smallest depth value. During rendering and compositing, depth testing of the UI layer is disabled, so that the UI layer is overlaid on the physical mirror layer and the virtual mirror layer.

[0071] According to embodiments of this disclosure, this disclosure also provides an AR device and a readable storage medium.

[0072] Figure 9 A schematic block diagram of an example AR device 900 that can be used to implement embodiments of the present disclosure is shown. The AR device is intended to represent various forms of augmented reality display devices, such as AR glasses, AR head-mounted displays, mixed reality helmets, and other suitable augmented reality devices. The AR device can also be integrated into various forms of mobile devices, such as smartphones, tablets, wearable devices, and other computing devices with augmented reality capabilities. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0073] like Figure 9As shown, the AR device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 902 or a computer program loaded into a random access memory (RAM) 903 from a storage unit 908. The RAM 903 may also store various programs and data required for the operation of the AR device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0074] Multiple components in the AR device 900 are connected to the I / O interface 905, including: an input unit 906, such as a camera, depth sensor, inertial measurement unit, microphone, etc.; an output unit 907, such as a near-eye display optical module, spatial audio speaker, etc.; a storage unit 908, such as flash memory, solid-state drive, etc.; and a communication unit 909, such as a wireless network chip, Bluetooth module, 5G communication module, etc. The communication unit 909 allows the AR device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0075] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose vision processing units (VPUs), artificial intelligence (AI) computing chips, various computing units running SLAM and rendering algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the AR device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform image processing methods by any other suitable means (e.g., by means of firmware).

[0076] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0077] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0078] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0079] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0080] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0081] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0082] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0083] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0084] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Acquire video streams from the physical world and rendering data from the virtual world's rendering layer; Within the same rendering cycle, a horizontal flip operation is performed on the physical world video stream and the virtual world rendering layer respectively to generate a physical mirror layer and a virtual mirror layer; At least one element in the user interface (UI) layer should maintain its original orientation and not participate in the horizontal flipping operation. The physical mirror layer, virtual mirror layer, and UI layer are rendered and composited according to a preset layer order to output a target image containing a mirrored selfie.

2. The method according to claim 1, characterized in that, The step of performing a horizontal flip operation on the physical world video stream and the virtual world rendering layer respectively within the same rendering cycle includes: In response to receiving both the physical video frame ready signal and the virtual rendering frame ready signal for the same frame, a synchronous flip instruction is sent to the image processing module. In response to the synchronous flip instruction, the image processing module performs the horizontal flip operation on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer based on the flip parameters.

3. The method according to claim 2, characterized in that, For the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer, perform the horizontal flip operation based on the flip parameters, including: The flipping parameters are configured based on the current AR scene; the flipping parameters include at least: flip axis coordinates, flipping direction, and horizontal flipping coordinate mapping rules; Based on the configured flip parameters, the horizontal flip operation is performed on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer.

4. The method according to claim 3, characterized in that, The step of performing the horizontal flip operation on the texture data corresponding to the physical world video stream and the texture data corresponding to the virtual world rendering layer based on the configured flip parameters includes: The image processing module samples the texture data of the physical world video stream and the virtual world rendering layer. Calculate the sampling coordinates according to the horizontal flip coordinate mapping rule; Based on the sampling coordinates, pixel colors are obtained from the texture data to generate flipped texture data.

5. The method according to claim 2, characterized in that, Before performing the horizontal flipping operation, the following is also included: Set a unified timestamp identifier for the frame data corresponding to the physical video frame ready signal and the virtual rendering frame ready signal; In the image processing module, it verifies whether the timestamp identifiers carried by the texture data to be processed are consistent, and performs the horizontal flipping operation only on texture data with consistent timestamp identifiers.

6. The method according to claim 1, characterized in that, Before performing a horizontal flip operation on the virtual world rendering layer, the method further includes: offset compensation of the rendering coordinates of the virtual world rendering layer according to pre-stored calibration parameters, wherein the calibration parameters are used to compensate for the positional deviation between the physical camera and the virtual camera.

7. The method according to claim 2, characterized in that, Maintaining at least one element in the user interface (UI) layer in its original orientation, excluding it from the horizontal flipping operation, includes: In the rendering pipeline, the physical world video stream and the texture data of the virtual world rendering layer are allocated to the first rendering queue and configured to pass through a shader flow that includes horizontal flipping. The texture data of at least one element in the UI layer is assigned to the second rendering queue and configured to skip the shader process that includes horizontal flipping. The second rendering queue has a higher rendering priority than the first rendering queue.

8. The method according to claim 1, characterized in that, The step of rendering and compositing the physical mirror layer, virtual mirror layer, and UI layer according to a preset layer order includes: Different depth values ​​are set for the physical image layer, the virtual image layer, and the UI layer, with the UI layer having the smallest depth value. During rendering and compositing, depth testing of the UI layer is disabled, so that the UI layer is overlaid on the physical mirror layer and the virtual mirror layer.

9. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire video streams from the physical world and rendering data from the virtual world rendering layer. The flip module is used to perform horizontal flipping operations on the physical world video stream and the virtual world rendering layer respectively within the same rendering cycle, generating a physical mirror layer and a virtual mirror layer; The flipping module is also used to maintain at least one element in the UI layer of the user interface in its original orientation and not participate in the horizontal flipping operation; The compositing module is used to render and composite the physical mirror layer, virtual mirror layer, and UI layer according to a preset layer order, and output a target image containing the mirrored selfie scene.

10. An AR device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

Images