AR sky special effect rendering method, system, electronic device and readable storage medium
By acquiring grayscale images using a sky segmentation algorithm on mobile devices and performing temporal stitching and weighted average mixing, the resolution and accuracy issues of the sky segmentation algorithm on mobile devices are resolved, boundary processing is optimized, and the AR experience is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU YIXIAN XIANJIN TECH CO LTD
- Filing Date
- 2022-12-12
- Publication Date
- 2026-07-31
AI Technical Summary
In existing technologies, sky segmentation algorithms are limited by computing power on mobile devices, resulting in limited resolution and accuracy of sky segmentation. This causes discontinuity in the real-time rendering of the sky mask, obvious boundary jumps, and a lack of synchronization between the camera view and the segmented mask view, especially noticeable during fast movement.
Grayscale images are obtained by using a preset sky segmentation algorithm, sequence markers are added in chronological order and stitched together to form a temporal grayscale image, and then mixed using a symmetrical weighted average method. The image is then rendered and colored by combining special effects content and camera image texture to generate augmented reality images. The temporal dimension of the mask image and the weighted average mixing method are introduced.
The issues of boundary jitter and discontinuity have been resolved, the boundary processing of grayscale images has been optimized, the AR experience has been improved, and more continuous contour curves of sky and non-sky areas have been achieved.
Smart Images
Figure CN116168133B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of augmented reality, and in particular to an AR sky effect rendering method, system, electronic device, and computer-readable storage medium. Background Technology
[0002] Sky segmentation algorithms are used to separate an image into sky and non-sky areas to enable augmented reality (AR) effects, such as replacing the sky area with blooming fireworks or a soaring dragon.
[0003] In related technologies, the real-time image on the mobile device is first divided into a sky area and a non-sky area, and the segmentation information is stored in a grayscale image. Furthermore, based on the grayscale image obtained from the segmentation, the real-time camera image and special effects content are mixed according to the segmentation information to replace or cover the original sky area with special effects content, thereby achieving an AR experience effect.
[0004] However, the sky segmentation algorithm described above is limited by computing power on mobile devices, resulting in limited resolution and accuracy. This leads to discontinuities in the segmented sky mask during real-time rendering, with noticeable jumps between the sky and non-sky regions.
[0005] Additionally, it may cause the camera's live feed and the segmented masked feed (AR effect) to become out of sync, with the segmented feed lagging behind the live camera feed, especially during faster movements where the lag is more noticeable.
[0006] Currently, no effective solution has been proposed to address the poor reliability of existing sky segmentation and AR rendering methods. Summary of the Invention
[0007] This application provides a sky segmentation rendering method, system, computer device, and computer-readable storage medium to at least solve the problem of poor reliability in related technologies for sky segmentation AR rendering methods.
[0008] In a first aspect, embodiments of this application provide an AR sky effect rendering method, applied to a terminal device, the method comprising:
[0009] A grayscale image containing sky segmentation information is obtained by using a preset sky segmentation algorithm;
[0010] In chronological order, multiple grayscale images are labeled with sequence markers and then stitched together into a single temporal grayscale image. The grayscale image labeled N is the current grayscale image, and the (N-1)th grayscale image to the first grayscale image are arranged in reverse chronological order.
[0011] The time-series grayscale images are mixed using a symmetric weighted average method to obtain a mixed grayscale image;
[0012] The special effects content and camera image texture are acquired, and the mixture grayscale image, special effects content, and camera image texture are rendered and colored to generate an augmented reality image containing AR content.
[0013] In some embodiments, the time-series grayscale images are mixed using a symmetric weighted average method, including:
[0014] The following formula is used to blend multiple grayscale images with added markers:
[0015]
[0016]
[0017] Where, r i P is the r-channel parameter of the i-th grayscale image, where r is the r-channel parameter of the blended grayscale image. i It is the weighted value of the i-th image, and N is the sequence of the current grayscale images. b It affects the weighting parameters.
[0018] In some embodiments, the grayscale image uses three channels: R, G, and B, and the parameter values of each channel are equal.
[0019] When the channel parameter value at the target location is 1, the target location is a sky region.
[0020] When the channel parameter value at the target location is 0, the target location is a non-sky area.
[0021] When the channel parameter value of the target location is between 1 and 0, the channel parameter is used to indicate the confidence level that the target location is a sky area or a non-sky area.
[0022] In some embodiments, obtaining a grayscale image containing sky segmentation information from the camera image using a preset sky segmentation algorithm includes:
[0023] The terminal device acquires camera images in real time, preprocesses the camera images, and obtains camera image textures.
[0024] The camera image texture is segmented using the preset sky segmentation algorithm to generate a grayscale image containing sky segmentation information.
[0025] In some embodiments, rendering and coloring are performed based on the mixed grayscale image, special effects content, and camera image texture, including:
[0026] The mixed grayscale image, the special effects content, and the camera image texture are alpha-blended to generate an augmented reality image containing AR effects. During the alpha blending process, the textures of the real-time image and the special effects content are superimposed pixel by pixel.
[0027] In some embodiments, the method further includes:
[0028] Input the temporal grayscale image, the special effects content, and the camera image texture into the shader;
[0029] The shader performs weighted average blending and alpha blending, wherein the input parameters of the shader include: the sequence of the current grayscale image and the influence weight parameters;
[0030] During the alpha blending process, the weighted average blending effect is dynamically adjusted in real time by changing the input parameters of the shader.
[0031] In some embodiments, the camera image texture is processed using the preset sky segmentation algorithm to generate a grayscale image containing sky segmentation information, including:
[0032] The camera image texture is scaled to a preset size, and the spatial resolution of the camera image texture is converted into the number of channels;
[0033] The encoding module performs multi-layer convolution operations on the camera image texture of the preset size to obtain multiple convolution parameters and multiple feature maps of different sizes.
[0034] The decode module calculates the convolution parameters of different layers in the first network branch to obtain the mask baseline.
[0035] The weight module outputs convolution parameters in the second branch, and performs convolution operation on the mask baseline output in the first branch based on the convolution parameters output in the second branch to output the grayscale image, wherein the grayscale image has a preset resolution.
[0036] Secondly, embodiments of this application provide an AR sky effects rendering system, applied to a terminal device. The system includes: a sky segmentation module, a temporal arrangement module, and a shader module, wherein...
[0037] The sky segmentation module is used to obtain a grayscale image containing sky segmentation information by using a preset sky segmentation algorithm;
[0038] The time-series arrangement module is used to add sequence markers to multiple grayscale images in chronological order and stitch them together into a time-series grayscale image, wherein the grayscale image marked as N is the current grayscale image, and the grayscale images from N-1 to 1 are arranged in reverse chronological order.
[0039] The shader module is used to blend the temporal grayscale image using a symmetric weighted average method to obtain a blended grayscale image.
[0040] In addition, special effects content and camera image texture are acquired, and rendering and coloring are performed based on the mixed grayscale image, special effects content and camera image texture to generate an augmented reality image containing AR content.
[0041] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.
[0042] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.
[0043] Compared to related technologies, the AR sky effect rendering method provided in this application obtains a grayscale image containing sky segmentation information through a preset sky segmentation algorithm; adds sequence markers to multiple grayscale images in chronological order and stitches them into a temporal grayscale image; mixes the temporal grayscale image using a symmetrical weighted average method to obtain a mixed grayscale image; acquires the special effects content and camera image texture, and performs rendering and coloring based on the mixed grayscale image, special effects content, and camera image texture to generate an augmented reality image containing AR content. Through this application, the introduction of the temporal dimension concept of the mask image and the weighted average mixing method in the rendering "post-processing" stage not only solves the problems of boundary jumps and discontinuities in existing methods, but also achieves boundary softening processing of the sky area in the grayscale image and optimizes the contour curve of the non-sky area. Attached Figure Description
[0044] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0045] Figure 1 This is a schematic diagram of the application environment of the AR special effects rendering method according to the embodiments of this application;
[0046] Figure 2 This is a flowchart of an AR special effects rendering method according to an embodiment of this application;
[0047] Figure 3 This is a schematic diagram illustrating the mixing of multiple grayscale images using a symmetrical weighted average method according to an embodiment of this application.
[0048] Figure 4 This is a comparative schematic diagram of the grayscale image mixing effect under different influence weight parameters according to the embodiments of this application;
[0049] Figure 5 This is a schematic diagram illustrating the generation of augmented reality images based on a mixed grayscale image, camera image texture, and special effects content, according to an embodiment of this application.
[0050] Figure 6 This is a schematic diagram of a sky segmentation algorithm according to an embodiment of this application;
[0051] Figure 7 This is an AR sky effect rendering system according to an embodiment of this application;
[0052] Figure 8 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0053] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.
[0054] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.
[0055] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0056] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.
[0057] In this document, it should be understood that the terms used may be technical means used to implement part of this invention or other summary technical terms. For example, terms may include:
[0058] Sky segmentation algorithm: used to separate the sky area and non-sky area in the camera image to enable the creation of augmented reality (AR) effects on the image;
[0059] Mask image: An image obtained after segmenting the camera view using a sky segmentation algorithm. It can be used to obtain transparency parameters and mix real-time images with special effects. It is also called a grayscale image or mask image.
[0060] Rendering process: (1) No post-processing solution: Completely trust the segmented mask image and directly use the mask image to perform alpha blending of real-time image and special effects content; (2) Post-processing solution: Blur the segmented mask image and weaken the sharp edges in the mask image to obtain a better blended image. The processing methods include, but are not limited to, erosion and dilation of the mask image, Gaussian filtering, etc.
[0061] A1pha blending: Transparency blending, is a pixel-by-pixel blending of source and background color data. It blends the three components of a given source color (red, green, and blue) with the corresponding components of the background color according to a preset function.
[0062] Moving average method: It is a smoothing or forecasting method that uses the average of adjacent items in a time series as a trend value or forecast value. It is divided into simple moving average method and weighted moving average method.
[0063] The AR effects rendering method provided in this embodiment can be applied to, for example... Figure 1 In the environment shown, Figure 1 This is a schematic diagram illustrating the application environment of the AR special effects rendering method according to an embodiment of this application. For example... Figure 1 As shown, the user captures a real-time image of an outdoor scene using a mobile terminal 10. Further, the terminal device 10 runs a sky segmentation algorithm to separate the sky and non-sky areas in the real-time image, obtaining a grayscale image. Further, the mobile terminal 10 retrieves AR effects content from a local machine or a cloud server 11, and merges this effect content, the grayscale image, and the real-time image texture to add AR experience content to the sky area of the camera's real-time image, meeting the user's AR experience needs.
[0064] Among them, the mobile terminal 10 can be a smartphone, tablet computer, or PC, and the cloud server 11 can be deployed on a distributed cloud computing platform.
[0065] In some of these embodiments, Figure 2 This is a flowchart of an AR special effects rendering method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:
[0066] S201, Generate a grayscale image containing sky segmentation information using a preset sky segmentation algorithm;
[0067] As can be understood, the sky segmentation algorithm is used to separate the sky region and the non-sky region in an image.
[0068] It should be noted that the preset sky segmentation algorithm used in this embodiment, based on a special network structure, can reduce the resolution of the masking image, thereby reducing the performance dependence on local computing power and improving the compatibility of the device.
[0069] In addition, after the terminal device acquires the camera image, it first needs to undergo preprocessing to obtain the camera image texture; then, it performs a segmentation step on the camera image texture using a preset segmentation algorithm to obtain the grayscale image mentioned above.
[0070] S202, according to the time order, add sequence markers to multiple grayscale images and stitch them into a time-series grayscale image, where the grayscale image marked as N is the current grayscale image, and the grayscale images from N-1 to 1 are arranged in reverse chronological order;
[0071] It is understandable that the above "time sequence" refers to the time sequence in which the user operates the terminal device to capture video footage.
[0072] For example, a video frame is divided into 1-N image frames. After these image frames are processed by a sky segmentation algorithm, 1-N grayscale images are obtained. Correspondingly, the grayscale image at the current time (latest) is given a sequence label of N, the earliest grayscale image is given a sequence label of 1, and the remaining grayscale images from the (N-1)th to the first are arranged in reverse chronological order.
[0073] S203, the time-series grayscale images are mixed using a symmetrical weighted average method to obtain a mixed grayscale image;
[0074] The moving average method has already been introduced in the aforementioned terminology explanation section. Furthermore, the moving average method includes the simple moving average method and the weighted moving average method;
[0075] Among them, the simple moving average method averages the most recent K periods of data to obtain the trend prediction value; the weighted moving average method averages the most recent K periods of data by assigning different weight parameters.
[0076] Figure 3 This is a schematic diagram illustrating the mixing of multiple grayscale images using a symmetric weighted average method according to an embodiment of this application.
[0077] like Figure 3 As shown, firstly, 16 adjacent grayscale images are arranged in chronological order and then stitched together into a single chronological grayscale image. Further, in the shader, a symmetrical weighted average method is used to perform grayscale blending to obtain a mixed grayscale image. During the blending process, the weighting parameters can be adjusted through the shader to dynamically obtain the optimal mixed grayscale image.
[0078] S204 renders and colors images based on a mixed grayscale image and special effects content to generate augmented reality images containing AR content.
[0079] Among them, rendering and shading can be implemented in environments such as Unity 3D and Unreal Engine;
[0080] The aforementioned special effects are used to cover the "original sky area," which can be a panoramic image or a video texture. It should be noted that when the special effects are panoramic video textures, they need to be anti-distortion processed. Common methods can be used to implement anti-distortion processing, which will not be elaborated in this embodiment.
[0081] In addition to mixing grayscale images and special effects content, rendering and shading also require the use of camera image textures, which is well known to those skilled in the art. Specifically, based on the aforementioned mixed grayscale images, camera image textures, and AR special effects content, alpha blending is performed in the shader to generate an augmented reality image containing AR special effects.
[0082] Through the above steps S201 to S204, compared with the existing sky segmentation effect rendering method, in this embodiment, by introducing the concept of time dimension of mask image and weighted average mixing method in the rendering "post-processing" stage, not only can the boundary jump and discontinuity problems in the existing method be solved, but also the boundary softening processing of grayscale image can be achieved, and the contour curve of non-sky area can be optimized.
[0083] In some embodiments, the time-series grayscale images are mixed using a symmetric weighted average method, including:
[0084] The following formula is used to blend multiple sorted grayscale images:
[0085]
[0086]
[0087] Where, r i P is the r-channel parameter of the i-th grayscale image, where r is the parameter of the r-channel of the blended grayscale image. i It is the weighted value of the i-th image, and N is the sequence of the current grayscale images. b It affects the weighting parameters.
[0088] Figure 4 This is a comparative schematic diagram of the grayscale image mixing effect under different influence weight parameters according to the embodiments of this application, such as... Figure 4 As shown:
[0089] Influence on weight parameter N b The closer the value is to 1, the greater the influence of the previous few images on the grayscale image blending result; the closer the Nb parameter is to N, the greater the influence of the latest few images on the blending result.
[0090] A smaller Nb parameter will gradually increase the latency of the algorithm; a larger Nb parameter will improve the real-time performance of the algorithm, but will also increase the erratic nature of the algorithm.
[0091] Different influence weight parameters will lead to different mixing effects. After experimentation and summarization, the inventors of this case found that when the special effects content is a real scene image, the preferred choice is N=16 and Nb=2.
[0092] as follows Figure 5 This is a schematic diagram illustrating the generation of augmented reality images based on a mixed grayscale image, camera image texture, and special effects content, according to an embodiment of this application.
[0093] In this embodiment, during the post-processing stage, the grayscale images sorted in time are mixed by symmetrical weighted average to soften the boundaries of the grayscale images, optimize the contour curves of the sky area and the non-sky area, thereby improving the AR experience.
[0094] In some embodiments, it should be noted that the grayscale image uses three channels: R, G, and B, and the parameter values of each channel are equal.
[0095] When the channel parameter value at the target location is 1, the target location is a sky region.
[0096] When the channel parameter value at the target location is 0, the target location is a non-sky area.
[0097] When the channel parameter value of the target location is between 1 and 0, the channel parameter is used to indicate the confidence level of whether the target location is a sky area or a non-sky area. That is, the closer to 1, the more likely the location is to be a sky area, and the closer to 0, the more likely the location is to be a non-sky area.
[0098] Rendering and coloring are based on a blend of grayscale images and special effects, including:
[0099] The grayscale image, special effects content, and camera image texture are mixed and then alpha-blended to generate an augmented reality image with AR effects.
[0100] In the alpha blending process, the textures of real-time images and special effects are overlaid pixel by pixel.
[0101] Specifically, the process of mixing the above three elements can be represented by the following function expression:
[0102] pic(r,g,b,a)=camera(r,g,b,a)×gery(1-r)+panoramic(r,g,b,a)×gery(r)
[0103] Here, pic represents the parameter value of the rendering result, camera represents the real-time camera image texture, gery represents the grayscale image of the sky segmentation, and panoramic represents the special effects content (panoramic image or video texture).
[0104] In some embodiments, specific mask image blending is performed through shaders, as well as alpha blending during the rendering process, specifically:
[0105] Input the temporal grayscale image, special effects content, and camera image texture into the shader;
[0106] The grayscale image is weighted and blended using a shader, along with alpha blending. The shader's input parameters include: the sequence N of the current grayscale image in the real-time image and the influence weight parameter N. b ;
[0107] During alpha blending, the blending effect is dynamically adjusted in real time by changing the shader's input parameters. In this embodiment, dynamic adjustment can be achieved while maintaining continuity by adjusting the shader input parameters.
[0108] In some embodiments, the sky segmentation algorithm used in this embodiment is deeply optimized for mobile devices, characterized in that the resolution of the output mask image is 1 / 8 of the original image. Figure 6 This is a schematic diagram of a sky segmentation algorithm according to an embodiment of this application, as shown below. Figure 6 As shown:
[0109] In some embodiments, the sky segmentation algorithm used in this embodiment is deeply optimized for mobile devices, characterized in that the resolution of the output mask image is 1 / 8 of the original image. Figure 6 This is a schematic diagram of a sky segmentation algorithm according to an embodiment of this application, as shown below. Figure 6 As shown:
[0110] The semantic deep neural network for sky segmentation consists of two modules: an encoding module and a decoding module. The encoding module includes Blocks 1-4, which uses a modified version of MobileNetV3 as its backbone. Optionally, to reduce computation, before performing convolution calculations on the input image through the encoding module, the 1*3*512*512 RGB input image is first transformed into a 1*48*128*128 image using a spacetodepth operation. The spacetodepth operation also converts spatial resolution into channel count, thereby reducing the computational load of the network and ensuring that the size of the re-input image is large enough to achieve the best pixel-level image segmentation results.
[0111] Furthermore, such as Figure 6 As shown, the image after being downsized is input into Block1 of the encoding module for convolution calculation. Block1 uses a 1×1 convolution with 48 input channels and 40 output channels, and does not perform downsampling. It is then followed by batchnorm and hswish activation functions. Next, the image is fed into Block2, Block3 and Block4 of the MobilenetV3 structure for convolution calculation, respectively, to obtain feature maps with sizes of 64*64, 32*32 and 16*16.
[0112] After obtaining the convolutional parameters of each Block output through the above steps, the decode module calculates the convolutional parameters of each Block output and outputs the mask baseline. Specifically, as shown in the figure, the output of Block 4 (output4) is fed into decode4 for calculation, then concatenated with the output of Block 3, and then decode3 is calculated again, and so on, until Block 2 finishes calculation, outputting the convolutional parameters, which are then fed into the prototype network branch. Preferably, in this embodiment, decode uses FPN to upsample the convolutional parameters of each Block output to obtain feature map information, which can make full use of multi-scale information. It should be noted that the semantic deep neural network has two different network branches: one is the weight branch, which is used to output the convolutional parameters, and the other is the prototype branch, which is used for the mask baseline.
[0113] Finally, the convolution parameters output by the weight branch are used to perform convolution calculation on the mask baseline output by the prototype branch to obtain the final semantic segmentation mask. Specifically, the weight branch is calculated directly from the output of the Block4 branch. The output parameters of Block4 are globally averaged and connected to a fully connected layer, with an output dimension of 91, that is, 91 convolution parameters are output.
[0114] like Figure 6 As shown, the Prototype branch first performs a 3-kernel convolution on the output of decode2, converting its channels from 24 to 6. Then, the 91 convolution parameters output by the weight branch are arranged into the parameters of three 1X1 convolutions, resulting in six 1X1X6 convolution kernels. Six biases are added, and after three 1X1 convolutions, a feature map with one channel is output, which is the final semantic mask.
[0115] It should be noted that the Loss function used for training the semantic deep neural network in this embodiment is as follows:
[0116] diceloss*0.1+0.8*BinaryFocalLoss+0.1*JaccardLoss
[0117] The sky mask value is obtained through the above-mentioned sky semantic segmentation, which can then be used as a mask (grayscale image) for virtual-real fusion. Since this grayscale image has a smaller resolution compared to grayscale images in other existing technologies, it can reduce the performance dependence of local computing power and improve device compatibility.
[0118] This embodiment also provides an AR sky effects rendering system, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0119] Figure 7 This application discloses an AR sky effect rendering system according to an embodiment of the present application. The system is characterized by being applied to a terminal device and includes: a sky segmentation module 70, a temporal arrangement module 71, and a shader module 72.
[0120] The sky segmentation module 70 is used to obtain a grayscale image containing sky segmentation information through a preset sky segmentation algorithm;
[0121] The time sequence arrangement module 71 is used to add sequence markers to multiple grayscale images in chronological order and stitch them into a time sequence grayscale image, wherein the grayscale image marked as N is the current grayscale image, and the grayscale images from N-1 to 1 are arranged in reverse chronological order.
[0122] Shader module 72 is used to blend temporal grayscale images using a symmetric weighted average method to obtain a blended grayscale image, and to acquire special effects content and camera image textures, and to render and shade based on the blended grayscale image, special effects content and camera image textures to generate an augmented reality image containing AR content.
[0123] In one embodiment, Figure 8 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 8 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 8As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network connection, the internal memory provides an environment for the operation of the operating system and computer programs, the computer programs are executed by the processor to implement an AR sky effects rendering method, and the database stores data.
[0124] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.
[0125] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0127] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. An AR sky special effect rendering method, characterized in that, When applied to a terminal device, the method includes: A grayscale image containing sky segmentation information is obtained by using a preset sky segmentation algorithm; In chronological order, multiple grayscale images are labeled with sequence markers and then stitched together into a single temporal grayscale image. The grayscale image labeled N is the current grayscale image, and the (N-1)th grayscale image to the first grayscale image are arranged in reverse chronological order. The time-series grayscale images are mixed using a symmetric weighted average method to obtain a mixed grayscale image; The special effects content and camera image texture are acquired, and the rendering and coloring are performed based on the mixed grayscale image, special effects content and camera image texture to generate an augmented reality image containing AR content; The time-series grayscale images are mixed using a symmetric weighted average method, including: The following formula is used to blend multiple grayscale images with added markers: wherein, is the r channel parameter of the i-th gray scale image, is the r channel parameter of the mixed gray scale image, is the weighting value of the i-th image, and N is the sequence of the current gray scale image, is the influence weight parameter.
2. The method of claim 1, wherein, The grayscale image uses three channels: R, G, and B, and the parameter values for each channel are equal. When the channel parameter value at the target location is 1, the target location is a sky region. When the channel parameter value at the target location is 0, the target location is a non-sky area. When the channel parameter value of the target location is between 1 and 0, the channel parameter is used to indicate the confidence level that the target location is a sky area or a non-sky area.
3. The method of claim 1, wherein, The step of obtaining a grayscale image containing sky segmentation information from the camera image using a preset sky segmentation algorithm includes: The terminal device acquires camera images in real time, preprocesses the camera images, and obtains camera image textures. The camera image texture is segmented using the preset sky segmentation algorithm to generate a grayscale image containing sky segmentation information.
4. The method of claim 3, wherein, Rendering and coloring are performed based on the mixed grayscale image, special effects content, and camera image texture, including: The mixed grayscale image, the special effects content, and the camera image texture are alpha-blended to generate an augmented reality image containing AR effects. During the alpha blending process, the textures of the real-time image and the special effects content are superimposed pixel by pixel.
5. The method of claim 4, wherein, The method further includes: Input the temporal grayscale image, the special effects content, and the camera image texture into the shader; The shader performs weighted average blending and alpha blending, wherein the input parameters of the shader include: the sequence of the current grayscale image and the influence weight parameters; During the alpha blending process, the weighted average blending effect is dynamically adjusted in real time by changing the input parameters of the shader.
6. The method of claim 4, wherein, The camera image texture is processed using the preset sky segmentation algorithm to generate a grayscale image containing sky segmentation information, including: The camera image texture is scaled to a preset size, and the spatial resolution of the camera image texture is converted into the number of channels; The encoding module performs multi-layer convolution operations on the camera image texture of the preset size to obtain multiple convolution parameters and multiple feature maps of different sizes. The decode module calculates the convolution parameters of different layers in the first network branch to obtain the mask baseline. The weight module outputs convolution parameters in the second branch, and performs convolution operation on the mask baseline output in the first branch based on the convolution parameters output in the second branch to output the grayscale image, wherein the grayscale image has a preset resolution.
7. An AR sky special effect rendering system, comprising: When applied to terminal devices, the system includes: a sky segmentation module, a timing arrangement module, and a shader module, wherein... The sky segmentation module is used to obtain a grayscale image containing sky segmentation information by using a preset sky segmentation algorithm; The time-series arrangement module is used to add sequence markers to multiple grayscale images in chronological order and stitch them together into a time-series grayscale image, wherein the grayscale image marked as N is the current grayscale image, and the grayscale images from N-1 to 1 are arranged in reverse chronological order. The shader module is used to blend the temporal grayscale image using a symmetric weighted average method to obtain a blended grayscale image. In addition, special effects content and camera image texture are acquired, and rendering and coloring are performed based on the mixed grayscale image, special effects content and camera image texture to generate an augmented reality image containing AR content; The time-series grayscale images are mixed using a symmetric weighted average method, including: The following formula is used to blend multiple grayscale images with added markers: wherein, is the r channel parameter of the i-th gray scale image, is the r channel parameter of the mixed gray scale image, is the weighting value of the i-th image, and N is the sequence of the current gray scale image, is the influence weight parameter.
8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.
9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.