Special effects follow-up rendering method and device in augmented reality

Abstract

The present invention provides a special effect follow-up rendering method and device in augmented reality, the method comprising: based on the projection relationship between the feature point coordinates of the target object and the image coordinates, according to the real-time video stream, sequentially determining the pose information of the target object in each frame of the image; determining the position information of the target object according to the distance information and angle information of the target object at each frame of the image acquired by the radar; correcting the pose information according to the position information to obtain the corrected pose of the target object in each frame of the image, and performing special effect rendering of the video stream based on the corrected pose. The method obtains pose-related information through a camera and a radar respectively, integrates the pose or position information measured by two types of sensors, and can output the optimized and accurate six-degree-of-freedom pose of the object in real time, so that devices such as smart phones with cameras and radars can perform follow-up special effect rendering on moving objects without adding additional hardware costs.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality, and in particular to a special effects follow-up rendering method and device in augmented reality. Background Art

[0002] Augmented reality (AR) augments the real world by rendering virtual effects on the user's field of view through a camera, which is expected to provide unprecedented immersive experiences in entertainment, education, and healthcare. Recently, the design of mobile AR applications and systems has attracted strong interest from both industry and academia. Current mobile AR solutions (e.g., ARKit and ARCore) enable smartphones to recognize planes as anchors (i.e., a set of key feature points on a plane) and further allow users to pin virtual effects or virtual objects on the plane anchors.

[0003] Many well-known mobile AR applications have been developed based on these frameworks. Such applications use visual odometry or visual inertial odometry to track the pose of the smartphone (pose includes the position and orientation of the object, and the position and orientation have three degrees of freedom respectively, so the pose is a variable with six degrees of freedom). However, such applications can only estimate the pose of the smart device itself, but lack the ability to track the pose of the object in the image, which is crucial for presenting continuous AR effects on moving objects. For example, when a toy dragon (which can spray flames) in the image changes its pose, the virtual flame rendered by the existing ARCore technology may lose its target and produce an erroneous rendering effect. On the contrary, if the six-degree-of-freedom pose of the toy dragon can be obtained, the virtual flame can be rendered in the correct position in real time to achieve a follow-up effect. Similarly, through six-degree-of-freedom pose tracking, the effect of virtual length measurement can be attached to the Rubik's Cube, and the length measurement effect remains in the correct position regardless of whether the Rubik's Cube's pose changes. In short, six-degree-of-freedom pose tracking of objects is a key technology to bridge the gap between virtual and reality in mobile AR applications.

[0004] Existing vision-based 6DOF posture tracking solutions rely on expensive and complex sensor equipment, or require complex pre-modeling of specific objects to be tracked and pre-training of deep neural networks. However, due to the limited computing resources of mobile devices and the complex actual environment, these solutions cannot be used on commercial smartphones. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a method and device for special effect follow-up rendering in augmented reality.

[0006] The present invention provides a special effect follow-up rendering method in augmented reality, comprising: based on the projection relationship between the feature point coordinates of the target object and the image coordinates, according to the real-time video stream, sequentially determining the position information of the target object in each frame of the image; according to the distance information and angle information of the target object at each frame of the image acquired by a radar, determining the position information of the target object; correcting the position information according to the position information to obtain the corrected position of the target object in each frame of the image, and performing special effect rendering of the video stream based on the corrected position.

[0007] According to a special effects follow-up rendering method in augmented reality according to an embodiment of the present invention, the position and posture information of the target object in each frame of the image is determined in sequence according to the real-time video stream, and also includes: determining the ORB feature points of the target object in the image, matching the feature points of two adjacent frames, and constructing a projection relationship between the feature point coordinates and the image coordinates.

[0008] According to the special effects follow-up rendering method in augmented reality according to one embodiment of the present invention, before determining the target object position information based on the distance information and angle information of the target object at each frame of image acquired by the radar, it also includes: determining the signal propagation time based on the frequency difference between the radar transmitting signal and the received signal, and determining the distance information of the target object in each frame of image based on the propagation time.

[0009] According to the special effects follow-up rendering method in augmented reality according to one embodiment of the present invention, before determining the position information of the target object based on the distance information and angle information of the target object at each frame of image acquired by the radar, it also includes: determining the corresponding angle according to the phase difference between adjacent antennas in each group of antenna arrays according to two groups of antenna arrays arranged orthogonally; and determining the angle information of the target object in each frame of the target object based on the angle determined by the two groups of antenna arrays.

[0010] According to the special effects follow-up rendering method in augmented reality of one embodiment of the present invention, after obtaining the corrected pose of the target object in each frame image, it also includes: determining the camera error according to the robust sum function of the error of each feature point projected to the image coordinate; determining the radar error according to the robust sum function of the probability density of the radar measurement error; and determining the optimized corrected pose of the target object based on minimizing the camera error and the radar error.

[0011] According to a special effects follow-up rendering method in augmented reality according to an embodiment of the present invention, the method of determining the optimized corrected pose of the target object based on minimizing the camera error and the radar error includes: performing pose graph optimization solution based on the g2o framework to obtain the optimized corrected pose of the target object.

[0012] According to the special effects follow-up rendering method in augmented reality according to an embodiment of the present invention, the special effects rendering of the video stream based on the corrected posture is specifically: performing special effects rendering of the video stream according to the optimized corrected posture.

[0013] The present invention also provides a special effect follow-up rendering device in augmented reality, comprising: a camera module, which is used to determine the position information of the target object in each frame of the image in turn according to the real-time video stream based on the projection relationship between the feature point coordinates of the target object and the image coordinates; a radar module, which is used to determine the position information of the target object according to the distance information and angle information of the target object at each frame of the image acquired by the radar; a tracking module, which is used to correct the position information according to the position information to obtain the corrected position of the target object in each frame of the image, and perform special effect rendering of the video stream based on the corrected position

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned special effects motion rendering methods in augmented reality are implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned special effects follow-up rendering methods in augmented reality are implemented.

[0016] The method and device for special effects motion rendering in augmented reality provided by the present invention obtain posture-related information through a camera and a radar respectively, integrate the posture or position information measured by the two types of sensors, and can output the optimized and accurate six-degree-of-freedom posture of the object in real time, thereby enabling devices such as smart phones with cameras and radars to perform motion special effects rendering on moving objects without adding additional hardware costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of a special effect follow-up rendering method in augmented reality provided by the present invention;

[0019] Figure 2 is a schematic diagram of a camera imaging model provided by the present invention;

[0020] Figure 3It is a schematic diagram of distance measurement by frequency difference provided by the present invention;

[0021] Figure 4 It is a schematic diagram of angle measurement through phase difference provided by the present invention;

[0022] Figure 5 It is a structural schematic diagram of a special effect follow-up rendering device in augmented reality provided by the present invention;

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Nowadays, sensors that can generate sparse point clouds are gradually integrated into commercial smartphones. The present invention uses relatively sparse measurement results (such as millimeter wave radar) and combines them with visual technology to track the six-degree-of-freedom posture of an object in real time.

[0026] Combine the following Figure 1-Figure 6 The present invention describes a special effect follow-up rendering method and device in augmented reality. Figure 1 is a flow chart of a special effect follow-up rendering method in augmented reality provided by the present invention, such as Figure 1 As shown, the present invention provides a special effect follow-up rendering method in augmented reality, comprising:

[0027] 101. Based on the projection relationship between the feature point coordinates of the target object and the image coordinates, the position information of the target object in each frame of the image is determined in turn according to the real-time video stream.

[0028] First, the camera imaging model is used to determine the position and posture of the object. For example, feature points in the image determined by the camera are extracted, and the feature points between two adjacent frames are matched to construct the three-dimensional feature points X in the space. C The relationship between the two-dimensional pixel feature points x projected onto the image plane is as follows:

[0029]

[0030] Among them, π is the projection function, [f x f y ] T is the focal length of the camera, [c x cy ] T is the camera optical center, they are all intrinsic parameters of the camera.

[0031] 102. Determine the position information of the target object according to the distance information and angle information of the target object at each frame of image acquired by the radar.

[0032] Use radar, such as the current millimeter wave radar, to determine the location information of an object. Millimeter wave radar uses frequency modulated continuous wave signals to measure distance and angle, and ultimately determines the three-dimensional position of the object from the distance and angle.

[0033] 103. Correct the posture information according to the position information to obtain a corrected posture of the target object in each frame of the image, and perform special effects rendering of the video stream based on the corrected posture.

[0034] In one embodiment, the posture information is corrected according to the position information to obtain a corrected posture of the target object in each frame image, including: correcting the three-dimensional position in the posture information in the camera model according to the position information obtained by the radar model.

[0035] Here, an object tracking model is constructed. The camera model and radar model are obtained through visual image processing and radar signal processing respectively, and then the camera model and radar model together form a tracking model. The tracking model uses the precise position information obtained by the radar model to correct the three-dimensional position in the pose information in the camera model, thereby improving the accuracy of the target object's pose. Finally, the special effects rendering of the video stream is performed based on the corrected target object pose.

[0036] Camera model: Through visual image processing, an observation model h(·) for tracking the object’s posture can be provided. It can reflect the six-degree-of-freedom posture of the target on the image plane. For the kth feature point in the image at the i-th time, there is:

[0037]

[0038] in, is the feature point x i (k) The corresponding 3D point in the object reference frame, v i (k) is the random noise of the feature point pose. In most existing vision systems, v i (k) is Gaussian distributed noise, so the camera observation model can be effectively optimized using the least squares method. CR , t CRThey are: relative rotation from radar coordinate system R to camera coordinate system C; relative rotation of feature point i from object coordinate system O to radar coordinate system R; relative displacement of feature point i from object coordinate system O to radar coordinate system R; relative displacement of radar coordinate system R to camera coordinate system C.

[0039] Through radar signal processing, the position of the target in three-dimensional space can be determined. By calculating the three-dimensional position of the object at two time points, the movement direction and distance of the object during this period are given, and the motion model g(·) for tracking the object's posture is provided:

[0040]

[0041] Among them, U represents the displacement of the object, is the difference vector of the three-dimensional position of the object measured by the radar at time i and time j (i.e., the displacement vector of the object), w ij is the error caused by using the radar motion model. They are: the three-dimensional coordinates of the three-dimensional space point P in the radar coordinate system R at time j (that is, the jth frame); the three-dimensional coordinates of the three-dimensional space point P in the radar coordinate system at time i (that is, the i-th frame); the relative displacement from the object coordinate system O to the radar coordinate system R at time i (that is, the i-th frame).

[0042] The special effects follow-up rendering method in augmented reality provided by the present invention obtains posture-related information through a camera and a radar respectively, integrates the posture or position information measured by the two types of sensors, and can output the optimized and accurate six-degree-of-freedom posture of the object in real time, so that devices such as smart phones with cameras and radars can perform follow-up special effects rendering on moving objects without adding additional hardware costs.

[0043] In one embodiment, before determining the position and posture information of the target object according to each frame image of the real-time video stream, it also includes: determining the ORB feature points of the target object in the image, matching the feature points of two adjacent frames, and constructing a projection relationship between the feature point coordinates and the image coordinates.

[0044] The above embodiments have been described. Figure 2 As shown in the figure, ORB features include feature points and descriptors. Feature points are used to filter out "special" points, while descriptors are used to describe the features around a point. The feature points of an image can be simply understood as more prominent points in the image, such as contour points, bright spots in darker areas, dark spots in brighter areas, etc.

[0045] In one embodiment, before determining the target object position information based on the distance information and angle information of the target object at each frame of image acquired by the radar, the method further includes: determining the signal propagation time based on the frequency difference between the radar transmitting signal and the receiving signal, and determining the distance information of the target object in each frame of image based on the propagation time.

[0046] like Figure 3 As shown, the frequency difference between the transmitted signal (TX signal) and the received signal (RX signal) represents the signal propagation time, which can be used to determine the distance of the object. The time-varying distance between the radar and the object is expressed as D(t), and the transmitted and received signals can be expressed as:

[0047] S TX (t) = exp[j(2πf c t+πKt 2 +φ 0 )]

[0048]

[0049] Among them, α is the decay rate, f c The starting frequency, K is the slope of the linear frequency modulation period, c is the speed of light, j is the imaginary unit, φ 0 is the initial phase, and t is the signal propagation time. Then both TX and RX signals are put into the mixer and low-pass filter LPF(·) to extract the intermediate frequency signal (IF signal), as shown below:

[0050]

[0051] Its frequency f IF Contains distance information. Using the Fast Fourier Transform (FFT) on the IF signal, we can extract f IF , then the distance is calculated by:

[0052]

[0053] In one embodiment, before determining the position information of the target object according to the distance information and angle information of the target object at each frame of image acquired by the radar, the method further includes:

[0054] According to two groups of antenna arrays arranged orthogonally, corresponding angles are determined according to the phase differences of adjacent antennas in each group of antenna arrays; and angle information of the target object in each frame of the target object is determined according to the angles determined by the two groups of antenna arrays.

[0055] Through a carefully designed antenna array, millimeter-wave radar can also obtain the direction of an object. Figure 4As shown, multiple antennas are arranged into two orthogonal linear antenna arrays. Each linear array can obtain an angle (i.e., the angle between the array direction and the RX signal direction) by the phase difference between two adjacent antennas with an interval of d:

[0056]

[0057] Where θ is the angle, λ is the wavelength, and Δφ is the phase difference. Since the radar has two orthogonal linear antenna arrays, two different angles θ can be acquired x and θ y . Therefore, the unit vector of the object direction can be expressed as:

[0058]

[0059] Through the above distance and angle measurements, the millimeter wave radar can obtain the position P of the object in the three-dimensional space in the radar reference. R , as shown below:

[0060] P R =Dn

[0061] In one embodiment, after obtaining the corrected pose of the target object in each frame image, it also includes: determining the camera error based on the robust sum function of the error of each feature point projected to the image coordinate; determining the radar error based on the robust sum function of the probability density of the radar measurement error; and determining the optimized corrected pose of the target object based on minimizing the camera error and the radar error.

[0062] In the overall pose optimization, the object's six-degree-of-freedom pose is optimized and output in real time according to the object tracking model. In order to effectively estimate the pose, the present invention adopts an optimization framework based on factor graph. The current object pose θ i It is constrained by two factor nodes, namely the displacement factor t i With the rotation factor R i The overall optimization problem can be expressed as:

[0063]

[0064]

[0065] In the above formula, the camera projection error for:

[0066]

[0067]

[0068] That is, the camera error is determined based on the robust sum function of the error of each feature point projected to the image coordinates, including:

[0069]

[0070] Millimeter wave radar error for:

[0071]

[0072] Among them, Ψ(·) is the probability density function of the millimeter wave radar measurement error. ρ(·) is the robust sum function. The use of the robust sum function here can enhance the resistance of the optimization algorithm to outliers. i represents the time or the i-th frame.

[0073] That is, the radar error is determined based on the robust sum function of the probability density of the radar measurement error, including:

[0074]

[0075] The embodiment of the present invention further corrects the camera error and the radar error through a robust sum function, thereby improving the accuracy of the position and posture of the target object, and further improving the authenticity of the special effect rendering.

[0076] In one embodiment, the pose graph is optimized and solved based on the g2o framework to obtain the optimized corrected pose of the target object. That is, the overall optimization problem can be solved by pose graph optimization with the help of g2o. In one embodiment, the special effects rendering of the video stream based on the corrected pose is specifically: performing special effects rendering of the video stream according to the optimized corrected pose. After the optimization is completed, a more accurate six-degree-of-freedom pose of the object can be obtained, so that a real special effects follow-up rendering effect can be achieved on the moving object.

[0077] The following is a description of the special effect follow-up rendering device in augmented reality provided by the present invention. The special effect follow-up rendering device in augmented reality described below and the special effect follow-up rendering method in augmented reality described above can be referenced to each other.

[0078] Figure 5 is a schematic diagram of the structure of the special effects follow-up rendering device in augmented reality provided by the present invention, such as Figure 5 As shown, the special effect follow-up rendering device in the augmented reality includes: a camera module 501, a radar module 502 and a tracking module 503. Among them, the camera module 501 is used to determine the pose information of the target object in each frame of the image in turn according to the real-time video stream based on the projection relationship between the feature point coordinates of the target object and the image coordinates; the radar module 502 is used to determine the position information of the target object according to the distance information and angle information of the target object at each frame of the image acquired by the radar; the tracking module 503 is used to correct the pose information according to the position information, obtain the corrected pose of the target object in each frame of the image, and perform special effect rendering of the video stream based on the corrected pose.

[0079] In one embodiment of the device, it also includes a posture optimization module, which is used to obtain the corrected posture of the target object in each frame image: determine the camera error based on the robust sum function of the error of each feature point projected to the image coordinate; determine the radar error based on the robust sum function of the probability of radar displacement vector error and rotation vector error; determine the optimized corrected posture of the target object based on minimizing the camera error and the radar error.

[0080] The device embodiments provided in the embodiments of the present invention are for implementing the above-mentioned method embodiments. For specific processes and detailed contents, please refer to the above-mentioned method embodiments, which will not be repeated here.

[0081] The special effects motion rendering device in augmented reality provided by the embodiment of the present invention obtains posture-related information through a camera and a radar respectively, integrates the posture information measured by the two types of sensors, and can output the optimized and accurate six-degree-of-freedom posture of the object in real time, thereby enabling devices such as smart phones with cameras and radars to perform motion special effects rendering on moving objects without adding additional hardware costs.

[0082] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor (processor) 601, a communication interface (Communications Interface) 602, a memory (memory) 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 complete mutual communication through the communication bus 604. The processor 601 may call the logic instructions in the memory 603 to execute the special effect follow-up rendering method in augmented reality, the method comprising: based on the projection relationship between the feature point coordinates of the target object and the image coordinates, according to the real-time video stream, sequentially determining the pose information of the target object in each frame of the image; determining the position information of the target object according to the distance information and angle information of the target object at each frame of the image acquired by the radar; correcting the pose information according to the position information to obtain the corrected pose of the target object in each frame of the image, and performing special effect rendering of the video stream based on the corrected pose.

[0083] In addition, the logic instructions in the above-mentioned memory 603 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0084] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the special effects follow-up rendering method in augmented reality provided by the above-mentioned methods, and the method includes: based on the projection relationship between the feature point coordinates of the target object and the image coordinates, according to the real-time video stream, sequentially determining the posture information of the target object in each frame of the image; determining the position information of the target object according to the distance information and angle information of the target object at each frame of the image acquired by the radar; correcting the posture information according to the position information to obtain the corrected posture of the target object in each frame of the image, and performing special effects rendering of the video stream based on the corrected posture.

[0085] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the special effects follow-up rendering method in augmented reality provided by the above-mentioned embodiments, the method comprising: based on the projection relationship between the feature point coordinates of the target object and the image coordinates, according to the real-time video stream, sequentially determining the posture information of the target object in each frame of the image; determining the position information of the target object according to the distance information and angle information of the target object at each frame of the image acquired by the radar; correcting the posture information according to the position information to obtain the corrected posture of the target object in each frame of the image, and performing special effects rendering of the video stream based on the corrected posture.

[0086] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A special effects follow-up rendering method in augmented reality, characterized in that: include: Based on the projection relationship between the feature point coordinates of the target object and the image coordinates, the position and posture information of the target object in each frame of the image is determined in turn according to the real-time video stream; Determine the position information of the target object according to the distance information and angle information of the target object at each frame of image acquired by the radar; Correcting the posture information according to the position information to obtain a corrected posture of the target object in each frame image, and performing special effects rendering of the video stream based on the corrected posture; After obtaining the corrected position and posture of the target object in each frame of the image, the method further includes: Determine the camera error based on the robust sum function of the error of projecting each feature point to the image coordinates; Determine the radar error based on the robust sum function of the probability density of radar measurement errors; Determine the optimized corrected pose of the target object based on minimization of camera error and radar error; In the overall pose optimization, the six-degree-of-freedom pose of the object is optimized and output in real time according to the object tracking model; based on the factor graph optimization framework, the current object pose is constrained by the displacement factor and the rotation sound node.

2. The special effects follow-up rendering method in augmented reality according to claim 1, characterized in that: Before determining the position and posture information of the target object in each frame of the image in sequence according to the real-time video stream, the method further includes: Determine the ORB feature points of the target object in the image, match the feature points of two adjacent frames, and construct the projection relationship between the feature point coordinates and the image coordinates.

3. The special effects follow-up rendering method in augmented reality according to claim 1, characterized in that: Before determining the position information of the target object according to the distance information and angle information of the target object at each frame of image acquired by the radar, the method further includes: The signal propagation time is determined according to the frequency difference between the radar transmitting signal and the receiving signal, and the distance information of the target object in each frame image is determined according to the propagation time.

4. The special effects follow-up rendering method in augmented reality according to claim 1, characterized in that: Before determining the position information of the target object according to the distance information and angle information of the target object at each frame of image acquired by the radar, the method further includes: According to two groups of antenna arrays arranged orthogonally, the corresponding angles are determined according to the phase differences of adjacent antennas in each group of antenna arrays; According to the angles determined by the two groups of antenna arrays, the angle information of the target object in each frame of the image of the target object is determined.

5. The special effects follow-up rendering method in augmented reality according to claim 1, characterized in that: The method of determining the optimized corrected position and posture of the target object based on minimization of the camera error and the radar error includes: The pose graph is optimized and solved based on the g2o framework to obtain the optimized corrected pose of the target object.

6. The special effects follow-up rendering method in augmented reality according to claim 1, characterized in that: The special effects rendering of the video stream based on the corrected posture is specifically as follows: According to the optimized corrected pose, special effects rendering of the video stream is performed.

7. A special effects follow-up rendering device in augmented reality, characterized in that: include: The camera module is used to determine the position information of the target object in each frame of the image in turn according to the real-time video stream based on the projection relationship between the feature point coordinates of the target object and the image coordinates; The radar module is used to determine the position information of the target object according to the distance information and angle information of the target object at each frame of image acquired by the radar; A tracking module, used to correct the posture information according to the position information, obtain the corrected posture of the target object in each frame image, and perform special effects rendering of the video stream based on the corrected posture; After obtaining the corrected position and posture of the target object in each frame of the image, the method further includes: Determine the camera error based on the robust sum function of the error of projecting each feature point to the image coordinates; Determine the radar error based on the robust sum function of the probability density of radar measurement errors; Determine the optimized corrected pose of the target object based on minimization of camera error and radar error; In the overall pose optimization, the six-degree-of-freedom pose of the object is optimized and output in real time according to the object tracking model; based on the factor graph optimization framework, the current object pose is constrained by the displacement factor and the rotation sound node.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the special effects motion rendering method in augmented reality as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the special effects motion rendering method in augmented reality as claimed in any one of claims 1 to 6 are implemented.

Images