Augmented reality special effect rendering method and device, equipment and storage medium

Abstract

The application provides an AR special effect rendering method and device, equipment and a storage medium, and belongs to the fields of Internet and computers. The method comprises the following steps: obtaining object pose information based on a first image frame containing a target object; determining model pose information and model size information of an AR special effect model according to parameter information of a camera and the object pose information; performing size adjustment processing on the AR special effect model according to the model size information to obtain an adjusted AR special effect model; determining relative pose information of the AR special effect model with respect to the camera according to pose information of the camera and the model pose information; and projecting the adjusted AR special effect model into the first image frame based on the relative pose information to generate a special effect image frame corresponding to the first image frame. The application provides a way of adding AR special effects to a target object, which can generate an AR special effect image without constructing a virtual map, thereby reducing the amount of calculation required for generating the AR special effect image.

Description

Technical Field

[0001] This application relates to the fields of the Internet and computers, and in particular to an AR special effects rendering method, apparatus, device and storage medium. Background Technology

[0002] Currently, AR (Augmented Reality) effects are being used more and more widely.

[0003] In related technologies, when adding AR effects to target objects in an image, the target object is first detected and acquired. Then, a SLAM (Simultaneous Localization and Mapping) system is used to track camera movement and construct a virtual map. Further, the target object is placed in the virtual map, and based on the rendering effect of the AR effect on the target object (e.g., the AR effect is fixed on top of the target object), an AR effect model is placed in the virtual map. Then, based on the camera's position in the virtual map, the position and orientation of the target object and the AR effect model within the camera's shooting area are determined, generating the AR effect image corresponding to the aforementioned image.

[0004] However, in the aforementioned related technologies, when generating AR special effects images based on the SLAM system, the computational load is large because it is necessary to construct a virtual map and model the target object and AR special effects model in the virtual map. Summary of the Invention

[0005] This application provides an AR special effects rendering method, apparatus, device, and storage medium, which can reduce the computational load required to generate AR special effects images. The technical solution is as follows:

[0006] According to one aspect of the embodiments of this application, an AR special effects rendering method is provided, the method comprising:

[0007] Based on a first image frame containing the target object, object pose information is obtained; wherein, the object pose information includes the spatial position information and spatial orientation information of the target object in the first image frame;

[0008] Based on the camera's parameter information and the object's pose information, the model pose information and model size information of the AR effect model are determined; wherein, the model pose information includes the spatial position information and spatial posture information of the AR effect model;

[0009] Based on the model size information, the AR effect model is resized to obtain the adjusted AR effect model.

[0010] Based on the camera pose information and the model pose information, determine the relative pose information of the AR effect model with respect to the camera;

[0011] Based on the relative pose information, the adjusted AR effect model is projected onto the first image frame to generate an effect image frame corresponding to the first image frame.

[0012] According to one aspect of the embodiments of this application, an AR special effects rendering apparatus is provided, the apparatus comprising:

[0013] An object pose acquisition module is used to acquire object pose information based on a first image frame containing a target object; wherein, the object pose information includes the spatial position information and spatial orientation information of the target object in the first image frame;

[0014] The special effects pose acquisition module is used to determine the model pose information and model size information of the AR special effects model based on the camera parameter information and the object pose information; wherein, the model pose information includes the spatial position information and spatial posture information of the AR special effects model;

[0015] The model size adjustment module is used to adjust the size of the AR effect model according to the model size information to obtain the adjusted AR effect model.

[0016] The relative pose acquisition module is used to determine the relative pose information of the AR effect model relative to the camera based on the pose information of the camera and the pose information of the model.

[0017] The special effects image acquisition module is used to project the adjusted AR special effects model onto the first image frame based on the relative pose information, and generate a special effects image frame corresponding to the first image frame.

[0018] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the above-described AR special effects rendering method.

[0019] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set or instruction set is stored in the computer-readable storage medium, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the above-described AR special effects rendering method.

[0020] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned AR effects rendering method.

[0021] The technical solutions provided in this application embodiment may have the following beneficial effects:

[0022] By combining the pose information of the target object with the camera's parameter information, the pose and size information of the AR effect model are determined. The size of the AR effect model is then adjusted based on the size information. The relative pose information between the AR effect model and the camera is determined based on the model's pose information. Based on this relative pose information, the adjusted AR effect model is mapped onto the first image frame, ultimately obtaining the corresponding effect image frame. This provides a method for adding AR effects to a target object. It determines the mapping effect of the AR effect model in the first image frame based on the relative pose information between the AR effect model and the camera, generating AR effect images without the need to construct a virtual map, thus reducing the computational load required to generate AR effect images. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of a special effects display system provided in one embodiment of this application;

[0025] Figure 2 This is a flowchart of an AR special effects rendering method provided in one embodiment of this application;

[0026] Figure 3 An exemplary schematic diagram of a bounding box and a target object is shown;

[0027] Figure 4 This is a flowchart of an AR special effects rendering method provided in another embodiment of this application;

[0028] Figure 5 An exemplary schematic diagram of an AR effects display interface is shown;

[0029] Figure 6 This is a block diagram of an AR special effects rendering device provided in one embodiment of this application;

[0030] Figure 7 This is a block diagram of an AR special effects rendering device provided in another embodiment of this application;

[0031] Figure 8 This is a schematic diagram of a computer device structure provided in one embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0033] Please refer to Figure 1 The diagram illustrates a special effects display system provided in one embodiment of this application. The special effects display system may include a terminal 10 and a server 20.

[0034] Terminal 10 can be an electronic device such as a mobile phone, tablet computer, game console, e-book reader, multimedia playback device, wearable device, or PC (Personal Computer). Terminal 10 can include any client with AR effects display functionality. The AR effects can be added to the target object or displayed separately on the screen; this embodiment does not limit this. Optionally, the client can be a client for an application that requires downloading and installation, or a client for an application that can be used instantly.

[0035] In this embodiment, the terminal 10 corresponds to an image acquisition unit. In one possible implementation, the image acquisition unit is a device that establishes a connection with the terminal 10 via a network, such as... Figure 1 The image acquisition device 30 is included. Optionally, after acquiring image information, the image acquisition unit transmits the image information to the terminal 10 via the network. In another possible implementation, the image acquisition unit is a hardware component of the terminal 10. Optionally, after receiving a start request for the image acquisition unit, the terminal 10 activates the image acquisition unit and acquires image information in real time based on the image acquisition unit. The start request can be user-triggered. Optionally, the image information includes one or more image frames; this embodiment does not limit this. Of course, when the image information includes multiple image frames, the multiple image frames can be continuous (e.g., a video stream) or non-continuous; this embodiment does not limit this.

[0036] Server 20 provides backend services to terminal 10. Server 20 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. Optionally, server 20 can be the backend server for the aforementioned client. For example, as shown... Figure 1 As shown, after acquiring the aforementioned image information, the server 20 performs AR effect rendering processing on the image information to generate a corresponding AR effect image, and then sends the AR effect image to the terminal 10, which then displays the AR effect image to the user. Of course, in practical applications, after acquiring the aforementioned image information, the terminal 10 can also use its own computer program to perform AR effect rendering processing on the image information to generate a corresponding AR effect image; this embodiment does not limit this. Optionally, the server 20 can simultaneously provide background services to clients in multiple terminals 10.

[0037] Optionally, the terminal 10 and the server 20 communicate with each other via a network. The technical solution of this application will now be described in conjunction with several embodiments.

[0038] Please refer to Figure 2 The diagram illustrates a flowchart of an AR effects rendering method provided in one embodiment of this application. This method can be applied to... Figure 1 In the special effects display system shown, if the execution entity of each step can be terminal 10 or server 20 (hereinafter collectively referred to as "computer equipment"), the method may include the following steps (201-205):

[0039] Step 201: Obtain object pose information based on the first image frame containing the target object.

[0040] The first image frame refers to the image frame to be processed that contains the target object. The target object is the object to be processed in the image frame. In the embodiments of this application, the computer device renders and generates AR effects for the target object by processing the image frame. Optionally, the first image frame may contain one or more target objects, and when the first image frame contains multiple target objects, the categories of the target objects may be the same or different, which is not limited in the embodiments of this application.

[0041] In one possible implementation, the target object is any object with a solid shape. Optionally, after acquiring image information, the computer device iterates through and filters the image information to obtain images containing objects with arbitrary solid shapes as images to be processed. In another possible implementation, the target object is an object of a predefined category. Optionally, after acquiring image information, the computer device processes the image information to determine the category of the objects contained in each image, and then obtains images containing objects of the predefined categories as images to be processed.

[0042] It should be noted that the aforementioned image information includes one or more image frames, and the aforementioned image to be processed includes one or more image frames to be processed. Optionally, when the image to be processed includes multiple image frames to be processed, the aforementioned first image frame is any one of the multiple image frames; or, the aforementioned first image frame may also be an image frame that meets the requirements among the multiple image frames. The aforementioned requirements are judgment requirements for image frames set according to actual circumstances. For example, if the aforementioned image information is a video stream, that is, the image information includes multiple image frames and the multiple image frames are consecutive, then the aforementioned requirement is the first appearance of the target object; correspondingly, the aforementioned first image frame is the image frame corresponding to the first appearance of the target object in the video stream. Of course, in practical applications, the aforementioned requirements may also include minimizing the number of target objects, including target objects of a specific category, displaying target objects from a frontal perspective, etc.

[0043] In this embodiment, after acquiring the first image frame, the computer device obtains object pose information based on the first image frame containing the target object. The object pose information includes the spatial position information and spatial orientation information of the target object in the first image frame. The spatial position information indicates the object's coordinates in the world coordinate system, and the spatial orientation information includes the object's orientation matrix in the world coordinate system. Optionally, after acquiring the first image frame, the computer device performs target detection processing on the first image frame, combines the camera's parameter information, transforms the target object from the camera coordinate system to the world coordinate system, and then obtains the target object's coordinate information and orientation matrix in the camera coordinate system, generating the aforementioned object pose information.

[0044] Step 202: Determine the model pose information and model size information of the AR special effects model based on the camera parameter information and object pose information.

[0045] The camera parameter information refers to the camera's parameter information when acquiring the first image frame. Optionally, different cameras have different parameter information, and the same camera may also have different parameter information under different shooting conditions. This application embodiment does not limit this. The shooting conditions include, but are not limited to, at least one of the following: shooting angle, shooting position, shooting environment, etc. In this application embodiment, the camera parameter information includes the camera's pose matrix and intrinsic parameter matrix. The pose matrix includes the camera's position information and attitude rotation matrix. The camera's attitude rotation matrix indicates the mapping relationship between the camera coordinate system and the world coordinate system, and the camera's intrinsic parameter matrix includes the camera's focal length and optical center. Optionally, in this application embodiment, the camera parameter information is included in the image information, meaning the computer device can acquire the camera parameter information when acquiring the image information. Of course, in practical applications, if the image acquisition unit is a hardware component of the computer device, the computer device can also directly acquire the camera parameter information from itself. For example, the computer device directly acquires the camera parameter information through an IMU (Inertial Measurement Unit).

[0046] AR effects models refer to three-dimensional stereoscopic effects models. Computer devices can render and display AR effects on an image plane using these models. In one possible implementation, different categories of target objects correspond to different AR effects. After acquiring the first image frame, the computer device performs target recognition processing on the first image frame to obtain the category information of the target object, and then determines the corresponding AR effects model based on this category information. The category information indicates the category of the target object. In another possible implementation, different categories of target objects correspond to the same AR effects. After determining that the first image frame contains a target object, the computer device directly acquires the AR effects model.

[0047] In this embodiment, after acquiring the object pose information, the computer device determines the model pose information and model size information of the AR effect model based on the camera parameter information and the object position information. The model pose information includes the spatial position information and spatial orientation information of the AR effect model; the model size information indicates the display size of the effect display model. Optionally, after acquiring the object pose information, the computer device, in conjunction with the camera parameter information, determines the coordinate information and orientation matrix of the AR effect in the world coordinate system, and generates the aforementioned model pose information and model size information.

[0048] Step 203: Based on the model size information, adjust the size of the AR effect model to obtain the adjusted AR effect model.

[0049] In this embodiment of the application, after obtaining the above-mentioned model size information, the computer device performs size adjustment processing on the above-mentioned AR special effects model according to the model size information, thereby obtaining the adjusted AR special effects model.

[0050] Optionally, different AR effect models correspond to different model size information. In practical applications, the root model size information can be flexibly set and adjusted according to the AR effect model. For example, if the AR effect model is a cylindrical surrounding model, the above-mentioned model size adjustment includes the cylinder radius; if the AR effect model is a square surrounding model, the above-mentioned model size adjustment includes the length and width of the square; and so on.

[0051] Step 204: Determine the relative pose information of the AR effect model to the camera based on the camera pose information and the model pose information.

[0052] Camera pose information is used to indicate the camera's position and orientation in the world coordinate system. Here, "camera" refers to the camera used when the first image frame was acquired. Optionally, the camera pose information includes an orientation rotation matrix and the camera's spatial position information. In this case, the camera's spatial position information indicates the camera's position in the world coordinate system when the first image frame was acquired. For example, the camera's spatial position information includes the camera's coordinates in the world coordinate system when the first image frame was acquired.

[0053] In this embodiment, after acquiring the model pose information, the computer device determines the relative pose information of the AR effect model relative to the camera based on the camera pose information and the model pose information. The relative pose information indicates the relative position and orientation between the camera and the target object.

[0054] Optionally, the aforementioned relative pose information includes spatial relative position information and spatial relative attitude information. The spatial relative position information indicates the relative position of the camera and the target object in the world coordinate system, and the spatial relative attitude information indicates the relative attitude of the camera and the target object in the world coordinate system. When acquiring the aforementioned relative pose information, the computer device determines the spatial relative position information of the AR effect model relative to the camera based on the spatial position information of the camera and the spatial position information of the AR effect model, and determines the spatial relative attitude information of the AR effect model relative to the camera based on the attitude rotation matrix and the spatial attitude information of the AR effect model.

[0055] Optionally, in this embodiment, the camera's pose information is included in the image information, meaning the computer device can obtain the camera's pose information when acquiring the image information. It should be noted that since this application requires relative pose information, it does not need to construct a virtual map to determine the camera's exact location. For ease of calculation, the coordinates included in the camera's spatial location information can be (0,0,0).

[0056] Step 205: Based on the relative pose information, the adjusted AR effect model is projected onto the first image frame to generate the effect image frame corresponding to the first image frame.

[0057] In this embodiment, after acquiring the relative pose information, the computer device projects the adjusted AR effect model onto a first image frame based on the relative pose information, generating an effect image frame corresponding to the first image frame. The effect image frame corresponding to the first image frame includes the target object and the AR effect applied to the target object, and the position of the target object in the effect image frame is the same as its position in the first image frame.

[0058] In summary, the technical solution provided in this application provides a way to add AR effects to a target object. By combining the pose information of the target object with the parameter information of the camera, the model pose information and model size information of the AR effect model are determined. Then, the size of the AR effect model is adjusted according to the model size information. The relative pose information between the AR effect model and the camera is determined according to the model pose information. Based on the relative pose information, the adjusted AR effect model is mapped to the first image frame. Finally, the special effects image frame corresponding to the first image frame is obtained. This provides a way to add AR effects to a target object. The mapping effect of the AR effect model in the first image frame is determined according to the relative pose information between the AR effect model and the camera. AR effect images can be generated without building a virtual map, reducing the amount of computation required to generate AR effect images.

[0059] The following section introduces the methods for obtaining object pose information.

[0060] In an exemplary embodiment, step 201 above includes the following steps:

[0061] 1. Perform target detection processing on the first image frame to obtain the first bounding box of the target object.

[0062] The first bounding box is used to indicate the planar position of the target object in the first image frame. For example, as shown... Figure 3 As shown, bounding box 31 is the first bounding box of the target object 32.

[0063] In this embodiment of the application, when the computer device acquires the first bounding box, it performs object detection processing on the first image frame to acquire the first bounding box of the target object. Optionally, the object detection processing is an object detection algorithm. For example, the object detection algorithm can be a one-stage algorithm (such as YOLO V1-V4, SSD (Single Shot Multibox Detector), etc.), a two-stage algorithm (RCNN, FastRCNN, FasterRCNN, etc.), an anchor-free algorithm, etc.

[0064] Optionally, in this embodiment of the application, after the computer device performs target detection processing on the first image frame, it acquires the first bounding box of the target object and the category information of the target object simultaneously. This category information indicates the category to which the target object belongs; different categories of target objects can correspond to different AR effect models.

[0065] It should be noted that, in this embodiment, if the first image frame contains multiple target objects, the computer device, after performing target detection processing on the first image frame, obtains multiple first bounding boxes and multiple category information, and the first bounding boxes and category information subsequently have a correspondence. Optionally, when obtaining bounding boxes for different objects, the computer device marks the bounding boxes corresponding to different objects with different marking information. In one possible implementation, the marking information is the category information, that is, after obtaining bounding boxes for different objects, the computer device marks the corresponding bounding boxes with the category information of the corresponding objects. In another possible implementation, the marking information is encoding information, that is, after obtaining bounding boxes for different objects, the computer device generates encoding information for the bounding box to mark the bounding box. Of course, the computer device can also use other methods to indicate the uniqueness of the bounding box, and this embodiment does not limit this.

[0066] 2. Determine the spatial pose information of the target object based on the planar center position of the first bounding box and the camera parameter information.

[0067] In this embodiment of the application, after acquiring the first bounding box, the computer device determines the spatial pose information of the target object based on the planar center position of the first bounding box and the camera's parameter information. The planar center position indicates the coordinates of the center position of the first bounding box in the camera coordinate system; exemplarily, this planar center position is the coordinate of the center position of the first bounding box in the camera coordinate system.

[0068] Optionally, in this embodiment of the application, when the computer device acquires the above-mentioned spatial attitude information, it determines the connecting ray between the optical center of the camera and the center point of the first bounding box in the world coordinate system based on the planar center position of the first bounding box and the parameter information of the camera, and then determines the deflection angle of the target object in the world coordinate system based on the connecting ray, and determines the spatial attitude information of the target object based on the deflection angle.

[0069] Optionally, the camera parameter information mentioned above includes an attitude rotation matrix and a camera intrinsic parameter matrix, and the spatial attitude information of the target object includes the attitude matrix of the target object in the world coordinate system. For example, assuming the attitude rotation matrix is ​​R = [r0, r1, r2; r3, r4, r5; r6, r7, r8], the camera intrinsic parameter matrix is ​​K = [fx, 0, cx; 0, fy, cy; 0, 0, 1], and the coordinates of the plane center of the first bounding box are (uc, vc), then the connecting ray [vx, vy, vz] is:

[0070] [vx,vy,vz]=RK -1 [uc,vc,1];

[0071] The deflection angle is:

[0072] θ = arctan(vy / vx);

[0073] The pose matrix R0 of the target object is:

[0074]

[0075] In this context, fx and fy are the camera focal lengths, and cx and cy are the camera optical centers.

[0076] 3. Obtain the first distance.

[0077] The first distance is used to indicate the distance between the target object and the camera when the first image frame is acquired. In this embodiment, the computer device acquires the first distance before acquiring the spatial location information of the target object.

[0078] In one possible implementation, when acquiring the first distance, the computer device acquires the depth information of the first image frame, and then determines the first distance based on the depth information.

[0079] In another possible implementation, since the information determined in this application is the relative pose information between the AR effect model and the camera, it is not necessary to construct a virtual map. That is, this application does not need to obtain the accurate distance between the target object and the camera. Therefore, in order to reduce the amount of computation, the computer device can randomly determine a certain value as the first distance mentioned above. For example, the first distance mentioned above is 1.

[0080] 4. Determine the spatial position information of the target object based on the planar center position of the first bounding box, the camera parameter information, and the first distance.

[0081] In this embodiment of the application, after obtaining the first distance, the computer device determines the spatial position information of the target object based on the planar center position of the first bounding box, the camera parameter information, and the first distance.

[0082] Optionally, the camera parameter information includes an attitude rotation matrix and a camera intrinsic parameter matrix, and the spatial position information of the target object includes the target object's coordinates in the world coordinate system. For example, assuming the first distance is d1, and the camera's coordinates in the world coordinate system are (0,0,0) when the first image frame is acquired, then the target object's coordinates in the world coordinate system are [pc x ,pc y ,pc z ]for:

[0083] [pc x ,pc y ,pc z ]=d1*RK -1 [uc,vc,1];

[0084] Where R represents the attitude rotation matrix, K represents the camera intrinsic parameter matrix, and (uc, vc) are the coordinates of the center of the first bounding box in the plane.

[0085] The following section introduces the methods for obtaining model pose and size information.

[0086] Optionally, step 202 above includes the following steps:

[0087] 1. Determine the model size information based on the spatial vertex position information of the first bounding box.

[0088] The first bounding box is used to indicate the planar position of the target object in the first image frame. In this embodiment, when the computer device obtains the above-mentioned model size information, it obtains the spatial vertex information of the first bounding box, and then determines the model size information based on the spatial vertex information of the first bounding box. The spatial vertex information is used to indicate the coordinates of the vertices of the first bounding box in the world coordinate system.

[0089] Optionally, the spatial vertex position information of the first bounding box is included in the spatial position information of the target object. After acquiring the first bounding box, the computer device determines the spatial vertex position of the first bounding box based on the planar vertex position of the first bounding box and in conjunction with the camera parameter information of the camera. The planar vertex position indicates the coordinate information of the vertices of the first bounding box in the camera coordinate system. For example, the planar vertex position is the coordinate of the vertex position of the first bounding box in the camera coordinate system.

[0090] For example, suppose the AR effect model is a cylindrical surrounding model, such as Figure 3 As shown, the planar vertex coordinates of the first bounding box include the coordinates of the top-left corner (u1, v1) and the coordinates of the top-right corner (u2, v2), then the spatial vertex coordinates include the spatial coordinates of the top-left corner [p1]. x p1 y p1 z ] and the spatial coordinates of the upper right corner [p2 x p2 y p2 z ], and the spatial coordinates of the upper left corner are [p1] x p1 y p1 z ]for:

[0091] [p1 x p1 y p1 z ]=d1*RK -1 [u1,v1,1];

[0092] upper right corner spatial coordinates [p2] x p2 y p2 z ]for:

[0093] [p2 x p2 y p2 z ]=d1*RK -1 [u2,v2,1];

[0094] The rotation radius r of the AR special effects model is:

[0095]

[0096] Where R represents the attitude rotation matrix, K represents the camera intrinsic parameter matrix, and d1 represents the first distance mentioned above.

[0097] 2. Determine the spatial position information of the AR effect model based on the spatial vertex position information or the spatial center position information of the first bounding box.

[0098] In this embodiment of the application, when the computer device obtains the spatial location information of the AR effect model, it determines the spatial location information of the AR effect based on the spatial vertex location information or the spatial center location information of the first bounding box.

[0099] Optionally, the spatial center position information of the first bounding box is included in the spatial position information of the target object. After acquiring the first bounding box, the computer device determines the spatial center position of the first bounding box based on the planar center position of the first bounding box and in combination with the camera parameter information of the camera.

[0100] It should be noted that in this application embodiment, the spatial position acquisition methods for different AR effect models are not the same. For example, taking a cylindrical surround model as an example, if the surround center of the AR effect model is the center of the target object, the computer device directly uses the spatial center position information of the first bounding box as the spatial position information of the AR effect model; if the surround center of the AR effect model is the top center of the target object, then the coordinates of the surround center of the AR effect model [p] are used to determine the spatial position information. x ,p y ,p z ]for:

[0101] [pc x ,pc y ,pc z ]=d1*RK -1 [(u1+u2) / 2,(v1+v2) / 2,1];

[0102] Where R represents the attitude rotation matrix, K represents the camera intrinsic parameter matrix, d1 represents the first distance mentioned above, and (u1,v1) and (u2,v2) represent the upper left and upper right corner coordinates of the planar vertices of the first bounding box, respectively.

[0103] 3. Determine the spatial pose information of the AR effect model based on the spatial pose information of the target object.

[0104] In this embodiment of the application, when the computer device acquires the spatial pose information of the AR effect model, it determines the spatial pose information of the AR effect model based on the spatial pose information of the target object. Optionally, the computer device can directly determine the spatial pose information of the target object as the spatial pose information of the AR effect model.

[0105] It should be noted that the above description of the rendering method of AR effects in static images is based on the perspective of the first image frame. In the embodiments of this application, the first image frame can be an image frame obtained from a video stream. The rendering method of AR effects in dynamic image frames will be described below.

[0106] Please refer to Figure 4 The diagram illustrates a flowchart of an AR effects rendering method provided in another embodiment of this application. This method can be applied to... Figure 1 In the special effects display system shown, if the executing entity of each step can be terminal 10 or server 20 (hereinafter collectively referred to as "computer equipment"), the method may include the following steps (401-406):

[0107] Step 401: Obtain at least one associated image frame from the video stream to which the first image frame belongs.

[0108] An associated image frame refers to an image frame belonging to the same video stream as the first image frame and including the aforementioned target object. Optionally, the aforementioned image information includes a video stream. After acquiring the video stream, the computer device performs frame segmentation processing on the video stream, and then obtains multiple image frames containing the target object from the multiple image frames. Among these multiple image frames to be processed, therein are the first image frame and at least one associated image frame corresponding to the first image frame.

[0109] In this embodiment of the application, after acquiring the above-mentioned multiple image frames to be processed, the computer device takes the image frame to be processed corresponding to the first appearance of the target object as the first image frame, and takes the other image frames as the associated image frames.

[0110] Step 402: Perform image processing on the associated image frame to determine the second bounding box of the target object.

[0111] The second bounding box is used to indicate the planar position of the target object in the associated image frame. In this embodiment, after acquiring the associated image frame, the computer device performs image processing on the associated image frame to determine the second bounding box of the target object. Optionally, the image processing includes target tracking processing and / or target detection processing. In one possible implementation, the computer device uses target tracking processing to acquire the second bounding box. Optionally, the computer device determines the position of the second bounding box in the associated image frame based on the position of the bounding box in the previous adjacent image frame. In another possible implementation, the computer device uses target detection processing to acquire the second bounding box. Optionally, the computer device performs target detection processing on a global region of the associated image to acquire the second bounding box.

[0112] Of course, in practical applications, computer devices can also employ target tracking and target detection processing to obtain the aforementioned second bounding box. Optionally, in this embodiment, to ensure the accuracy of the target detection result, the computer device corrects the second bounding box at preset time intervals when obtaining it based on associated image frames. For example, taking a video stream as an example, after determining the first bounding box corresponding to the first image frame, the computer device obtains an image frame containing the target object as an associated image frame of the first image frame. Subsequently, the computer device performs target tracking and detection on the associated image frames in chronological order to obtain the second bounding box, wherein target tracking and detection refers to a method of performing target detection on image frames based on the position of the bounding box in the previous adjacent image frame. In addition, to ensure the accuracy of the target detection result, for the aforementioned multiple associated image frames, every few image frames, the computer device performs global target detection on the associated image frames to determine the second bounding box.

[0113] Optionally, the above object detection processing is an object detection algorithm. For example, the object detection algorithm can be a one-stage algorithm, a two-stage algorithm, an anchor-free algorithm, etc.

[0114] Optionally, in this embodiment of the application, the computer device performs image processing on the associated image frame to obtain the second bounding box of the target object, and at the same time obtains the category information of the target object, so as to ensure that the target object in the first image frame is the target object in the associated image frame, that is, to ensure that the tracking and detection results for the target object are accurate.

[0115] Step 403: Determine the second distance based on the first bounding box, the first distance, and the second bounding box.

[0116] In this embodiment of the application, after acquiring the second bounding box, the computer device determines the second distance based on the first bounding box, the first distance, and the second bounding box. The first bounding box indicates the planar position of the target object in the first image frame, the first distance indicates the distance between the target object and the camera when the first image frame is acquired, and the second distance indicates the distance between the target object and the camera when the associated image frame is acquired.

[0117] Optionally, after acquiring the first bounding box and the second bounding box, the computer device acquires the size parameters of the first bounding box and the second bounding box, and then determines the second distance based on the size parameters and the first distance. The size information includes the width and / or height of the bounding box.

[0118] For example, assuming the width of the first bounding box is w1 and the height is h1, and the width of the second bounding box is w2 and the height is h2, then the second distance d2 is:

[0119]

[0120] Where d1 represents the first distance mentioned above.

[0121] As can be seen from the above, the first bounding box and the second bounding box satisfy the following formula:

[0122]

[0123] Step 404: Determine the associated pose information based on the second distance and the parameter information of the camera corresponding to the associated image frame.

[0124] In this embodiment, after acquiring the second distance, the computer device determines the associated pose information based on the second distance and the parameter information of the camera corresponding to the associated image frame. The associated pose information refers to the pose information of the camera corresponding to the associated image frame. Optionally, the associated pose information includes target spatial position information and a target pose rotation matrix. The target spatial position information indicates the coordinate position of the camera in the world coordinate system when acquiring the associated image frame; the target pose rotation matrix indicates the correspondence between the camera coordinate system and the world coordinate system when acquiring the associated image.

[0125] Optionally, the camera parameter information corresponding to the aforementioned associated image frame refers to the camera parameter information at the time of acquiring the associated image frame. In this embodiment, the camera parameter information corresponding to the aforementioned associated image frame includes a target pose rotation matrix and a target intrinsic parameter matrix. The target intrinsic parameter matrix includes the camera focal length and camera optical center at the time of acquiring the associated image frame.

[0126] Optionally, in this embodiment of the application, when the computer device obtains the above-mentioned associated pose information, it obtains the planar center position of the second bounding box, and then determines the spatial center position of the second bounding box based on the planar center position of the second bounding box and the parameter information of the camera corresponding to the above-mentioned associated image frame, and determines the above-mentioned target spatial position information based on the spatial center position of the second bounding box and the spatial center position of the first bounding box.

[0127] For example, assuming the planar center coordinates of the second bounding box are (uc2, vc2), then the spatial center coordinates of the second bounding box in the camera coordinate system are [pc2]. x ,pc2 y ,pc2 z ]for:

[0128] [pc2 x ,pc2 y ,pc2 z ]=d2*K2 -1[uc2,vc2,1];

[0129] Target spatial location coordinates [t] x ,t y ,t z ]for:

[0130] [t x ,t y ,t z ] = [pc x ,pc y ,pc z ]-R2[pc2 x ,pc2 y ,pc2 z ];

[0131] Where d2 represents the second distance mentioned above, R2 represents the target attitude rotation matrix, K2 represents the target intrinsic parameter matrix, [pc x ,pc y ,pc z [] represents the coordinates of the spatial center of the first bounding box.

[0132] Optionally, the aforementioned camera coordinate system refers to the three-dimensional Cartesian coordinate system corresponding to the associated image frame. For example, the origin of the camera coordinate system is the optical center of the camera, the x-axis is parallel to the x-axis of the image frame, the y-axis is parallel to the y-axis of the image frame, the z-axis is the camera optical axis, and the z-axis is perpendicular to the plane of the image frame.

[0133] Step 405: Determine the associated relative pose information based on the associated pose information and the model pose information.

[0134] In this embodiment, after acquiring the aforementioned associated pose information, the computer device determines the associated relative pose information based on the associated pose information and the aforementioned model pose information. The associated relative pose information refers to the relative pose information of the AR effect model relative to the camera corresponding to the associated image frame.

[0135] Optionally, the aforementioned associated relative pose information includes associated spatial relative position information and associated spatial relative attitude information. When acquiring the aforementioned associated relative pose information, the computer device determines the aforementioned associated spatial relative position information based on the aforementioned target spatial position information and the spatial position information of the AR effect model, and determines the aforementioned associated spatial relative attitude information based on the target attitude rotation matrix and the spatial attitude information of the AR effect model.

[0136] Step 406: Based on the associated relative pose information, the adjusted AR effect model is projected onto the associated image frame to generate the effect image frame corresponding to the associated image frame.

[0137] In this embodiment of the application, after obtaining the aforementioned associated relative pose information, the computer device projects the adjusted AR effect model onto the associated image frame based on the associated relative pose information, thereby generating the effect image frame corresponding to the associated image frame.

[0138] It should be noted that, in the embodiments of this application, for the video stream, the display style of AR effects in the image frames changes as the target object changes. For example, as shown... Figure 5 As shown, the first image frame 51 includes a target object 52 and an AR effect 53. As the target object 52 shrinks, the AR effect shrinks as the target object 52 shrinks, generating the second image frame 54. As the target object 52 enlarges, the AR effect 53 enlarges as the target object 52 enlarges, generating the third image frame 55. Of course, when the target object 52 moves to the edge of the image frame, the AR effect also moves to the edge of the image frame, as in the fourth image frame 56.

[0139] In summary, the technical solution provided in this application, after determining the size and initial pose of the AR effect model in the video stream through the first image frame, determines the relative relationship between the current camera pose and the initial pose of the AR effect model based on the changes in the camera pose in the associated image frames. Then, the size-adjusted AR effect model is mapped to the image frames, realizing that the AR effect in the video stream changes according to the changes in the target object. Moreover, the effect image frame corresponding to the associated image frame can be determined based on the first image frame, without needing to determine the pose of the target object or adjust the size of the AR effect model for each frame, thus reducing the computational load of the computer device.

[0140] The following section describes how to obtain the second bounding box.

[0141] In one possible implementation, step 402 above includes the following steps:

[0142] 1. In response to the fact that the associated image frame is not a sampled image frame, target tracking processing is performed on the associated image frame based on the bounding box position of the target object in the previous adjacent image frame to obtain the second bounding box of the target object;

[0143] 2. In response to the associated image frame being a sampled image frame, target detection processing is performed on the associated image frame to obtain the second bounding box of the target object.

[0144] Optionally, in this embodiment, when the computer device acquires the second bounding box, it samples multiple associated image frames based on a preset time interval to acquire sampled image frames. The preset time interval can be any time interval, and staff can flexibly set and adjust it according to actual conditions; this embodiment does not limit this.

[0145] In this embodiment of the application, after acquiring the above-mentioned sampled image frame, the computer device acquires the above-mentioned second bounding box in different ways for different associated image frames.

[0146] Optionally, if the associated image frame is not a sampled image frame, the computer device performs target tracking processing on the associated image frame based on the bounding box position of the target object in the previous adjacent image frame to obtain a second bounding box of the target object. For example, the computer device uses the bounding box position of the target object in the previous adjacent image frame as a reference and performs target tracking processing on the target object in the surrounding area corresponding to that bounding box position to obtain the second bounding box.

[0147] Optionally, if the associated image is a sampled image frame, the computer device performs target detection processing on the associated image frame to obtain a second bounding box of the target object. For example, the computer device performs a global traversal of the associated image and obtains the second bounding box by performing target detection processing on a global region of the associated image frame.

[0148] In another possible implementation, step 402 above includes the following steps:

[0149] 1. Based on the bounding box position of the target object in the previous adjacent image frame of the associated image frame, perform target tracking processing on the associated image frame to obtain the first target bounding box of the target object in the associated image frame;

[0150] 2. Based on the position of the first target bounding box and the position of the bounding box of the target object in the previous adjacent image frame of the associated image frame, determine the position offset of the first target bounding box;

[0151] 3. In response to the position offset being less than the threshold, the first target bounding box is determined as the second bounding box;

[0152] 4. In response to a position offset greater than a threshold, target detection processing is performed on the associated image frame to obtain the second bounding box of the target object.

[0153] In this embodiment, when acquiring the second bounding box, the computer device performs target detection processing on the associated image frame based on the bounding box position of the target object in the previous adjacent image frame to acquire the first target bounding box of the target object in the associated image frame. Optionally, the computer device uses the bounding box position of the target object in the previous adjacent image frame as a reference and performs target tracking processing on the target object in the surrounding area corresponding to the bounding box position to acquire the first target bounding box.

[0154] Optionally, after acquiring the first target bounding box, the computer device determines the position offset of the first target bounding box based on its position and the position of the bounding box of the target object in the previous adjacent image frame. This offset indicates the positional change between the first target bounding box and the bounding box of the target object in the previous adjacent image frame.

[0155] Subsequently, after acquiring the aforementioned positional offset, the computer device compares the positional offset with a threshold. Optionally, if the positional offset is less than or equal to the threshold, the target detection result indicated by the first target bounding box is determined to be an accurate detection result, and the first target bounding box is identified as the second bounding box; if the positional offset is greater than the threshold, the target detection result indicated by the first target bounding box is determined to be inaccurate, and target detection processing is performed on the associated image frame to obtain the second bounding box of the target object. Optionally, in this case, the computer device performs a global traversal of the associated image and obtains the second bounding box by performing target detection processing on the global region of the associated image frame. The threshold can be any threshold, and operators can flexibly set and adjust the threshold according to actual conditions; this embodiment does not limit this.

[0156] In summary, in one scenario, the computer device samples associated image frames to obtain sampled image frames, then performs target tracking processing on associated image frames that are not sampled, and performs target detection processing on sampled image frames to avoid bounding box acquisition errors caused by target tracking processing. The position of the bounding box in the image frames is detected and adjusted at certain time intervals to ensure the accuracy of the second bounding box acquisition. In another scenario, if the computer device determines that the first target bounding box has a large deviation, it performs global target detection processing on associated image frames with large target detection deviations, and then adjusts the position of the target box in the associated image frames with large target detection deviations to ensure the accuracy and reliability of the second bounding box acquisition.

[0157] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0158] Please refer to Figure 6This diagram illustrates a block diagram of an AR effects rendering apparatus according to an embodiment of this application. The apparatus has the function of implementing the aforementioned AR effects rendering method; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be a computer device or can be installed within a computer device. The apparatus 600 may include: an object pose acquisition module 601, an effects pose acquisition module 602, a model size adjustment module 603, a relative pose acquisition module 604, and an effects image acquisition module 605.

[0159] The object pose acquisition module 601 is used to acquire object pose information based on a first image frame containing the target object; wherein, the object pose information includes the spatial position information and spatial orientation information of the target object in the first image frame.

[0160] The special effects pose acquisition module 602 is used to determine the model pose information and model size information of the AR special effects model based on the camera parameter information and the object pose information; wherein, the model pose information includes the spatial position information and spatial posture information of the AR special effects model.

[0161] The model size adjustment module 603 is used to adjust the size of the AR effect model according to the model size information to obtain the adjusted AR effect model.

[0162] The relative pose acquisition module 604 is used to determine the relative pose information of the AR effect model relative to the camera based on the pose information of the camera and the pose information of the model.

[0163] The special effects image acquisition module 605 is used to project the adjusted AR special effects model onto the first image frame based on the relative pose information, and generate a special effects image frame corresponding to the first image frame.

[0164] In an exemplary embodiment, the object pose acquisition module 601 includes: a bounding box acquisition unit, a pose information determination unit, a first distance acquisition unit, and a position information acquisition unit.

[0165] The bounding box acquisition unit is used to perform target detection processing on the first image frame and acquire a first bounding box of the target object; wherein the first bounding box is used to indicate the planar position of the target object in the first image frame.

[0166] The attitude information determination unit is used to determine the spatial attitude information of the target object based on the planar center position of the first bounding box and the parameter information of the camera.

[0167] A first distance acquisition unit is configured to acquire a first distance; wherein the first distance is used to indicate the distance between the target object and the camera when the first image frame is acquired.

[0168] The location information acquisition unit is used to determine the spatial location information of the target object based on the planar center position of the first bounding box, the parameter information of the camera, and the first distance.

[0169] In an exemplary embodiment, the attitude information determination unit is configured to determine, in a world coordinate system, a connecting ray between the optical center of the camera and the center point of the first bounding box based on the planar center position of the first bounding box and the parameter information of the camera; determine the deflection angle of the target object in the world coordinate system based on the connecting ray; and determine the spatial attitude information of the target object based on the deflection angle.

[0170] In an exemplary embodiment, the spatial position information of the target object includes the spatial vertex position information of the first bounding box and the spatial center position information of the first bounding box. The first bounding box is used to indicate the planar position of the target object in the first image frame. The special effects pose acquisition module 602 is used to determine the model size information based on the spatial vertex position information of the first bounding box; determine the spatial position information of the AR special effects model based on the spatial vertex position information or the spatial center position information of the first bounding box; and determine the spatial pose information of the AR special effects model based on the spatial pose information of the target object.

[0171] In an exemplary embodiment, the camera's pose information includes an attitude rotation matrix and the camera's spatial position information; wherein, the attitude rotation matrix refers to the mapping relationship between the camera coordinate system and the spatial coordinate system; the relative pose acquisition module 604 is used to determine the spatial relative position information of the AR effect model relative to the camera based on the camera's spatial position information and the AR effect model's spatial position information; and to determine the spatial relative pose information of the AR effect model relative to the camera based on the attitude rotation matrix and the AR effect model's spatial pose information; wherein, the relative pose information includes the spatial relative position information and the spatial relative pose information.

[0172] In an exemplary embodiment, such as Figure 7 As shown, the device 600 further includes: an associated image acquisition module 606, a bounding box determination module 607, a second distance determination module 608, and an associated pose acquisition module 609.

[0173] The associated image acquisition module 606 is used to acquire at least one associated image frame from the video stream to which the first image frame belongs, wherein the associated image frame includes the target object.

[0174] The bounding box determination module 607 is used to perform image processing on the associated image frame to determine a second bounding box of the target object; wherein the second bounding box is used to indicate the planar position of the target object in the associated image frame.

[0175] The second distance determination module 608 is used to determine a second distance based on a first bounding box, a first distance, and a second bounding box; wherein the first bounding box is used to indicate the planar position of the target object in the first image frame, the first distance is used to indicate the distance between the target object and the camera when the first image frame is acquired, and the second distance is used to indicate the distance between the target object and the camera when the associated image frame is acquired.

[0176] The associated pose acquisition module 609 is used to determine associated pose information based on the second distance and the camera parameter information corresponding to the associated image frame; wherein, the associated pose information refers to the pose information of the camera corresponding to the associated image frame.

[0177] The relative pose acquisition module 604 is further configured to determine associated relative pose information based on the associated pose information and the model pose information; wherein, the associated relative pose information refers to the relative pose information of the AR effect model relative to the camera corresponding to the associated image frame.

[0178] The special effects image acquisition module 605 is further configured to project the adjusted AR special effects model onto the associated image frame based on the associated relative pose information, thereby generating a special effects image frame corresponding to the associated image frame.

[0179] In an exemplary embodiment, the bounding box determination module 607 is configured to, in response to the associated image frame not being a sampled image frame, perform target tracking processing on the associated image frame based on the bounding box position of the target object in the previous adjacent image frame of the associated image frame to obtain a second bounding box of the target object; and in response to the associated image frame being a sampled image frame, perform target detection processing on the associated image frame to obtain a second bounding box of the target object.

[0180] In an exemplary embodiment, such as Figure 7 As shown, the device 600 further includes: an associated image sampling module 610.

[0181] The associated image sampling module 610 is used to sample multiple associated image frames based on a preset time interval to obtain the sampled image frames.

[0182] In an exemplary embodiment, the bounding box determination module 607 is configured to perform target tracking processing on the associated image frame based on the bounding box position of the target object in the previous adjacent image frame of the associated image frame to obtain a first target bounding box of the target object in the associated image frame; determine a position offset of the first target bounding box based on the position of the first target bounding box and the bounding box position of the target object in the previous adjacent image frame of the associated image frame; and determine the first target bounding box as the second bounding box in response to the position offset being less than a threshold.

[0183] In response to the position offset being greater than a threshold, target detection processing is performed on the associated image frame to obtain the second bounding box of the target object.

[0184] In an exemplary embodiment, such as Figure 7 As shown, the device 600 further includes: a special effects model determination module 611.

[0185] The special effects model determination module 611 is used to determine the AR special effects model corresponding to the target object based on the category information of the target object; wherein, the AR special effects models corresponding to different categories of target objects are different.

[0186] In summary, the technical solution provided in this application determines the model pose information and model size information of the AR effect model by combining the pose information of the target object with the parameter information of the camera. Then, the size of the AR effect model is adjusted according to the model size information. The relative pose information between the AR effect model and the camera is determined according to the model pose information. Based on the relative pose information, the adjusted AR effect model is mapped to the first image frame, and finally the special effect image frame corresponding to the first image frame is obtained. This provides a way to add AR effects to a target object. The mapping effect of the AR effect model in the first image frame is determined based on the relative pose information between the AR effect model and the camera. AR effect images can be generated without building a virtual map, reducing the amount of computation required to generate AR effect images.

[0187] Please refer to Figure 8 This diagram illustrates the structural block diagram of a computer device provided in one embodiment of this application. This computer device can be used to implement the functions of the aforementioned AR special effects rendering method. Specifically:

[0188] Computer device 800 includes a central processing unit (CPU) 801, a system memory 804 including random access memory (RAM) 802 and read-only memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the CPU 801. Computer device 800 also includes a basic input / output system (I / O system) 806 that facilitates information transfer between various devices within the computer, and a mass storage device 807 for storing the operating system 813, application programs 814, and other program modules 815.

[0189] The basic input / output system 806 includes a display 808 for displaying information and an input device 809 for user input, such as a mouse or keyboard. Both the display 808 and the input device 809 are connected to the central processing unit 801 via an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include the input / output controller 810 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, printer, or other types of output devices.

[0190] Mass storage device 807 is connected to central processing unit 801 via a mass storage controller (not shown) connected to system bus 805. Mass storage device 807 and its associated computer-readable media provide non-volatile storage for computer device 800. That is, mass storage device 807 may include computer-readable media (not shown) such as hard disk or CD-ROM (CompactDisc Read-Only Memory) drive.

[0191] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 804 and mass storage device 807 described above can be collectively referred to as memory.

[0192] According to various embodiments of this application, the computer device 800 can also be connected to a remote computer on a network, such as the Internet, for operation. That is, the computer device 800 can be connected to a network 812 via a network interface unit 811 connected to the system bus 805, or the network interface unit 811 can be used to connect to other types of networks or remote computer systems (not shown).

[0193] The memory also includes a computer program stored in the memory and configured to be executed by one or more processors to implement the above-described AR effects rendering method.

[0194] In an exemplary embodiment, a computer-readable storage medium is also provided, the storage medium storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being executed by a processor of a computer device to implement the above-described AR special effects rendering method.

[0195] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0196] In an exemplary embodiment, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned AR effects rendering method.

[0197] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not impose any limitations on this.

[0198] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An augmented reality (AR) special effects rendering method, characterized in that, The method includes: A target detection process is performed on a first image frame containing a target object to obtain a first bounding box of the target object. The target object is an object with a solid shape. The first bounding box is used to indicate the planar position of the target object in the first image frame. Based on the planar center position of the first bounding box and the camera's parameter information, a connecting ray between the camera's optical center and the center point of the first bounding box is determined in the world coordinate system. The planar center position of the first bounding box is used to indicate the coordinate information of the center position of the first bounding box in the camera coordinate system. The deflection angle of the target object in the world coordinate system is determined based on the connecting ray. The spatial pose information of the target object in the first image frame is determined based on the deflection angle. A first distance is obtained, which indicates the distance between the target object and the camera when the first image frame is acquired; Based on the planar center position of the first bounding box, the parameter information of the camera, and the first distance, the spatial position information of the target object in the first image frame is determined. The spatial position information of the target object includes the spatial vertex position information and the spatial center position information of the first bounding box. The model size information is determined based on the spatial vertex position information of the first bounding box; the spatial position information of the AR effect model is determined based on the spatial vertex position information or the spatial center position information of the first bounding box, and the spatial position information of different AR effect models is obtained in different ways; the spatial pose information of the AR effect model is determined based on the spatial pose information of the target object in the first image frame. Based on the model size information, the AR effect model is resized to obtain the adjusted AR effect model. Based on the camera pose information and the model pose information, the relative pose information of the AR effect model with respect to the camera is determined. The model pose information includes the spatial position information of the AR effect model and the spatial posture information of the AR effect model. Based on the relative pose information, the adjusted AR effect model is projected onto the first image frame to generate the effect image frame corresponding to the first image frame. Obtain at least one associated image frame from the video stream to which the first image frame belongs, wherein the associated image frame includes the target object; Image processing is performed on the associated image frame to determine a second bounding box of the target object, the second bounding box being used to indicate the planar position of the target object in the associated image frame; A second distance is determined based on the first bounding box, the first distance, and the second bounding box. The second distance is used to indicate the distance between the target object and the camera when the associated image frame is acquired. Based on the second distance and the parameter information of the camera corresponding to the associated image frame, the associated pose information is determined, wherein the associated pose information refers to the pose information of the camera corresponding to the associated image frame. Based on the associated pose information and the model pose information, the adjusted AR effect model is projected onto the associated image frame to generate the effect image frame corresponding to the associated image frame.

2. The method according to claim 1, characterized in that, The camera's pose information includes an attitude rotation matrix and the camera's spatial position information; wherein, the attitude rotation matrix is ​​used to indicate the mapping relationship between the camera coordinate system and the spatial coordinate system; Determining the relative pose information of the AR effect model with respect to the camera based on the camera's pose information and the model's pose information includes: Based on the spatial position information of the camera and the spatial position information of the AR effect model, determine the spatial relative position information of the AR effect model relative to the camera; Based on the posture rotation matrix and the spatial posture information of the AR effect model, the spatial relative posture information of the AR effect model with respect to the camera is determined; The relative pose information includes the spatial relative position information and the spatial relative posture information.

3. The method according to claim 1, characterized in that, The step of projecting the adjusted AR effect model onto the associated image frame based on the associated pose information and the model pose information, and generating an effect image frame corresponding to the associated image frame, includes: Based on the associated pose information and the model pose information, associated relative pose information is determined; wherein, the associated relative pose information refers to the relative pose information of the AR effect model relative to the camera corresponding to the associated image frame; Based on the associated relative pose information, the adjusted AR effect model is projected onto the associated image frame to generate the effect image frame corresponding to the associated image frame.

4. The method according to claim 1, characterized in that, The step of performing image processing on the associated image frame to determine the second bounding box of the target object includes: In response to the fact that the associated image frame is not a sampled image frame, target tracking processing is performed on the associated image frame based on the bounding box position of the target object in the previous adjacent image frame to obtain the second bounding box of the target object; In response to the associated image frame being a sampled image frame, target detection processing is performed on the associated image frame to obtain the second bounding box of the target object.

5. The method according to claim 4, characterized in that, The method further includes: Based on a preset time interval, multiple associated image frames are sampled to obtain the sampled image frames.

6. The method according to claim 4, characterized in that, The step of performing target tracking processing on the associated image frame based on the bounding box position of the target object in the previous adjacent image frame to obtain the second bounding box of the target object includes: Based on the bounding box position of the target object in the previous adjacent image frame of the associated image frame, target tracking processing is performed on the associated image frame to obtain the first target bounding box of the target object in the associated image frame; Based on the position of the first target bounding box and the position of the bounding box of the target object in the previous adjacent image frame of the associated image frame, the position offset of the first target bounding box is determined. In response to the position offset being less than a threshold, the first target bounding box is determined as the second bounding box; In response to the position offset being greater than a threshold, target detection processing is performed on the associated image frame to obtain the second bounding box of the target object.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Based on the category information of the target object, determine the AR effect model corresponding to the target object; Different types of target objects correspond to different AR effect models.

8. An AR special effects rendering device, characterized in that, The device includes: The object pose acquisition module is used to perform target detection processing on a first image frame containing a target object, and to obtain a first bounding box of the target object. The target object is an object with a solid shape. The first bounding box is used to indicate the planar position of the target object in the first image frame. The attitude information determination unit is used to determine, in the world coordinate system, a connecting ray between the optical center of the camera and the center point of the first bounding box, based on the planar center position of the first bounding box and the camera's parameter information, wherein the planar center position of the first bounding box is used to indicate the coordinate information of the center position of the first bounding box in the camera coordinate system; determine the deflection angle of the target object in the world coordinate system based on the connecting ray; and determine the spatial attitude information of the target object in the first image frame based on the deflection angle. A first distance acquisition unit is configured to acquire a first distance, wherein the first distance is used to indicate the distance between the target object and the camera when the first image frame is acquired; The location information acquisition unit is used to determine the spatial location information of the target object in the first image frame based on the planar center position of the first bounding box, the parameter information of the camera, and the first distance. The spatial location information of the target object includes the spatial vertex position information and the spatial center position information of the first bounding box. The special effects pose acquisition module is used to determine the model size information based on the spatial vertex position information of the first bounding box; to determine the spatial position information of the AR special effects model based on the spatial vertex position information or the spatial center position information of the first bounding box, wherein different AR special effects models have different methods of acquiring spatial position information; and to determine the spatial pose information of the AR special effects model based on the spatial pose information of the target object in the first image frame. The model size adjustment module is used to adjust the size of the AR effect model according to the model size information to obtain the adjusted AR effect model. The relative pose acquisition module is used to determine the relative pose information of the AR effect model relative to the camera based on the pose information of the camera and the pose information of the model. The model pose information includes the spatial position information of the AR effect model and the spatial posture information of the AR effect model. The special effects image acquisition module is used to project the adjusted AR special effects model onto the first image frame based on the relative pose information, and generate a special effects image frame corresponding to the first image frame. The associated image acquisition module is used to acquire at least one associated image frame from the video stream to which the first image frame belongs, wherein the associated image frame includes the target object; A bounding box determination module is used to perform image processing on the associated image frame to determine a second bounding box of the target object, wherein the second bounding box is used to indicate the planar position of the target object in the associated image frame; The second distance determination module is used to determine a second distance based on the first bounding box, the first distance, and the second bounding box. The second distance is used to indicate the distance between the target object and the camera when the associated image frame is acquired. The associated pose acquisition module is used to determine associated pose information based on the second distance and the parameter information of the camera corresponding to the associated image frame. The associated pose information refers to the pose information of the camera corresponding to the associated image frame. The special effects image acquisition module is further configured to project the adjusted AR special effects model onto the associated image frame based on the associated pose information and the model pose information, thereby generating a special effects image frame corresponding to the associated image frame.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to implement the AR effects rendering method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement the AR effects rendering method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes computer instructions that are executed to implement the AR effects rendering method as described in any one of claims 1 to 7.

Images