Interface display method, device, equipment, medium and product based on augmented reality

Abstract

The present application discloses an interface display method, device, equipment, medium and program product based on augmented reality, which relates to the field of augmented reality. The method includes: obtaining a live video frame, which is an image frame in a scene video acquired by a live application in a terminal through a camera, and the live video frame includes scene elements; based on the pose recognition result of the scene element in the live video frame, a three-dimensional virtual object corresponding to the pose recognition result is displayed in the live video frame. Through the above method, users can directly drive scene elements during live broadcast to achieve real-time interaction with three-dimensional virtual objects, thereby improving the user experience during live broadcast and the audience's perception of the live broadcast.

Description

Technical Field

[0001] The present application relates to the field of augmented reality, and in particular to an interface display method, device, equipment, medium and product based on augmented reality. Background Art

[0002] Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It is widely used in live shows to achieve interaction between users and virtual objects.

[0003] In related technologies, augmented reality technology is used to directly push the image of a virtual object to a live broadcast room through a live broadcast stream. Users can perceive the position of virtual objects based on the live broadcast stream and interact with the virtual objects.

[0004] However, in the above method, the user needs to observe the live streaming screen in real time to interact with the virtual object, and cannot interact with the virtual object in a realistic way, resulting in poor user experience and poor audience perception. Summary of the invention

[0005] The embodiments of the present application provide an interface display method, device, equipment, medium and product based on augmented reality, which can be used to solve the problem of being unable to directly interact with virtual objects. The technical solution is as follows:

[0006] In one aspect, a method for displaying an interface based on augmented reality is provided, the method comprising:

[0007] Acquire a live video frame, where the live video frame is an image frame in a scene video acquired by a live application in the terminal through a camera, and the live video frame includes scene elements;

[0008] Based on the position and posture recognition result of the scene element in the live video frame, a three-dimensional virtual object corresponding to the position and posture recognition result is displayed in the live video frame.

[0009] On the other hand, an interface display device based on augmented reality is provided, the device comprising:

[0010] An acquisition module, used to acquire a live video frame, wherein the live video frame is an image frame in a scene video acquired by a live application in the terminal through a camera, and the live video frame includes scene elements;

[0011] The display module is used to display a three-dimensional virtual object corresponding to the posture recognition result in the live video frame based on the posture recognition result of the scene element in the live video frame.

[0012] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement an augmented reality-based interface display method as described in any of the above-mentioned embodiments of the present application.

[0013] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement an augmented reality-based interface display method as described in any of the above-mentioned embodiments of the present application.

[0014] On the other hand, a computer program product is provided, including a computer program, which, when executed by a processor, implements the augmented reality-based interface display method as described in any of the above-mentioned embodiments of the present application.

[0015] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0016] By training the posture recognition model, live video frames are obtained, and after the live video frames are input into the posture recognition model, the posture recognition results of the scene elements in the live video frames are output, and based on the posture recognition results, the three-dimensional virtual objects corresponding to the posture recognition results are displayed in the live video frames. This enables users to directly drive scene elements and interact with scene elements during live broadcasts to achieve real-time interaction with virtual objects, thereby improving the user experience during live broadcasts and the audience's viewing experience when watching live broadcasts. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a schematic diagram of an augmented reality-based interface display method provided by an exemplary embodiment of the present application;

[0019] Figure 2 is a structural block diagram of an electronic device provided by an exemplary embodiment of the present application;

[0020] Figure 3 is a flow chart of an augmented reality-based interface display method provided by an exemplary embodiment of the present application;

[0021] Figure 4 is a schematic diagram of scene elements included in a live video frame provided by an exemplary embodiment of the present application;

[0022] Figure 5 is a flowchart of a method for training a posture recognition model provided by another exemplary embodiment of the present application;

[0023] Figure 6 is a flow chart of an augmented reality-based interface display method provided by another exemplary embodiment of the present application;

[0024] Figure 7 is a schematic diagram of selecting and replacing a target scene element provided by an exemplary embodiment of the present application;

[0025] Figure 8 is an application schematic diagram of an interface display method provided by an exemplary embodiment of the present application;

[0026] Fig. 9 is an application schematic diagram of an interface display method provided by another exemplary embodiment of the present application;

[0027] Fig.10 is a flow chart of a method for controlling a three-dimensional virtual object provided by another exemplary embodiment of the present application;

[0028] Fig.11 is a structural block diagram of an augmented reality-based interface device provided by an exemplary embodiment of the present application;

[0029] Fig.12 is a structural block diagram of an augmented reality-based interface device provided by another exemplary embodiment of the present application;

[0030] Fig.13 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0032] First, a brief introduction is given to the terms involved in the embodiments of the present application.

[0033] Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0034] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, training technologies for large image recognition models, operating / interactive systems, and mechatronics. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0035] Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0036] Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and videos, and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0037] Augmented reality technology is also known as augmented reality. AR augmented reality technology is a relatively new technology that integrates real-world information and virtual-world information content. It simulates and processes physical information that is difficult to experience in the real world based on computer and other scientific technologies, and effectively applies virtual information content in the real world. In this process, it can be perceived by human senses, thus achieving a sensory experience beyond reality. After the real environment and virtual objects overlap, they can exist simultaneously in the same screen and space.

[0038] Augmented reality technology can be applied to multiple fields, such as live broadcast, art teaching, project visualization, etc.

[0039] First, this embodiment is described by taking the live broadcast field as an example.

[0040] AR live broadcast is a form of live broadcast. By pushing the three-dimensional image of virtual objects in the live broadcast room, users can interact based on the position of the virtual objects during the live broadcast, thus achieving a live broadcast effect that combines the virtual and the real.

[0041] Among them, posture refers to position and attitude. The position and attitude of any object in the spatial coordinate system can be used to accurately and uniquely represent its position state.

[0042] In order to determine the position of any point in space, it is necessary to introduce a coordinate system in space. The most commonly used coordinate system is the spatial rectangular coordinate system. The spatial coordinate system refers to selecting an arbitrary point O in space and drawing three mutually perpendicular axes Ox, Oy, and Oz through point O. They all take O as the origin and have the same length unit. These three axes are called the x-axis (horizontal axis), the y-axis (vertical axis), and the z-axis (vertical axis). The positive direction conforms to the right-hand rule, that is, holding the z-axis with the right hand, when the four fingers of the right hand turn from the positive direction of the x-axis to the positive direction of the y-axis at an angle of half π, the direction of the thumb is the positive direction of the z-axis.

[0043] Schematically, taking a cube as an example, select any vertex A among the eight vertices of the cube as the origin, and use the extension lines of the three sides directly connected to vertex A as the x-axis (horizontal axis), y-axis (vertical axis), and z-axis (vertical axis) to establish a spatial coordinate system.

[0044] The position is described by the (x, y, z) coordinates of the object in the spatial coordinate system. For example, in a spatial coordinate system with a corner as the origin, the coordinate values ​​of a water cup are (2,3,4) and the unit is meter. This means that the three-dimensional distances between the water cup and the origin are 2 meters, 3 meters, and 4 meters respectively.

[0045] Posture refers to the appearance of the body. In the spatial coordinate system, it is usually described by the angle rx between the object and the x-axis, the angle ry between the object and the y-axis, and the angle rz between the object and the z-axis. In the AR live broadcast scenario, it refers to the posture or body orientation of the virtual object.

[0046] In related technologies, the image of a preset virtual object is pushed to the live broadcast room through augmented reality technology. The movement and posture of the virtual object are not controlled by the user in real time during the live broadcast. The user can grasp the posture of the virtual object in real time by observing the picture in the live stream and interact with the virtual object.

[0047] When interacting with virtual objects through the above method, users need to observe the live stream in real time to interact with the virtual objects, and cannot interact with the virtual objects in a realistic way, resulting in poor user experience and poor audience perception.

[0048] In response to the above problems, an embodiment of the present application provides an interface display method based on augmented reality, which realizes interaction with virtual objects by driving real objects, thereby improving the user's experience during live broadcast and the audience's viewing experience when watching the live broadcast.

[0049] In the embodiments of the present application, schematically, Figure 1 A schematic diagram of an augmented reality-based interface display method provided by an exemplary embodiment of the present application is shown. Figure 1 As shown:

[0050] When a user broadcasts live through a terminal, the terminal obtains a live video frame in a real-time video stream. The live video frame includes a scene space 100, and the scene space 100 includes a scene element 110. The scene element 110 is circled with the mouse and used as a target replacement object. At this time, the live video frame is input into a pre-trained posture recognition model, and the posture training model outputs the three-dimensional coordinates of the scene element 110 in the scene space 100 and the corresponding flip angle of the scene element 110. The live video frame is subjected to element cutout processing, and the scene element 110 is cut out from the current position. The rendering engine is used to render the preset three-dimensional virtual object 120 to the original position of the scene element 110, and the replacement between the scene element 110 and the three-dimensional virtual object 120 during the live broadcast is completed. When the user is broadcasting live, the interaction between the user and the virtual object 120 can be achieved by driving the scene element 110.

[0051] The terminal in the present application may be a desktop computer, a laptop computer, a mobile phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, etc. The terminal is installed and running an application that supports pushing real-time video streams, such as an application that supports pushing real-time video streams collected by a vehicle-mounted camera component.

[0052] Figure 2 The electronic device 200 includes an operating system 210 and an application 220 .

[0053] The operating system 210 is the basic software that provides application programs 220 with secure access to the computer hardware.

[0054] The application 220 is an application supporting a virtual scene. Optionally, the application 220 is an application supporting pushing a real-time video stream.

[0055] Combined with the above-mentioned noun introduction and implementation environment description, the interface display method based on augmented reality provided in the embodiment of the present application is described. Please refer to Figure 3 , which shows a flowchart of an augmented reality-based interface display method provided by an exemplary embodiment of the present application, such as Figure 3 As shown, the method includes:

[0056] Step 301, obtaining live video frames.

[0057] When a user uses a live broadcast application in a terminal for real-time live broadcast, the scene video captured by the terminal camera is used as the live video stream, and the live broadcast application pushes the video stream to the live broadcast room.

[0058] The live video frame is an image frame in a scene video acquired by a live application in a terminal through a camera, and the live video frame includes scene elements.

[0059] Optionally, the types of live broadcast application programs include but are not limited to one of the following types: live broadcast software, live broadcast applet, live broadcast website, browser that supports online live broadcast, etc.

[0060] Optionally, when a user uses a live broadcast application to broadcast live, the terminal types that can be selected include but are not limited to one of the following types: a smart phone, a laptop computer, a desktop computer, a tablet computer, etc.

[0061] Optionally, a method for obtaining a live video frame includes but is not limited to one of the following methods:

[0062] 1. Take screenshots of live video streams;

[0063] 2. Save the live video stream and use relevant video software to extract video frames.

[0064] Optionally, the scene elements in the live video frame include but are not limited to at least one of the following elements:

[0065] 1. Live broadcast room facilities: tables, chairs, sofas, air conditioners, bedside tables, etc.;

[0066] 2. Decoration of the live broadcast room: green plants, decorative ornaments, curtains;

[0067] 3. Lighting equipment in the live broadcast room: table lamps, chandeliers, lighting equipment, etc.;

[0068] 4. Static objects: toys, books, cups, etc.

[0069] 5. Dynamic objects: small animals, live broadcast users.

[0070] Indicative, Figure 4 is a schematic diagram of scene elements included in a live video frame in an embodiment of the present application, such as Figure 4 As shown:

[0071] When the user is broadcasting live in the live broadcast room, the live broadcast application pushes the video stream of the scene video to the live broadcast room and captures the live video frame 400 of the video stream. The live video frame 400 includes the live broadcast room decoration, live broadcast room facilities, and static objects: curtains 410, table 420, bedside table 430 and toys 440.

[0072] It is worth noting that in some embodiments, the user can broadcast live outdoors or in an indoor room used for live broadcast; if the user broadcasts live indoors, the scene elements in the live video frame can include any kind of elements, that is, in the live broadcast room, it can include any kind of facilities, equipment and other content, which is not limited in this embodiment.

[0073] It is worth noting that the live broadcast terminal used by the user can be arbitrary, the type of live broadcast application used based on the live broadcast terminal can be arbitrary, the method of obtaining live broadcast video frames can be arbitrary, and the live broadcast video frames can be image frames obtained based on the scene video at any time. This embodiment does not limit this.

[0074] Step 302: Based on the pose recognition result of the scene element in the live video frame, a three-dimensional virtual object corresponding to the pose recognition result is displayed in the live video frame.

[0075] Among them, the scene elements in the live video frame can be arbitrary, so during live broadcast, the user can use the terminal's supporting equipment to select the target scene element in the live video frame, that is, select one of the scene elements as the target to replace it with a preset three-dimensional virtual object.

[0076] Optionally, when the live broadcast terminal is a laptop computer, the camera on the laptop computer captures the scene video, and the image frame in the scene video is used as the live broadcast video frame. The live broadcast video frame includes different types of scene elements, such as live broadcast room facilities and static objects. At this time, the target scene element is designated using a mouse device that is matched with the laptop computer, that is, any scene element in the live broadcast video frame is selected using the mouse as the designated target scene element, and the target scene element is further replaced with a preset three-dimensional virtual object.

[0077] Optionally, when the live broadcast terminal is a smart phone, the camera on the smart phone captures the scene video, and the image frame in the scene video is used as the live broadcast video frame. The live broadcast video frame includes different types of scene elements, such as live broadcast room facilities and static objects. At this time, the target scene element is directly designated on the mobile phone screen using a finger, that is, the finger selects any scene element in the live broadcast video frame as the designated target scene element, and further replaces the target scene element with a preset three-dimensional virtual object.

[0078] The live broadcast terminal turns on the AR function, namely the augmented reality (AR) function, and uses the rendering engine to render the preset three-dimensional virtual object to the position of the target scene element, replacing the target scene element with the preset three-dimensional virtual object.

[0079] The type of the three-dimensional virtual object may be any, including but not limited to any one of a three-dimensional virtual human image, a three-dimensional virtual animal image, a three-dimensional virtual still life image, a three-dimensional virtual cartoon image, and the like.

[0080] Optionally, a static object in the scene element is used as the designated target scene element, and a three-dimensional virtual cartoon image is used as the replaced three-dimensional virtual object.

[0081] Among them, the pose recognition result of the scene element in the live video frame refers to the result of analyzing and outputting the target scene element through the pose recognition model.

[0082] Posture refers to the position and posture of an object in space and its own posture.

[0083] Optionally, the pose of the target scene element is the position and orientation of the target scene element in the scene space in the live video frame.

[0084] The live video frame is input into the preset pose recognition model. The pose recognition model will perform pose recognition and analysis on the specified target scene elements in the live video frame. The pose recognition model will output the pose recognition results of the target scene elements.

[0085] Optionally, the pose recognition result includes: the three-dimensional coordinates of the target scene element in the scene space in the live video frame, and the flip angle corresponding to the target scene element.

[0086] The posture recognition results output by the posture recognition model include the posture information of the target scene element. Based on the posture recognition results, the three-dimensional virtual object corresponding to the posture recognition result is displayed in the live video frame, that is, the preset three-dimensional virtual object is displayed at the original position of the target scene element.

[0087] Optionally, a display method of displaying a three-dimensional virtual object corresponding to the pose recognition result in a live video frame includes but is not limited to one of the following methods:

[0088] 1. Cut out the target scene element in the live video frame and display the 3D virtual object at the original position of the target scene element;

[0089] 2. Overlay processing is performed on the target scene element in the live video frame, and the three-dimensional virtual object is overlaid on the target scene element, so that the three-dimensional virtual object is displayed at the original position of the target scene element.

[0090] It is worth noting that when users use different terminals for live broadcast, the method of selecting the target scene element of the live video frame using the equipment matched with the terminal is arbitrary, and this embodiment does not limit this.

[0091] It is worth noting that the type of target scene element specified in the live video frame can be arbitrary, and the type of three-dimensional virtual object used as a replacement can be arbitrary; the preset posture recognition model used in the above method can be any type of model, the posture recognition result output by the posture recognition model can be arbitrary, the amount and type of information included in the posture recognition result can be arbitrary, and the display method of the three-dimensional virtual object corresponding to the posture recognition result in the live video frame based on the posture recognition result can be arbitrary, and this embodiment does not limit this.

[0092] In summary, by obtaining live video frames, inputting the live video frames into the posture recognition model, outputting the posture recognition results of the scene elements in the live video frames, and displaying the three-dimensional virtual objects corresponding to the posture recognition results in the live video frames based on the posture recognition results, the live broadcast effect combining virtuality and reality can be achieved. Users can directly drive scene elements and interact with scene elements during live broadcast to achieve real-time interaction with virtual objects, which improves the user experience during live broadcast and the audience's viewing experience when watching live broadcasts.

[0093] In some embodiments, the posture recognition model can be trained using different data or in different ways, and the live video frame is input into the posture recognition model. Based on the posture recognition result output by the posture recognition model, a three-dimensional virtual object corresponding to the posture recognition result can be displayed in the live video frame. Figure 5 is a flowchart of a method for training a posture recognition model provided by an embodiment of the present application, such as Figure 5 As shown:

[0094] Step 501: Acquire a sample image.

[0095] Select sample images from the sample image library as training data.

[0096] The sample image includes a sample scene element. The sample image is annotated with reference pose data corresponding to the sample scene element. The reference pose data includes the three-dimensional coordinates of the sample scene element in the scene space and the flip angle corresponding to the sample scene element.

[0097] Optionally, the sample image is a two-dimensional image, and the content in the sample image includes at least one scene element, including but not limited to at least one of the following elements: static objects, people or animals, etc.

[0098] A scene element is designated as a sample scene element in the sample image. The sample scene element is mainly used to train the pose recognition model. The sample image is annotated with reference pose data corresponding to the sample scene element, that is, the three-dimensional coordinates of the sample scene element in the scene space with the sample image as the sample image, and the flip angle corresponding to the sample scene element.

[0099] Optionally, the three-dimensional coordinates include (x, y, z) coordinates in three-dimensional space.

[0100] Optionally, the flip angle includes the following angles: pitch angle, yaw angle, and roll angle.

[0101] Pitch angle, yaw angle, and roll angle are concepts based on the body coordinate system and inertial coordinate system.

[0102] Among them, the body coordinate system refers to a three-dimensional orthogonal rectangular coordinate system fixed on the aircraft or plane that follows the right-hand rule, and its origin is located at the center of mass of the aircraft. The OX axis is located in the aircraft reference plane, parallel to the fuselage axis and points to the front of the aircraft, the OY axis is perpendicular to the aircraft reference plane and points to the right of the aircraft, and the OZ axis is perpendicular to the XOY plane in the reference plane and points to the bottom of the aircraft.

[0103] The inertial coordinate system is created to simplify the transformation from the world coordinate system to the object coordinate system. The origin of the inertial coordinate system coincides with the origin of the object coordinate system, and the axes of the inertial coordinate system are parallel to the axes of the world coordinate system. After the introduction of the inertial coordinate system, the object coordinate system only needs to be rotated to transform to the inertial coordinate system, and the transformation from the inertial coordinate system to the world coordinate system only needs to be translated.

[0104] The pitch angle is the angle between the OX axis of the body coordinate system and the horizontal plane: when the OX axis of the body coordinate system is above the XOY plane of the inertial coordinate system, the pitch angle is positive, otherwise it is negative; that is, the angle between the vector parallel to the fuselage axis and pointing to the front of the aircraft and the ground. It is worth noting that both the body coordinate system and the inertial coordinate system here use the right-hand coordinate system.

[0105] The yaw angle is the angle between the projection of the aircraft axis on the horizontal plane and the earth's axis.

[0106] Among them, the earth's axis is the earth's oblique axis, also known as the earth's rotation axis. Specifically, it refers to the axis around which the earth rotates. The intersection of its north end and the earth's surface is the North Pole, and the intersection of its south end and the earth's surface is the South Pole. The fuselage axis is the axis along the direction of the aircraft body.

[0107] The roll angle is the angle between the OZ axis of the body coordinate system and the vertical plane passing through the body axis. The roll is positive when the body rolls to the right, and negative when the body rolls to the right.

[0108] In some embodiments, the yaw angle, pitch angle and roll angle are used to describe the attitude of aircraft such as aircraft or missiles. In this embodiment, the pitch angle, yaw angle and roll angle are used to represent the orientation of the sample scene element in the scene space corresponding to the sample image.

[0109] It is worth noting that the sample image can be any type of two-dimensional image or image of other dimensions, and the source of the sample image can be arbitrary; if the sample image comes from a sample image library, the type and quantity of sample images in the sample image library can be arbitrary; the type of sample scene elements included in the sample image can be arbitrary, and the number and type of scene elements included in the sample image can be arbitrary; the reference pose data corresponding to the sample scene elements annotated in the sample image can be arbitrary, and the reference pose data includes but is not limited to the three-dimensional coordinates of the sample scene elements in the scene space, and the flip angle corresponding to the sample scene elements, that is, the reference pose data can also include other types of data; this embodiment is not limited to this.

[0110] Step 502: Perform pose recognition on sample scene elements in the sample image using a candidate pose recognition model to obtain predicted pose data.

[0111] The candidate pose recognition model is a preset model, and the predicted pose data is used to represent the predicted pose of the sample scene elements.

[0112] Optionally, the candidate pose recognition model is a convolutional neural network based model, which is composed of multiple convolutional layers stacked together.

[0113] Among them, Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input information according to their hierarchical structure. Therefore, they are also called "Shift-Invariant Artificial Neural Networks (SIANN)".

[0114] After the sample image is input into the candidate pose recognition model, the candidate pose recognition model performs pose recognition on the sample scene elements in the sample image and obtains predicted pose data.

[0115] The predicted pose data includes the predicted three-dimensional coordinates of the sample scene elements in the scene space with the sample image as the sample image, and the predicted flip angle corresponding to the sample scene elements.

[0116] Optionally, the predicted three-dimensional coordinates include (x, y, z) coordinates in three-dimensional space.

[0117] Optionally, the predicted flip angle includes the following angles: pitch angle, yaw angle, and roll angle.

[0118] It is worth noting that the candidate pose recognition model is a preset model, and the type of the model can be arbitrary. The predicted pose data output by the candidate pose recognition model includes but is not limited to the predicted three-dimensional coordinates of the sample scene elements in the scene space, and the predicted flip angle corresponding to the sample scene elements, that is, the predicted pose data may also include other types of data, which is not limited in this embodiment.

[0119] Step 503: Based on the difference between the reference pose data and the predicted pose data, the candidate pose recognition model is trained to obtain a pose recognition model.

[0120] The reference pose data is the actual pose information of the sample scene element in the scene space corresponding to the sample image, and the predicted pose data is the predicted pose information of the sample scene element in the scene space corresponding to the sample image by the candidate pose recognition model.

[0121] The difference between the reference pose data and the predicted pose data is calculated to obtain the pose loss value, and the candidate model is trained based on the difference until the difference between the predicted pose data output by the candidate pose recognition model and the reference pose data no longer decreases.

[0122] Schematically, the position of the sample scene element in the scene space corresponding to the sample image is P0, and the reference pose data S0 is:

[0123] 1. Coordinates in three-dimensional space (x0, y0, z0);

[0124] 2. Flip angle: pitch angle A0, yaw angle B0, roll angle C0.

[0125] The position of the sample scene element output by the candidate pose recognition model in the scene space corresponding to the sample image is P1, and the predicted pose data S1 is:

[0126] 1. Coordinates in three-dimensional space (x1, y1, z1);

[0127] 2. Flip angle: pitch angle A1, yaw angle B1, roll angle C1.

[0128] The difference between the reference pose data and the predicted pose data is calculated to obtain the pose loss value, which includes two parts: coordinate loss value and flip angle loss value.

[0129] Schematically, the coordinate loss values ​​of the three-dimensional space are (x0-x1, y0-y1, z0-z1), and the flip angle loss values ​​are: (A0-A1), (B0-B1), (C0-C1).

[0130] The method of training the candidate pose recognition model based on the pose loss value includes but is not limited to at least one of the following methods:

[0131] 1. The coordinate loss value and the flip angle loss value are directly fused to obtain the pose loss value, and the candidate pose recognition model is trained based on the pose loss value;

[0132] 2. Perform weighted fusion on the coordinate loss value and the flip angle loss value, that is, use their corresponding weights to fuse the coordinate loss value and the flip angle loss value, calculate the weighted sum, obtain the pose loss value, and train the candidate pose recognition model based on the pose loss value;

[0133] Optionally, the posture loss value is returned to the candidate posture recognition model for training, and the above process is repeated until the difference between the predicted posture data output by the candidate posture recognition model and the reference posture data no longer changes. The training is completed and the posture recognition model is obtained.

[0134] It is worth noting that the method of calculating the difference between the reference pose data and the predicted pose data to obtain the pose loss value can be arbitrary, including but not limited to the above method; the method of training the candidate pose recognition model based on the pose loss value can be arbitrary, including but not limited to at least one of the above methods; this embodiment does not limit this.

[0135] It is worth noting that the above method trains the candidate pose recognition model until the difference between the predicted pose data output by the candidate pose recognition model and the reference pose data no longer changes, and the training is completed. In some embodiments, the sign of completion of training can be arbitrary, that is, whether the training of the candidate pose recognition model is completed can be judged by other methods or other data indicators.

[0136] To summarize, by obtaining a sample image, the sample image includes sample scene elements, the sample image is annotated with reference pose data corresponding to the sample scene elements, and a candidate pose recognition model is used to perform pose recognition on the sample scene elements in the sample image to obtain predicted pose data; based on the difference between the reference pose data and the predicted pose data, a method for training the candidate pose recognition model is implemented, and the pose recognition model obtained has more accurate pose recognition results for the sample scene elements.

[0137] In some embodiments, during live broadcast, users can achieve real-time interaction with virtual objects through an augmented reality-based interface display method. Figure 6 A flowchart of an augmented reality-based interface display method provided by another exemplary embodiment of the present application is shown. Figure 6 As shown, the method includes:

[0138] Step 601, obtaining live video frames.

[0139] Similar to the above step 301, the live video frame is an image frame in the scene video acquired by the live application in the terminal through the camera, and the live video frame includes scene elements.

[0140] Step 602: input the live video frame into the posture recognition model.

[0141] The posture recognition model is the model trained from step 501 to step 503 above.

[0142] The live video frame includes at least one type of scene element. When the user uses the terminal for live broadcast, the user can use the device supporting the terminal to select the target scene element in the live video frame to replace it with a preset three-dimensional virtual object.

[0143] Optionally, when the live broadcast terminal is a laptop computer, a mouse device matching the laptop computer is used to specify the target scene element.

[0144] The terminal receives an element specifying operation, wherein the element specifying operation is used to specify a scene element to be replaced in a scene video.

[0145] The method of receiving the element specified operation and selecting the target scene element includes but is not limited to at least one of the following methods:

[0146] 1. Receive a range framing operation on a live video frame of a scene video, where the image range selected by the range framing operation includes scene elements; for example, according to the range framing prompt, use a mouse to frame out a target scene element.

[0147] 2. Receive a long press operation on the live video frame of the scene video, where the target point corresponding to the long press operation is a point within the display range of the scene element; for example, long press the left mouse button to circle the area around the target scene element, which only includes the target scene element; or long press the target point within the display range of the target scene element.

[0148] It is worth noting that when users use different terminals for live broadcast, the method of selecting the target scene element of the live video frame using the device matching the terminal is arbitrary, that is, the method of the terminal receiving the element specification operation can be arbitrary, and this embodiment does not limit this.

[0149] Step 603: Perform posture recognition on scene elements in the live video frame through a posture recognition model to obtain a posture recognition result.

[0150] After the terminal receives the element designation operation, it selects a scene element in the live video frame as the target scene element. After the live video frame is input into the posture recognition model, the posture recognition model will perform posture recognition on the target scene element in the live video frame to obtain a posture recognition result.

[0151] Optionally, the posture recognition result is P, and the posture recognition result includes posture recognition data S:

[0152] 1. The coordinates of the target scene element in three-dimensional space (x, y, z);

[0153] 2. Flip angles of target scene elements: pitch angle A, yaw angle B, roll angle C.

[0154] It is worth noting that the posture recognition results output by the posture recognition model can be arbitrary, and the type and value of the posture recognition data can be arbitrary.

[0155] Step 604: perform element cutout processing on the live video frame to cut out the scene elements from the live video frame.

[0156] Cutting is one of the most common operations in image processing. It is to separate a part of a picture or image from the original picture or image into a separate layer. The main function is to prepare for later synthesis. The methods include direct selection with lasso tool, selection tool, eraser tool, quick mask, pen drawing path and then converting to selection, filter extraction, plug-in filter extraction, channel, calculation, application of image method, etc.

[0157] In this embodiment, the live video frame is subjected to element cutout processing, and the original target scene element is cutout based on the posture recognition result output by the posture recognition model, that is, the position of the target scene element in the scene video.

[0158] It is worth noting that the method of performing the cutout processing on the target scene element can be arbitrary, and this embodiment does not limit this.

[0159] Step 605: Display the three-dimensional virtual object in the live video frame according to the posture recognition result, and replace the scene elements.

[0160] Among them, the three-dimensional virtual object is a preset virtual image, and the type of the three-dimensional virtual object can be arbitrary, including but not limited to any one of a three-dimensional virtual human image, a three-dimensional virtual animal image, a three-dimensional virtual still life image, a three-dimensional virtual cartoon image, etc.

[0161] The live broadcast terminal turns on the AR function, namely the augmented reality (AR) function, and uses the rendering engine to render the preset three-dimensional virtual object to the position of the target scene element, replacing the target scene element with the preset three-dimensional virtual object.

[0162] In the computer field, rendering in computer graphics refers to the process of using software to generate images from models. Models are descriptions of three-dimensional objects in a strictly defined language or data structure, which includes geometry, viewpoint, texture, and lighting information. Models in three-dimensional scenes are rendered according to the set environment, lighting, materials, and rendering parameters.

[0163] In the field of graphics, rendering is the last step of computer graphics (CG), and is also the stage that ultimately makes the image conform to the 3D scene.

[0164] The rendering engine is one of the rendering tools. It renders based on the original model, adds color, lighting, shadow and other content, and finally renders it to the screen and finally presents it to the audience.

[0165] In the above step 604, based on the posture recognition result output by the posture recognition model, the original target scene elements are cut out. After the cut out processing, the original target scene elements disappear, and the preset three-dimensional virtual objects are rendered to the original positions of the target scene elements using the rendering engine. After the scene elements are replaced, the live stream pushes the rendered images to the audience in the live broadcast room, and the images viewed by the audience are images of the three-dimensional virtual objects in the scene video.

[0166] It is worth noting that the method of displaying the three-dimensional virtual object in the live video frame according to the posture recognition result can be arbitrary, including but not limited to using a rendering engine; if a rendering engine is used to render a preset three-dimensional virtual object to the position of the target scene element to achieve the replacement of the target scene element, the type of rendering engine can be arbitrary; in the above method, the target scene element is cut out to achieve the replacement of the target scene element. In some embodiments, other methods can also be used to achieve the replacement, which is not limited in this embodiment.

[0167] Indicative, Figure 7 FIG. 1 is a schematic diagram of selecting and replacing a target scene element provided by an exemplary embodiment of the present application, such as Figure 7 As shown:

[0168] The terminal obtains a live video frame in a real-time video stream, the live video frame includes a scene space 700, the scene space 700 includes a scene element 710, and the scene element 710 is circled with the mouse to be used as a target replacement object. At this time, the live video frame is input into a pre-trained posture recognition model, and the posture training model outputs a posture recognition result 720 of the scene element 710 in the scene space 700, i.e., the position information of the scene element 710. The live video frame is subjected to element cutout processing, the scene element 710 is cut out from the current position, and the preset three-dimensional virtual object 730 is rendered to the original position of the scene element 710 using a rendering engine, completing the replacement between the scene element 710 and the three-dimensional virtual object 730 during live broadcast.

[0169] In summary, by obtaining live video frames, inputting the live video frames into the posture recognition model, outputting the posture recognition results of the scene elements in the live video frames, and displaying the three-dimensional virtual objects corresponding to the posture recognition results in the live video frames based on the posture recognition results, the live broadcast effect combining virtuality and reality can be achieved. Users can directly drive scene elements and interact with scene elements during live broadcast to achieve real-time interaction with virtual objects, which improves the user experience during live broadcast and the audience's viewing experience when watching live broadcasts.

[0170] The method provided in this embodiment improves the display effect of the picture, realizes a live broadcast effect combining the virtual and the real, and improves the user and audience experience by inputting the live video frame into the posture recognition model, outputting the posture recognition results corresponding to the scene elements, and displaying the three-dimensional virtual objects in the live video frame according to the posture recognition results.

[0171] The method provided in this embodiment inputs the live video frame into the posture recognition model, so that the posture recognition model performs posture recognition on the scene elements in the live video frame to obtain a posture recognition result. The posture recognition result includes the three-dimensional coordinates of the scene elements in the scene space and the flip angle corresponding to the scene elements, which can improve the accuracy of the posture recognition result.

[0172] The method provided in this embodiment performs element cutout processing on the live video frame to cut out the target scene elements to be replaced from the live video frame; based on the posture recognition result, the three-dimensional virtual object is displayed in the live video frame to replace the scene elements, thereby achieving a live broadcast effect that combines the virtual and the real, and improving the user and audience experience.

[0173] The method provided in this embodiment specifies the scene elements that need to be replaced in the scene video corresponding to the live video frame by receiving the element specifying operation, thereby improving the accuracy and efficiency of scene element replacement.

[0174] The method provided in this embodiment receives a range framing operation on a live video frame of a scene video, where the image range framed by the range framing operation includes scene elements; or receives a long press operation on a live video frame of a scene video, where the target point corresponding to the long press operation is a point within the display range of the scene element, and the scene element to be replaced can be specified, thereby improving the accuracy and efficiency of scene element replacement.

[0175] In some embodiments, the interface display method based on augmented reality can be applied to AR live broadcast to replace real scene elements with virtual objects to achieve real-time interaction between the host and the virtual objects. Figure 8 is an application diagram of an interface display method provided by an exemplary embodiment of the present application, such as Figure 8 As shown:

[0176] After the user starts live broadcast and turns on the AR function, the target scene elements in the live video frame can be replaced with preset three-dimensional virtual objects through the above-mentioned augmented reality-based interface display method, and real-time interaction with the virtual objects can be achieved by driving the target scene elements.

[0177] When the audience watches the live broadcast, they can see that the three-dimensional virtual object 820 is displayed in the scene space 800 corresponding to the live video frame. The host 810 touches the real target scene element, and the real-time picture of the host 810 touching the three-dimensional virtual object 820 is displayed in the live stream. The interaction between the three-dimensional virtual objects 820 is achieved by driving the real target scene elements, such as Figure 8 As shown, the position and orientation of the three-dimensional virtual object 820 are changed. That is, the three-dimensional virtual object 820 can be directly changed in direction by the hand of the anchor 810 to achieve interaction.

[0178] In summary, through the above method, the real target scene elements are replaced with preset three-dimensional virtual objects, so that the audience can see different virtual objects in the video content pushed in the live broadcast room, which improves the display effect of the live broadcast and realizes the interaction of the combination of virtual and real. When the anchor directly drives the real target scene elements, the scene of direct interaction with the three-dimensional virtual object can be seen in the live video, without the need to observe the posture of the three-dimensional virtual object at any time to adjust the action, which improves the experience of the anchor during the live broadcast and the experience of the user when watching the live broadcast.

[0179] In some embodiments, the interface display method based on augmented reality can also be applied to AR live broadcast, replacing real scene elements with virtual elements to change the display effect of the live broadcast environment. Fig. 9 is an application diagram of an interface display method provided by another exemplary embodiment of the present application. Fig. 9 As shown:

[0180] When the audience watches the live broadcast, they can see that in the scene space 900 corresponding to the live video frame, the live broadcast screen of the anchor 910 is displayed, and the scene space 900 includes some scene elements, such as: target scene element 920. Among them, the target scene element 920 is a desk lamp.

[0181] After the host 910 turns on the AR function, the target scene element 920 can be selected through the above-mentioned augmented reality-based interface display method and replaced with a virtual element, such as a virtual cherry tree 930.

[0182] The picture that the audience sees in the live broadcast room is: the anchor 910 broadcasts live in the scene space 900, and the scene space 900 also includes a virtual cherry tree 930.

[0183] Optionally, the target scene element 920 in the scene space 900 may be replaced with picture elements and landscape elements of different themes, and items in the live broadcast room that are prone to violations may be converted into other scenery.

[0184] It is worth noting that, through the above-mentioned interface display method based on augmented reality, any element in the scene space can be replaced, and the original target scene element can be replaced with any virtual element, which is not limited in this embodiment.

[0185] To sum up, the interface display method based on augmented reality can replace the real target scene elements with virtual elements, change the background or environment of the anchor during the live broadcast, and achieve the effect of optimizing the background; or, change the objects in the live broadcast scene and optimize the layout in the live broadcast scene, thereby improving the user experience during the live broadcast and the audience's viewing experience when watching the live broadcast.

[0186] In some embodiments, when a user performs AR live broadcast, he or she may also specify a specific three-dimensional virtual object before interacting with the three-dimensional virtual object, and control the changes in the appearance state and activity state of the three-dimensional virtual object. Fig.10 is a flowchart of a method for controlling a three-dimensional virtual object provided by another exemplary embodiment of the present application. Fig.10 As shown, the following steps are included.

[0187] Step 1001: Receive a virtual object selection operation.

[0188] The virtual object selection operation is used to select a three-dimensional virtual object from multiple candidate virtual objects.

[0189] Optionally, there are 5 candidate virtual objects, namely virtual object A, virtual object B, virtual object C, virtual object D, and virtual object E.

[0190] A virtual object A is selected as an interactive three-dimensional virtual object. The color of the virtual object A is red, and the type of the virtual object A is an animal.

[0191] It is worth noting that the number of candidate virtual objects can be arbitrary, the type of candidate virtual objects can be arbitrary, the color of candidate virtual objects can be arbitrary, and after receiving the virtual object selection operation, any candidate virtual object can be selected as an interactive three-dimensional virtual object, which is not limited in this embodiment.

[0192] Step 1002: In response to the scene element being active in the scene space, displaying an active animation of a three-dimensional virtual object in a live video.

[0193] After selecting a 3D virtual object, a rendering engine is used to render the 3D virtual object to the position of the scene element in the scene space. When the scene element moves, the corresponding activity animation of the 3D virtual object will also be displayed in the live video.

[0194] The activity animation includes at least one of the movement and position change of the three-dimensional virtual object.

[0195] Optionally, the action of the three-dimensional virtual object is running, and the position of the three-dimensional virtual object changes as the running action proceeds.

[0196] The activity state of the scene element includes any one of a controlled activity state and an automatic activity state.

[0197] Optionally, when the scene element is a static object, that is, the object cannot move by itself, the activity state of the scene element is a controlled activity state. When the user drives the scene element to move during live broadcast, the three-dimensional virtual object will also perform corresponding activities.

[0198] Optionally, when the scene element is a dynamic object, that is, the object can move by itself, the activity state of the scene element is an automatic activity state. The user does not need to drive the scene element during live broadcast, and the scene element can move by itself. When the scene element moves, the three-dimensional virtual object will also perform corresponding activities.

[0199] It is worth noting that the type of scene elements can be arbitrary, the activity state of scene elements can be arbitrary, and the actions of scene elements when they are active can be arbitrary, which are not limited in this embodiment.

[0200] Step 1003: Receive an appearance update operation.

[0201] The appearance update operation is used to control the update of the performance effect of the three-dimensional virtual object.

[0202] Optionally, the color appearance of the three-dimensional virtual object is changed, and the original color of the three-dimensional virtual object is red, and it is changed to blue.

[0203] It is worth noting that when receiving an appearance update operation and controlling a three-dimensional virtual object to update its performance effect, the manner of updating its appearance can be arbitrary, including but not limited to changing its color appearance, etc.; when changing the color appearance of a three-dimensional virtual object, its color can be arbitrary, and this embodiment does not limit this.

[0204] Step 1004 : Displaying the three-dimensional virtual object is switched from a first presentation effect to a second presentation effect based on an appearance update operation.

[0205] After receiving the appearance update operation, the performance effect of the three-dimensional virtual object observed by the audience and users through the live video will also be updated. The first performance effect is the initial performance effect of the three-dimensional virtual object, and the second performance effect is the performance effect after receiving the appearance update operation.

[0206] It is worth noting that the first presentation effect and the second presentation effect of the three-dimensional virtual object can be arbitrary, and the presentation effect switching can be performed any number of times, which is not limited in this embodiment.

[0207] In summary, by receiving virtual object selection operations, you can specify a three-dimensional virtual object as an interactive object for the user's live broadcast, and perform appearance update operations on the three-dimensional virtual object to change the performance of the three-dimensional virtual object in the live video; when the scene elements are driven to make them active, or when the scene elements themselves are active, the three-dimensional virtual object will also be active accordingly, which can increase the fun of the live broadcast, improve the user's interactive experience during the live broadcast, and improve the audience's viewing experience when watching the live broadcast.

[0208] The method provided in this embodiment receives a virtual object selection operation, selects a three-dimensional virtual object from multiple candidate virtual objects, and can specify a virtual object for interaction, thereby improving the interactive experience.

[0209] The method provided in this embodiment displays the activity animation of the three-dimensional virtual object in the live video based on the activity state of the scene elements in the scene space, wherein the activity animation includes at least one of the action and position change of the three-dimensional virtual object, and the activity state of the scene elements includes any one of a controlled activity state and an automatic activity state, thereby increasing the fun of the live broadcast and enriching the expressiveness of the three-dimensional virtual object in the live broadcast.

[0210] The method provided in this embodiment receives an appearance update operation, controls the three-dimensional virtual object to update the performance effect, and switches the display of the three-dimensional virtual object from a first performance effect to a second performance effect based on the appearance update operation, thereby changing the appearance performance effect of the three-dimensional virtual object and increasing the interest of the live broadcast.

[0211] Fig.11 is a structural block diagram of an augmented reality-based interface display device provided by an exemplary embodiment of the present application, such as Fig.11 As shown, the device includes the following parts:

[0212] An acquisition module 1110 is used to acquire a live video frame, where the live video frame is an image frame in a scene video acquired by a live application in the terminal through a camera, and the live video frame includes scene elements;

[0213] The display module 1120 is used to display a three-dimensional virtual object corresponding to the posture recognition result in the live video frame based on the posture recognition result of the scene element in the live video frame.

[0214] In an optional embodiment, if Fig.12 As shown, the display module 1120 includes:

[0215] An input unit 1121 is used to input the live video frame into a posture recognition model, and output the posture recognition result corresponding to the scene element, wherein the posture recognition model is a pre-trained model;

[0216] The display unit 1122 is used to display the three-dimensional virtual object in the live video frame according to the posture recognition result.

[0217] In an optional embodiment, the device further comprises:

[0218] The acquisition module 1110 is further used to acquire a sample image, wherein the sample image includes a sample scene element, and the sample image is annotated with reference pose data corresponding to the sample scene element, wherein the reference pose data includes a three-dimensional coordinate of the sample scene element in a scene space, and a flip angle corresponding to the sample scene element;

[0219] The recognition module 1130 is used to perform posture recognition on the sample scene element in the sample image through a candidate posture recognition model to obtain predicted posture data;

[0220] The training module 1140 is used to train the candidate posture recognition model based on the difference between the reference posture data and the predicted posture data to obtain the posture recognition model.

[0221] In an optional embodiment, the input unit 1121 is also used to input the live video frame into a posture recognition model; perform posture recognition on the scene elements in the live video frame through the posture recognition model to obtain the posture recognition result, which includes the three-dimensional coordinates of the scene elements in the scene space and the flip angle corresponding to the scene elements.

[0222] In an optional embodiment, the display unit 1122 is further used to perform element cutout processing on the live video frame to cut out the scene elements from the live video frame; display the three-dimensional virtual object in the live video frame according to the posture recognition result to replace the scene elements.

[0223] In an optional embodiment, the device further comprises:

[0224] The receiving module 1150 is used to receive an element specifying operation, where the element specifying operation is used to specify the scene element to be replaced in the scene video.

[0225] In an optional embodiment, the receiving module 1150 is further used to receive a range framing operation on the live video frame of the scene video, and the image range framed by the range framing operation includes the scene element; or, to receive a long press operation on the live video frame of the scene video, and the target point corresponding to the long press operation is a point within the display range of the scene element.

[0226] In an optional embodiment, the receiving module 1150 is further used to receive a virtual object selection operation, where the virtual object selection operation is used to select the three-dimensional virtual object from multiple candidate virtual objects.

[0227] In an optional embodiment, the display module 1120 is further used to display an activity animation of the three-dimensional virtual object in the live video in response to the scene element being in an active state in the scene space, wherein the activity animation includes at least one of an action and a position change of the three-dimensional virtual object; and wherein the activity state of the scene element includes any one of a controlled activity state and an automatic activity state.

[0228] In an optional embodiment, the receiving module 1150 is further used to receive an appearance update operation, and the appearance update operation is used to control the update of the performance effect of the three-dimensional virtual object; based on the appearance update operation, the display of the three-dimensional virtual object is switched from the first performance effect to the second performance effect.

[0229] To summarize, the device provided in the embodiment of the present application obtains live video frames by training a posture recognition model, and after inputting the live video frames into the posture recognition model, outputs posture recognition results for scene elements in the live video frames, and displays three-dimensional virtual objects corresponding to the posture recognition results in the live video frames based on the posture recognition results. This enables users to directly drive scene elements and interact with scene elements during live broadcasts to achieve real-time interaction with virtual objects, thereby improving the user experience during live broadcasts and the audience's viewing experience when watching live broadcasts.

[0230] It should be noted that: the interface display device based on augmented reality provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the virtual object generation device provided in the above embodiment and the virtual object generation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0231] Fig.13 The block diagram of the structure of a computer device 1300 provided by an exemplary embodiment of the present application is shown. The computer device 1300 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer IV) player, a laptop computer or a desktop computer. The computer device 1300 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0232] Typically, the computer device 1300 includes a processor 1301 and a memory 1302 .

[0233] The processor 1301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may also include an AI processor, which is used to process computing operations related to machine learning.

[0234] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store at least one instruction, which is used to be executed by the processor 1301 to implement the interactive method of the virtual object provided in the method embodiment of the present application.

[0235] In some embodiments, the computer device 1300 also includes other components, which can be understood by those skilled in the art. Fig.13 The structure shown in the figure does not constitute a limitation on the terminal 1300, and the terminal 1300 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0236] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0237] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the virtual object interaction method as described in any of the above embodiments of the present application.

[0238] An embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the virtual object interaction method as described in any of the above embodiments of the present application.

[0239] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the virtual object interaction method described in any of the above embodiments.

[0240] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0241] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0242] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An interface display method based on augmented reality, characterized in that: The method is executed by a terminal, and includes: Acquire a live video frame, where the live video frame is an image frame in a scene video acquired by a live application in the terminal through a camera, and the live video frame includes scene elements; Inputting the live video frame into a posture recognition model; Performing posture recognition on the scene element in the live video frame through the posture recognition model to obtain a posture recognition result, wherein the posture recognition result includes the three-dimensional coordinates of the scene element in the scene space and the flip angle corresponding to the scene element, wherein the posture recognition model is a model pre-trained based on sample images annotated with reference posture data in a sample library; Displaying the three-dimensional virtual object in the live video frame according to the posture recognition result; In response to the scene element being in an active state in the scene space, displaying an activity animation of the three-dimensional virtual object in the live video, wherein the activity animation includes at least one of an action and a position change of the three-dimensional virtual object, and the activity state of the scene element includes any one of a controlled activity state and an automatic activity state; When the activity state of the scene element is a controlled activity state, the three-dimensional virtual object performs corresponding activities according to the driving of the scene element by the live broadcast user; The training method of the posture recognition model includes: A sample image is obtained, wherein the sample image includes a sample scene element, and the sample image is annotated with the reference pose data corresponding to the sample scene element, wherein the reference pose data includes the three-dimensional coordinates of the sample scene element in the scene space, and a flip angle corresponding to the sample scene element; pose recognition is performed on the sample scene element in the sample image by using a candidate pose recognition model to obtain predicted pose data; a pose loss value is obtained based on the difference between the reference pose data and the predicted pose data, and the candidate pose recognition model is trained based on the pose loss value until the difference between the predicted pose data output by the candidate pose recognition model and the reference pose data no longer changes, thereby obtaining the pose recognition model.

2. The method according to claim 1, characterized in that The step of displaying the three-dimensional virtual object in the live video frame according to the posture recognition result includes: Performing element cutout processing on the live video frame to cut out the scene element from the live video frame; The three-dimensional virtual object is displayed in the live video frame according to the posture recognition result, and the scene element is replaced.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: An element specifying operation is received, where the element specifying operation is used to specify the scene element to be replaced in the scene video.

4. The method according to claim 3, characterized in that The receiving element specifies an operation, including: receiving a range framing operation on a live video frame of the scene video, wherein the image range selected by the range framing operation includes the scene element; or, A long press operation on a live video frame of the scene video is received, wherein a target point corresponding to the long press operation is a point within a display range of the scene element.

5. The method according to claim 1 or 2, characterized in that: Before displaying the three-dimensional virtual object corresponding to the posture recognition result in the live video frame, the method further includes: A virtual object selection operation is received, where the virtual object selection operation is used to select the three-dimensional virtual object from a plurality of candidate virtual objects.

6. The method according to claim 1 or 2, characterized in that: The method further comprises: receiving an appearance update operation, wherein the appearance update operation is used to control the three-dimensional virtual object to update a performance effect; Displaying the three-dimensional virtual object is switched from a first presentation effect to a second presentation effect based on the appearance update operation.

7. An interface display device based on augmented reality, characterized in that: The device comprises: An acquisition module, used to acquire a live video frame, wherein the live video frame is an image frame in a scene video acquired by a live application in a terminal through a camera, and the live video frame includes scene elements; The display module includes an input unit and a display unit, wherein the input unit is used to input the live video frame into a posture recognition model; the posture recognition model is used to perform posture recognition on the scene element in the live video frame to obtain a posture recognition result, wherein the posture recognition result includes the three-dimensional coordinates of the scene element in the scene space and the flip angle corresponding to the scene element, and the posture recognition model is a model pre-trained based on sample images annotated with reference posture data in a sample library; the display unit is used to display the three-dimensional virtual object in the live video frame according to the posture recognition result; in response to the scene element being in an active state in the scene space, the activity animation of the three-dimensional virtual object is displayed in the live video, wherein the activity animation includes at least one of the action and position change of the three-dimensional virtual object, and the activity state of the scene element includes any one of a controlled activity state and an automatic activity state; when the activity state of the scene element is a controlled activity state, the three-dimensional virtual object performs corresponding activities according to the driving of the scene element by the live user; The acquisition module is further used to acquire a sample image, wherein the sample image includes a sample scene element, and the sample image is annotated with the reference pose data corresponding to the sample scene element, wherein the reference pose data includes the three-dimensional coordinates of the sample scene element in the scene space and the flip angle corresponding to the sample scene element; A recognition module, used for performing posture recognition on the sample scene element in the sample image through a candidate posture recognition model to obtain predicted posture data; A training module is used to obtain a posture loss value based on the difference between the reference posture data and the predicted posture data, and to train the candidate posture recognition model based on the posture loss value until the difference between the predicted posture data output by the candidate position recognition model and the reference posture data no longer changes, thereby obtaining the posture recognition model.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the augmented reality-based interface display method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the augmented reality-based interface display method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the interface display method based on augmented reality as described in any one of claims 1 to 6.

Images