A real-time video AR interaction method, device, equipment, and medium

By drawing and overlaying AR tag interfaces on backend devices, the problem of low rendering efficiency and poor timeliness of frontend devices is solved, and efficient AR interactive effects are achieved.

CN116527858BActive Publication Date: 2026-11-17SHENZHEN SUNDUN INFORMATION TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310300274.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-24
Publication Date
2026-11-17
Estimated Expiration
2043-03-24

AI Technical Summary

Technical Problem

Existing technologies for AR interaction in real-time video suffer from low rendering efficiency, poor rendering timeliness, and high latency of front-end devices, resulting in poor AR interaction effects, especially when there are many AR tags and a large amount of communication data is transmitted.

Method used

By drawing the AR tag interface on the backend device and overlaying it with the real-time video screen, the frontend device does not need to receive a large number of AR tags and virtual information, which reduces the amount of communication data transmitted and reduces the performance requirements of the frontend device.

Benefits of technology

It significantly reduces the amount of data transmitted in communication and the performance requirements of front-end devices, avoids problems such as low rendering efficiency, poor rendering timeliness, and high latency, and improves the AR interaction effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116527858B_ABST
    Figure CN116527858B_ABST
Patent Text Reader

Abstract

This invention discloses a real-time video AR interaction method, apparatus, device, and medium. The method includes: determining several AR tags to be drawn in the real-time video frame based on real-time PTZ information of a PTZ camera and several AR tags, performing coordinate transformation, drawing them on a drawing interface, generating an AR tag interface, and overlaying it with video frame data of the real-time video frame to obtain video frame data to be displayed and sending it to a front-end device to display the real-time interactive video frame; when receiving user operation coordinate information sent by the front-end device, determining the interactive object and the corresponding user operation response result, updating the AR tag interface, generating updated video frame data, and sending it to the front-end device to display the updated real-time interactive video frame. This invention effectively avoids the low rendering efficiency, poor timeliness, and high latency of the front-end client caused by the large amount of data transmitted through communication, which in turn affects the AR interaction effect of real-time video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of augmented reality technology, and in particular to a real-time video AR interaction method, apparatus, terminal device, and computer-readable storage medium. Background Technology

[0002] Real-time video with AR capabilities includes various interactive information such as AR tags, interactive patterns, and menus. AR tags are tags with augmented reality content; they add virtual display information to the image captured by a camera or other device to describe objects in the real-world scene. For example, if a subway entrance appears in the camera's view, its name, location, or other related information will not be displayed. Only the real-time image containing the subway entrance will be shown. The relevant information about the subway entrance is associated with its portion in the real-time video, and a virtual information tag displaying one or more pieces of information about the subway entrance is displayed on that portion of the video. This tag is the AR tag corresponding to that subway entrance.

[0003] To achieve AR interaction in real-time video, existing technologies typically employ front-end rendering, where AR tags are rendered into the real-time video via a front-end client. However, this method places high demands on the performance of the front-end device. Furthermore, when the number of AR tags to be rendered is large and the amount of virtual information to be displayed is extensive, the large amount of data transmitted during communication leads to low rendering efficiency, poor rendering timeliness, and high latency for the front-end client, thus affecting the AR interaction effect of the real-time video. Summary of the Invention

[0004] This invention provides a real-time video AR interaction method, apparatus, device, and medium. The backend device draws AR tags on the drawing interface and overlays the AR tag interface with the real-time video screen. The frontend device does not need to receive a large number of AR tags and the virtual information to be displayed, nor does it need to render the AR tags into the real-time video in real time. This significantly reduces the amount of data transmitted and the performance requirements of the frontend device. It effectively avoids the low rendering efficiency, poor rendering timeliness, and high latency of the frontend client caused by the large amount of data transmitted, which would affect the AR interaction effect of the real-time video.

[0005] To address the aforementioned technical problems, a first aspect of this invention provides a real-time video AR interaction method, comprising the following steps:

[0006] Based on the real-time PTZ information of the camera device and the PTZ information of the tags corresponding to several preset AR tags, determine several AR tags to be drawn in the real-time video frame.

[0007] Based on the real-time PTZ information of the PTZ camera, the PTZ information of the AR tags to be drawn is converted into the coordinate information of the AR tags to be drawn in the real-time video frame.

[0008] Based on the coordinate information, the plurality of AR tags to be drawn are drawn on a preset drawing interface to generate an AR tag interface;

[0009] The AR tag interface is overlaid with the video frame data corresponding to the real-time video screen to obtain the video frame data to be displayed and sent to the front-end device so that the front-end device can display the real-time interactive video screen according to the video frame data to be displayed.

[0010] When the user operation coordinate information in the real-time interactive video frame is received from the front-end device, the interactive object and the user operation response result corresponding to the interactive object are determined according to the user operation coordinate information.

[0011] Based on the interaction object and the user operation response, the AR tag interface is updated, and updated video frame data is generated based on the updated AR tag interface and sent to the front-end device so that the front-end device can display the updated real-time interactive video screen based on the updated video frame data.

[0012] As a preferred embodiment, the step of overlaying the AR tag interface with the video frame data corresponding to the real-time video screen to obtain the video frame data to be displayed and sending it to the front-end device, so that the front-end device can display the real-time interactive video screen according to the video frame data to be displayed, specifically includes the following steps:

[0013] The AR tag interface is superimposed on the video frame data corresponding to the real-time video screen to obtain the first video frame data, and the first video frame data is encoded to obtain the video frame data to be displayed.

[0014] The video frame data to be displayed is sent to the front-end device, so that the front-end device can decode the video frame data to obtain the target video frame data, and display the real-time interactive video screen according to the target video frame data.

[0015] As a preferred embodiment, updating the AR tag interface based on the interaction object and the user operation response result specifically includes the following steps:

[0016] Based on the interaction object and the user operation response result, perform the update operation corresponding to the user operation response result on the AR tag interface to generate the updated AR tag interface;

[0017] The update operation includes, but is not limited to, modifying the parameters of any AR tag in the AR tag interface, adding an AR tag in the AR tag interface, deleting any AR tag in the AR tag interface, and marking any AR tag in the AR tag interface.

[0018] As a preferred embodiment, the step of generating updated video frame data based on the updated AR tag interface and sending it to the front-end device, so that the front-end device can display the updated real-time interactive video screen based on the updated video frame data, specifically includes the following steps:

[0019] The updated AR tag interface is overlaid with the video frame data corresponding to the real-time video screen to obtain the second video frame data, and the second video frame data is encoded to obtain the updated video frame data.

[0020] The updated video frame data is sent to the front-end device so that the front-end device can decode the updated video frame data and display the updated real-time interactive video screen based on the decoded updated video frame data.

[0021] As a preferred embodiment, before drawing the plurality of AR tags to be drawn on a preset drawing interface according to the coordinate information and generating the AR tag interface, the method further includes the following steps:

[0022] Based on the resolution of the real-time video frame, the size of the preset initial drawing interface and the size of several AR tags to be drawn are scaled to obtain the preset drawing interface and several AR tags to be drawn after scaling.

[0023] Then, the step of drawing the plurality of AR tags to be drawn on a preset drawing interface according to the coordinate information to generate an AR tag interface specifically includes:

[0024] Based on the coordinate information, the AR tags to be drawn, scaled down to the specified size, are drawn on the drawing interface to generate the AR tag interface.

[0025] As a preferred embodiment, the AR tag interface includes an operation menu set within a preset area;

[0026] The operation menu includes several button icons, each button icon corresponding to any AR tag interface update operation command.

[0027] As a preferred embodiment, after updating the AR tag interface based on the interaction object and the user operation response result, the method further includes the following steps:

[0028] The update operation completion information is generated and fed back to the front-end device.

[0029] A second aspect of the present invention provides an AR interactive device for real-time video, comprising:

[0030] The AR tag confirmation module is used to determine several AR tags to be drawn in the real-time video frame based on the real-time PTZ information of the camera device and the PTZ information of the tags corresponding to several preset AR tags.

[0031] The coordinate transformation module is used to convert the PTZ information of the AR tags to be drawn into the coordinate information of the AR tags to be drawn in the real-time video frame based on the real-time PTZ information of the PTZ.

[0032] The AR tag drawing module is used to draw the plurality of AR tags to be drawn on a preset drawing interface according to the coordinate information, thereby generating an AR tag interface.

[0033] The AR tag interface overlay module is used to overlay the AR tag interface with the video frame data corresponding to the real-time video screen to obtain the video frame data to be displayed and send it to the front-end device so that the front-end device can display the real-time interactive video screen according to the video frame data to be displayed.

[0034] The user operation response module is used to determine the interactive object and the corresponding user operation response result based on the user operation coordinate information when it receives user operation coordinate information sent by the front-end device in the real-time interactive video screen.

[0035] The AR tag interface update module is used to update the AR tag interface according to the interactive object and the user operation response result, generate updated video frame data according to the updated AR tag interface and send it to the front-end device, so that the front-end device can display the updated real-time interactive video screen according to the updated video frame data.

[0036] A third aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the real-time video AR interaction method as described in any of the first aspects.

[0037] A fourth aspect of the present invention provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform an AR interaction method for real-time video as described in any of the first aspects.

[0038] Compared to existing technologies, the beneficial effects of this invention are that by drawing AR tags on the drawing interface through the backend device and overlaying the AR tag interface with the real-time video screen, the frontend device does not need to receive a large number of AR tags and the virtual information to be displayed, nor does it need to render AR tags into the real-time video in real time. This significantly reduces the amount of data transmitted and the performance requirements of the frontend device, effectively avoiding the low rendering efficiency, poor rendering timeliness, and high latency of the frontend client caused by the large amount of data transmitted, which in turn affects the AR interaction effect of the real-time video. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the AR interaction method for real-time video in an embodiment of the present invention.

[0040] Figure 2 This is a flowchart of the AR interaction process for real-time video in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the structure of the AR interactive device for real-time video in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] See Figure 1 and Figure 2 The first aspect of this invention provides a real-time video AR interaction method, comprising the following steps S1 to S6:

[0044] Step S1: Based on the real-time PTZ information of the camera device and the PTZ information of the tags corresponding to the preset AR tags, determine several AR tags to be drawn in the real-time video frame.

[0045] Step S2: Based on the real-time PTZ information of the PTZ, convert the PTZ information of the AR tags to be drawn into the coordinate information of the AR tags to be drawn in the real-time video frame.

[0046] Step S3: Based on the coordinate information, draw the plurality of AR tags to be drawn on a preset drawing interface to generate an AR tag interface;

[0047] Step S4: Overlay the AR tag interface with the video frame data corresponding to the real-time video screen to obtain the video frame data to be displayed and send it to the front-end device so that the front-end device can display the real-time interactive video screen according to the video frame data to be displayed.

[0048] Step S5: When the user operation coordinate information in the real-time interactive video frame sent by the front-end device is received, the interactive object and the user operation response result corresponding to the interactive object are determined according to the user operation coordinate information.

[0049] Step S6: Update the AR tag interface according to the interaction object and the user operation response result, generate updated video frame data according to the updated AR tag interface and send it to the front-end device, so that the front-end device can display the updated real-time interactive video screen according to the updated video frame data.

[0050] In step S1, to accurately determine the AR tags that should be displayed in the current real-time video frame, this embodiment determines the AR tags to be drawn based on the real-time PTZ information of the camera device and the PTZ information of several preset AR tags. It is worth noting that the real-time PTZ information can determine the horizontal angle, pitch angle, and lens zoom parameters of the current camera device. The PTZ information corresponding to each AR tag is set during the AR tag creation stage. For example, when creating a new AR tag in a video frame, firstly, the corresponding pixel coordinates in the video frame are selected, and key parameters such as tag name, remarks, tag color, tag model, and tag density are set. Then, based on the PTZ information of the PTZ corresponding to the video frame, the PTZ information corresponding to the pixel coordinates is calculated. Finally, the key parameters such as tag name, remarks, tag color, tag model, and tag density, as well as the calculated PTZ information, are stored as parameter information for the newly added AR tag. Therefore, by combining the real-time PTZ information and the PTZ information of several AR tags, several AR tags to be drawn in the real-time video frame can be determined.

[0051] In step S2, in order for the drawn AR tags to match the real-time video screen, this embodiment performs coordinate transformation on several AR tags to be drawn based on the real-time PTZ information of the PTZ, and converts the PTZ information of each AR tag to be drawn into its coordinate information in the real-time video screen.

[0052] In step S3, after determining the coordinate information of each AR tag to be drawn, several AR tags to be drawn are drawn on a preset drawing interface according to the coordinate information, thereby generating an AR tag interface. It is worth noting that since the coordinate information of the AR tag corresponds to the pixel coordinates in the real-time video image, the size of the drawing interface needs to be consistent with the resolution of the real-time video image.

[0053] In step S4, to include AR tags in the real-time video feed, this embodiment overlays the AR tag interface onto the corresponding video frame data of the real-time video feed. This ensures that each AR tag to be drawn is accurately displayed at its corresponding position in the real-time video feed. The overlaid video frame data is then sent to the front-end device, which displays the real-time interactive video feed based on the displayed video frame data for user viewing. Therefore, the front-end device does not need to render the AR tags directly into the real-time video, resulting in lower performance requirements for the front-end device.

[0054] In step S5, when the user performs an interactive operation on the real-time interactive video screen, the front-end device will obtain the operation coordinate information of the user in the real-time interactive video screen, which is used to locate the interactive object and determine the user operation response result. Therefore, when the user operation coordinate information sent by the front-end device is received, this embodiment determines the interactive object and the corresponding user operation response result based on the user operation coordinate information.

[0055] In step S6, in order to meet the user's interactive needs for real-time interactive video, this embodiment updates the AR tag interface according to the interactive object and the user's operation response to display the user's operation results and achieve the purpose of interaction. Furthermore, updated video frame data is generated according to the updated AR tag interface and sent to the front-end device so that the front-end device can display the updated real-time interactive video based on the updated video frame data, thereby allowing the user to view their operation results.

[0056] This invention provides a real-time video AR interaction method. The backend device draws AR tags on the drawing interface and overlays the AR tag interface with the real-time video screen. The frontend device does not need to receive a large number of AR tags and the virtual information to be displayed, nor does it need to render the AR tags into the real-time video in real time. This significantly reduces the amount of data transmitted and the performance requirements of the frontend device. It effectively avoids the low rendering efficiency, poor rendering timeliness, and high latency of the frontend client caused by the large amount of data transmitted, which would affect the AR interaction effect of the real-time video.

[0057] As a preferred embodiment, the step of overlaying the AR tag interface with the video frame data corresponding to the real-time video screen to obtain the video frame data to be displayed and sending it to the front-end device, so that the front-end device can display the real-time interactive video screen according to the video frame data to be displayed, specifically includes the following steps:

[0058] The AR tag interface is superimposed on the video frame data corresponding to the real-time video screen to obtain the first video frame data, and the first video frame data is encoded to obtain the video frame data to be displayed.

[0059] The video frame data to be displayed is sent to the front-end device, so that the front-end device can decode the video frame data to obtain the target video frame data, and display the real-time interactive video screen according to the target video frame data.

[0060] Specifically, in order to further reduce the amount of data transmitted during communication and thus reduce interaction latency, this embodiment superimposes the AR tag interface with the video frame data corresponding to the real-time video screen, encodes the obtained first video frame data, obtains the video frame data to be displayed, and sends it to the front-end device. The front-end device only needs to decode the video frame data to be displayed and display the real-time interactive video screen to the user based on the obtained target video frame data, thereby reducing the amount of data transmitted during communication and lowering the performance requirements of the front-end device.

[0061] As a preferred embodiment, updating the AR tag interface based on the interaction object and the user operation response result specifically includes the following steps:

[0062] Based on the interaction object and the user operation response result, perform the update operation corresponding to the user operation response result on the AR tag interface to generate the updated AR tag interface;

[0063] The update operation includes, but is not limited to, modifying the parameters of any AR tag in the AR tag interface, adding an AR tag in the AR tag interface, deleting any AR tag in the AR tag interface, and marking any AR tag in the AR tag interface.

[0064] Specifically, when a user performs interactive operations such as mouse clicks or touches in a real-time interactive video frame, the operation coordinate information corresponding to the operation operation can be used to further determine the operation object corresponding to the operation coordinate information. This operation object is the interaction object, and then the interaction object is responded to to form the user operation response result, such as adding, modifying, or deleting. In order to present the operation result to the user, this embodiment performs an update operation corresponding to the user operation response result on the AR tag interface based on the interaction object and the user operation response result, generating an updated AR tag interface.

[0065] For example, a user can right-click on an AR tag in a real-time interactive video frame to bring up a submenu for that AR tag. The submenu includes functions such as adding a tag, modifying a tag, deleting a tag, and viewing tags. Clicking the pixel coordinates corresponding to one of these sub-functions with the left mouse button triggers that function. When the interaction object is any AR tag and its corresponding "modify tag" sub-function, parameters such as tag name, notes, tag color, tag model, tag density, and priority of the AR tag are modified according to the user's modification parameter values. When the interaction object is any AR tag and its corresponding "add tag" sub-function, the tag name, notes, tag color, tag model, and tag density of the newly added tag are determined based on the tag creation information entered by the user, and the AR tag is added. When the interaction object is any AR tag and its corresponding "delete tag" sub-function, this embodiment deletes the corresponding AR tag from the storage database.

[0066] As a preferred embodiment, the step of generating updated video frame data based on the updated AR tag interface and sending it to the front-end device, so that the front-end device can display the updated real-time interactive video screen based on the updated video frame data, specifically includes the following steps:

[0067] The updated AR tag interface is overlaid with the video frame data corresponding to the real-time video screen to obtain the second video frame data, and the second video frame data is encoded to obtain the updated video frame data.

[0068] The updated video frame data is sent to the front-end device so that the front-end device can decode the updated video frame data and display the updated real-time interactive video screen based on the decoded updated video frame data.

[0069] It is worth noting that, in order to present the updated real-time interactive video to the user in a timely manner after the user interacts with the video, this embodiment overlays the updated AR tag interface with the video frame data corresponding to the real-time video, and encodes the obtained second video frame data to reduce the amount of data transmitted during communication. The updated video frame data is then obtained and sent to the front-end device. The front-end device only needs to decode the updated video frame data and display the updated real-time interactive video to the user based on the decoded updated video frame data, which places low demands on the performance of the front-end device.

[0070] As a preferred embodiment, before drawing the plurality of AR tags to be drawn on a preset drawing interface according to the coordinate information and generating the AR tag interface, the method further includes the following steps:

[0071] Based on the resolution of the real-time video frame, the size of the preset initial drawing interface and the size of several AR tags to be drawn are scaled to obtain the preset drawing interface and several AR tags to be drawn after scaling.

[0072] Then, the step of drawing the plurality of AR tags to be drawn on a preset drawing interface according to the coordinate information to generate an AR tag interface specifically includes:

[0073] Based on the coordinate information, the AR tags to be drawn, scaled down to the specified size, are drawn on the drawing interface to generate the AR tag interface.

[0074] It is worth noting that, since the real-time video footage from camera devices has various resolution formats, such as 4K (3840*1920), 2K (2560×1440), 1080P (1920*1080), 720P (1080*720), D1 (720×480, 704×576), the size of the AR tag drawing interface needs to be strictly consistent with the resolution of the current real-time video footage. The text display size and style of the AR tags in the drawing interface also need to be scaled according to the resolution of the real-time video footage to ensure that the AR tags in the drawing interface accurately correspond to the interactive objects in the real-time video footage.

[0075] Specifically, in this embodiment, the size of the preset initial drawing interface and the size of several AR tags to be drawn are scaled according to the resolution of the real-time video frame. For example, assuming that the initial drawing interface and the initial AR tags to be drawn are both 1080P in size format, and the resolution of the real-time video frame is D1, then the initial drawing interface and the initial AR tags to be drawn are reduced by 2.25 times (min(1920 / 480,1080 / 480)); assuming that the initial drawing interface and the initial AR tags to be drawn are both 1080P in size format, and the resolution of the real-time video frame is 4K, then the initial drawing interface and the initial AR tags to be drawn are enlarged by 1.7 times (min(3840 / 1920,1920 / 1080)).

[0076] As a preferred embodiment, the AR tag interface includes an operation menu set within a preset area;

[0077] The operation menu includes several button icons, each button icon corresponding to any AR tag interface update operation command.

[0078] It is worth noting that, in addition to supporting users to interact with each AR tag in the real-time interactive video screen, this embodiment also supports users to interact with the real-time interactive video screen through the button icons in the operation menu by setting an operation menu in the preset area of ​​the AR tag interface.

[0079] For example, the sub-functions included in the operation menu include: when the operation menu is not activated, displaying the word "AR" in the real-time interactive video screen and occupying a fixed area of ​​the real-time interactive video screen, the area occupied is small. When the front-end device detects that the user clicks on this area, it sends a signal to the back-end device, thereby causing the back-end device to pop up the operation menu function list in the corresponding area of ​​the real-time interactive video screen. The list includes, but is not limited to: enabling and disabling the AR tag display function; changing the AR tag display size function; setting the priority of AR tags and other character overlay layers. For example, if an area contains both tag information and OSD characters (which can be custom characters such as time, custom channel names, PTZ information, etc.), the priority setting can set the upper and lower layer relationship of the tag information and OSD characters, generally with the upper layer obscuring the lower layer; setting the dense layering of AR tags. For example, if 40 tags are added to the screen, different numbers of tags will be displayed at different magnifications. For example, from 1x to 3x, only 10 tags are displayed; from 3x to 10x, only the other 20 tags are displayed; and after 10x, only the remaining 10 tags are displayed. The GPS tag accuracy range display setting, when using a GPS-type AR tag, allows you to determine whether to display an accuracy range area alongside the GPS tag display. This is similar to adding a circle near the GPS tag point. The size of this circle can be adjusted by the user according to this accuracy range display setting to check if the GPS tag matches the actual vehicle or other object. The tag search function activation setting, when the front-end device detects a user clicking this function, sends a signal to the back-end device to activate it. The back-end device responds to this signal by searching for AR tags, marking target AR tags to display the search results, and updating the real-time interactive video feed back to the front-end device. The user can double-click the marked AR tag with the left mouse button to control the camera's pan / tilt unit to rotate to the position corresponding to the target AR tag.

[0080] It should be noted that the sub-functions included in the operation menu are expandable, and the function list is not fixed. This embodiment does not impose specific limitations on them.

[0081] As a preferred embodiment, after updating the AR tag interface based on the interaction object and the user operation response result, the method further includes the following steps:

[0082] The update operation completion information is generated and fed back to the front-end device.

[0083] See Figure 3 A second aspect of the present invention provides an AR interactive device for real-time video, comprising:

[0084] The AR tag confirmation module 301 is used to determine several AR tags to be drawn in the real-time video frame based on the real-time PTZ information of the camera device and the PTZ information of the tags corresponding to several preset AR tags.

[0085] The coordinate transformation module 302 is used to convert the PTZ information of the AR tags to be drawn into the coordinate information of the AR tags to be drawn in the real-time video frame based on the real-time PTZ information of the PTZ.

[0086] AR tag drawing module 303 is used to draw the plurality of AR tags to be drawn on a preset drawing interface according to the coordinate information, and generate an AR tag interface.

[0087] AR tag interface overlay module 304 is used to overlay the AR tag interface with the video frame data corresponding to the real-time video screen to obtain the video frame data to be displayed and send it to the front-end device so that the front-end device can display the real-time interactive video screen according to the video frame data to be displayed.

[0088] The user operation response module 305 is used to determine the interactive object and the corresponding user operation response result based on the user operation coordinate information when it receives user operation coordinate information sent by the front-end device in the real-time interactive video screen.

[0089] AR tag interface update module 306 is used to update the AR tag interface according to the interaction object and the user operation response result, generate updated video frame data according to the updated AR tag interface and send it to the front-end device, so that the front-end device can display the updated real-time interactive video screen according to the updated video frame data.

[0090] As a preferred embodiment, the AR tag interface overlay module 304 is used to overlay the AR tag interface with the video frame data corresponding to the real-time video screen to obtain the video frame data to be displayed and send it to the front-end device, so that the front-end device can display the real-time interactive video screen according to the video frame data to be displayed, specifically including:

[0091] The AR tag interface is superimposed on the video frame data corresponding to the real-time video screen to obtain the first video frame data, and the first video frame data is encoded to obtain the video frame data to be displayed.

[0092] The video frame data to be displayed is sent to the front-end device, so that the front-end device can decode the video frame data to obtain the target video frame data, and display the real-time interactive video screen according to the target video frame data.

[0093] As a preferred embodiment, the AR tag interface update module 306 is used to update the AR tag interface according to the interaction object and the user operation response result, specifically including:

[0094] Based on the interaction object and the user operation response result, perform the update operation corresponding to the user operation response result on the AR tag interface to generate the updated AR tag interface;

[0095] The update operation includes, but is not limited to, modifying the parameters of any AR tag in the AR tag interface, adding an AR tag in the AR tag interface, deleting any AR tag in the AR tag interface, and marking any AR tag in the AR tag interface.

[0096] As a preferred embodiment, the AR tag interface update module 306 is used to generate updated video frame data based on the updated AR tag interface and send it to the front-end device, so that the front-end device can display the updated real-time interactive video screen based on the updated video frame data, specifically including:

[0097] The updated AR tag interface is overlaid with the video frame data corresponding to the real-time video screen to obtain the second video frame data, and the second video frame data is encoded to obtain the updated video frame data.

[0098] The updated video frame data is sent to the front-end device so that the front-end device can decode the updated video frame data and display the updated real-time interactive video screen based on the decoded updated video frame data.

[0099] As a preferred embodiment, the device further includes a size scaling module for:

[0100] Based on the resolution of the real-time video frame, the size of the preset initial drawing interface and the size of several AR tags to be drawn are scaled to obtain the preset drawing interface and several AR tags to be drawn after scaling.

[0101] Then, the AR tag drawing module 303 is used to draw the plurality of AR tags to be drawn on a preset drawing interface according to the coordinate information, and generate an AR tag interface, specifically including:

[0102] Based on the coordinate information, the AR tags to be drawn, scaled down to the specified size, are drawn on the drawing interface to generate the AR tag interface.

[0103] As a preferred embodiment, the AR tag interface includes an operation menu set within a preset area;

[0104] The operation menu includes several button icons, each button icon corresponding to any AR tag interface update operation command.

[0105] As a preferred embodiment, the device further includes an operation completion information feedback module, used for:

[0106] After the AR tag interface update module 306 updates the AR tag interface according to the interaction object and the user operation response result, it generates update operation completion information and feeds it back to the front-end device.

[0107] It should be noted that the real-time video AR interaction device provided in this embodiment of the invention can realize all the processes of the real-time video AR interaction method described in any of the above embodiments. The functions and technical effects of each module in the device are the same as the functions and technical effects of the real-time video AR interaction method described in the above embodiments, and will not be repeated here.

[0108] A third aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the real-time video AR interaction method as described in any embodiment of the first aspect.

[0109] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. The terminal device may also include input / output devices, network access devices, buses, etc. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines. The memory can be used to store the computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function (such as sound playback or image playback). The data storage area may store data created based on the use of the mobile phone (such as audio data and phonebook entries). Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0110] A fourth aspect of the present invention provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform an AR interaction method for real-time video as described in any embodiment of the first aspect.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary hardware platforms, and of course, it can also be implemented entirely by hardware. Based on this understanding, all or part of the technical solution of the present invention that contributes to the background art can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0112] In summary, the AR interaction method, apparatus, device, and medium for real-time video provided by the embodiments of the present invention have the following beneficial effects:

[0113] (1) The AR tags are drawn on the drawing interface by the back-end device and the AR tag interface is superimposed on the real-time video screen. The front-end device does not need to receive a large number of AR tags and virtual information to be displayed, nor does it need to render the AR tags into the real-time video in real time. This significantly reduces the amount of data transmitted and the performance requirements of the front-end device, effectively avoiding the low rendering efficiency, poor rendering timeliness and high latency of the front-end client due to the large amount of data transmitted, which in turn affects the AR interaction effect of the real-time video.

[0114] (2) The front-end device does not need to calculate the coordinate information of the AR tag in real time, thus avoiding the inconsistency of calculation standards due to the differences in algorithms of different manufacturers.

[0115] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for AR interaction of real-time video, characterized in that, Includes the following steps: Based on the real-time PTZ information of the camera device and the PTZ information of the tags corresponding to several preset AR tags, determine several AR tags to be drawn in the real-time video frame. Based on the real-time PTZ information of the PTZ camera, the PTZ information of the AR tags to be drawn is converted into the coordinate information of the AR tags to be drawn in the real-time video frame. Based on the coordinate information, the plurality of AR tags to be drawn are drawn on a preset drawing interface to generate an AR tag interface; The AR tag interface is overlaid with the video frame data corresponding to the real-time video screen to obtain the video frame data to be displayed and sent to the front-end device so that the front-end device can display the real-time interactive video screen according to the video frame data to be displayed. When the user operation coordinate information in the real-time interactive video frame is received from the front-end device, the interactive object and the user operation response result corresponding to the interactive object are determined according to the user operation coordinate information. The AR tag interface is updated based on the interactive object and the user operation response result. Updated video frame data is generated based on the updated AR tag interface and sent to the front-end device so that the front-end device can display the updated real-time interactive video screen based on the updated video frame data. The method further includes the following steps before drawing the plurality of AR tags to be drawn on a preset drawing interface based on the coordinate information and generating the AR tag interface: Based on the resolution of the real-time video frame, the size of the preset initial drawing interface and the size of several AR tags to be drawn are scaled to obtain the preset drawing interface and several AR tags to be drawn after scaling. Then, the step of drawing the plurality of AR tags to be drawn on a preset drawing interface according to the coordinate information to generate an AR tag interface specifically includes: Based on the coordinate information, the AR tags to be drawn, scaled down to the specified size, are drawn on the drawing interface to generate the AR tag interface. 2.The AR interaction method of real-time video of claim 1, wherein, The step of overlaying the AR tag interface with the video frame data corresponding to the real-time video screen to obtain the video frame data to be displayed and sending it to the front-end device, so that the front-end device can display the real-time interactive video screen according to the video frame data to be displayed, specifically includes the following steps: The AR tag interface is superimposed on the video frame data corresponding to the real-time video screen to obtain the first video frame data, and the first video frame data is encoded to obtain the video frame data to be displayed. The video frame data to be displayed is sent to the front-end device, so that the front-end device can decode the video frame data to obtain the target video frame data, and display the real-time interactive video screen according to the target video frame data. 3.The AR interaction method of real-time video of claim 1, wherein, The step of updating the AR tag interface based on the interaction object and the user operation response result specifically includes the following steps: Based on the interaction object and the user operation response result, perform the update operation corresponding to the user operation response result on the AR tag interface to generate the updated AR tag interface; The update operation includes modifying the parameters of any AR tag in the AR tag interface, adding an AR tag in the AR tag interface, deleting any AR tag in the AR tag interface, and marking any AR tag in the AR tag interface. 4.The AR interaction method of real-time video of claim 1, wherein, The step of generating updated video frame data based on the updated AR tag interface and sending it to the front-end device, so that the front-end device can display the updated real-time interactive video screen based on the updated video frame data, specifically includes the following steps: The updated AR tag interface is overlaid with the video frame data corresponding to the real-time video screen to obtain the second video frame data, and the second video frame data is encoded to obtain the updated video frame data. The updated video frame data is sent to the front-end device so that the front-end device can decode the updated video frame data and display the updated real-time interactive video screen based on the decoded updated video frame data. 5.The AR interaction method of real-time video of claim 3, wherein, The AR tag interface includes an operation menu set within a preset area; The operation menu includes several button icons, each button icon corresponding to any AR tag interface update operation command. 6.The AR interaction method of real-time video of claim 1, wherein, After updating the AR tag interface based on the interaction object and the user operation response result, the method further includes the following steps: The update operation completion information is generated and fed back to the front-end device.

7. An AR interaction device for real-time video, characterized by, include: The AR tag confirmation module is used to determine several AR tags to be drawn in the real-time video frame based on the real-time PTZ information of the camera device and the PTZ information of the tags corresponding to several preset AR tags. The coordinate transformation module is used to convert the PTZ information of the AR tags to be drawn into the coordinate information of the AR tags to be drawn in the real-time video frame based on the real-time PTZ information of the PTZ. The AR tag drawing module is used to draw the plurality of AR tags to be drawn on a preset drawing interface according to the coordinate information, thereby generating an AR tag interface. The AR tag interface overlay module is used to overlay the AR tag interface with the video frame data corresponding to the real-time video screen to obtain the video frame data to be displayed and send it to the front-end device so that the front-end device can display the real-time interactive video screen according to the video frame data to be displayed. The user operation response module is used to determine the interactive object and the corresponding user operation response result based on the user operation coordinate information when it receives user operation coordinate information sent by the front-end device in the real-time interactive video screen. The AR tag interface update module is used to update the AR tag interface according to the interactive object and the user operation response result, generate updated video frame data according to the updated AR tag interface and send it to the front-end device, so that the front-end device can display the updated real-time interactive video screen according to the updated video frame data. The device further includes a size scaling module, used for: Based on the resolution of the real-time video frame, the size of the preset initial drawing interface and the size of several AR tags to be drawn are scaled to obtain the preset drawing interface and several AR tags to be drawn after scaling. Then, the AR tag drawing module is used to draw the plurality of AR tags to be drawn on a preset drawing interface according to the coordinate information, thereby generating an AR tag interface, specifically including: Based on the coordinate information, the AR tags to be drawn, scaled down to the specified size, are drawn on the drawing interface to generate the AR tag interface.

8. A terminal device, comprising: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the AR interaction method for real-time video as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the AR interaction method for real-time video as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Dynamic label adding method, equipment and system

    CN110267087A

  • Streaming media service method and system for fusing augmented reality label in video

    CN110365995A