Note following display method and device and electronic equipment

Abstract

The invention discloses an annotation following display method and device and electronic equipment, and belongs to the technical field of information sharing and annotation. The method comprises the following steps: receiving auxiliary stream data and annotation data sent by a server; determining a target area matched with the annotation area image in a target frame image corresponding to the auxiliary stream data; determining a display area of the annotation attribute information according to position information of the target area in the target frame image; and displaying the annotation attribute information in the display area. Through the mode, the annotation can be displayed along with the annotated content in the target image frame, when the annotated content in the target frame image corresponding to the auxiliary stream data is shifted, zoomed and the like, the problem that the annotation area and the annotated content are misplaced can be avoided, and the annotation accuracy is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of information sharing and annotation, and more particularly to a method, device and electronic device for displaying annotations. Background Art

[0002] An important application scenario in video conferencing systems is the sharing of auxiliary streams. Auxiliary streams can usually be understood as PPT, Word or other audio and video materials used for terminal presentations. In a remote conference, the sender sends auxiliary stream data to the server, which forwards the auxiliary stream data to the receiving end of the conference to achieve remote conference sharing. After the video conference sharing is successful, both the sender and the receiver can annotate the content in the auxiliary stream data.

[0003] However, after the annotation is completed, the display position of the annotation is fixed. When the auxiliary stream content changes such as displacement or scaling, the annotation area and the annotated content are easily misaligned, resulting in insufficient annotation accuracy. Summary of the invention

[0004] The embodiments of the present application provide a method, device and electronic device for displaying annotations following the user, so as to at least solve the problem of misalignment between the annotation area and the annotated content in a video conference.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an annotation follow-up display method, comprising:

[0007] Receiving auxiliary stream data and annotation data sent by a server, wherein the annotation data includes an annotation area image and annotation attribute information;

[0008] Determine a target region matching the annotated region image in a target frame image corresponding to the auxiliary stream data;

[0009] Determining a display area of ​​the annotation attribute information according to position information of the target area in the target frame image;

[0010] The annotation attribute information is displayed in the display area.

[0011] In a possible implementation manner, before receiving the auxiliary stream data and the annotation data sent by the server, the method further includes:

[0012] Displaying an annotation interface on the auxiliary stream interface corresponding to the auxiliary stream data, wherein the annotation interface has a transparent color;

[0013] In response to a user's touch operation on the annotation interface, obtaining annotation attribute information corresponding to the touch operation;

[0014] Determine the annotation area image according to the frame image and annotation attribute information currently displayed on the auxiliary stream interface;

[0015] The annotation area image and annotation attribute information are sent to the server.

[0016] Wherein, after acquiring the annotation attribute information corresponding to the touch operation, the method further includes:

[0017] The annotation attribute information is displayed on the annotation interface.

[0018] In a possible implementation manner, determining the target area matching the annotated area image in the target frame image corresponding to the auxiliary stream data includes:

[0019] Dividing the target frame image into a plurality of regions;

[0020] extracting first feature information of each of the plurality of regions and second feature information of the annotated region image;

[0021] According to the feature matching result of the first feature information and the second feature information, a target area among the multiple areas that matches the annotated area image is determined.

[0022] In a possible implementation, displaying the annotation attribute information in the display area includes:

[0023] Determining a scaling ratio of the target frame image according to the first feature information and the second feature information;

[0024] The annotation attribute information is updated according to the zoom ratio, and the updated annotation attribute information is displayed in the display area.

[0025] Among them, the annotation area image includes the annotation area and the background content area; the annotation area is the circumscribed rectangular area of ​​the graphic represented by the annotation attribute information; the background content area includes the annotation area, and the area of ​​the background content area is larger than the area of ​​the annotation area.

[0026] In a second aspect, an embodiment of the present application provides an annotation following display device, comprising:

[0027] A data receiving module, used for receiving auxiliary stream data and annotation data sent by a server, wherein the annotation data includes an annotation area image and annotation attribute information;

[0028] An image processing module, used for determining a target area in a target frame image corresponding to the auxiliary stream data that matches the annotated area image;

[0029] Determining a display area of ​​the annotation attribute information according to position information of the target area in the target frame image;

[0030] The annotation display module is used to display the annotation attribute information in the display area.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, characterized in that it includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect above are implemented.

[0032] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0033] The annotation following display method provided in the embodiment of the present application receives auxiliary stream data and annotation data sent by a server; determines a target area in a target frame image corresponding to the auxiliary stream data that matches the annotation area image; determines a display area for annotation attribute information based on the position information of the target area in the target frame image; and displays the annotation attribute information in the display area. In this way, remote conference sharing is realized by receiving auxiliary stream data sent by a server, and during conference sharing, the display area for annotation attribute information is determined based on the target area in a target frame image corresponding to the auxiliary stream data that matches the annotation area image, and the annotation attribute information is displayed in the display area, so that the annotation can be displayed following the annotated content in the target image frame. When the annotated content in the target frame image corresponding to the auxiliary stream data undergoes displacement, scaling, and other changes, the problem of misalignment between the annotation area and the annotated content can be avoided, thereby improving the accuracy of the annotation.

[0034] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] Figure 1 An architectural diagram of a video conferencing system is shown;

[0037] Figure 2 A schematic diagram showing a flow chart of an annotation following display method provided in an embodiment of the present application is shown;

[0038] Figure 3 A schematic diagram of the structure of a client provided in an embodiment of the present application is shown;

[0039] Figure 4 Another schematic diagram of the process of the annotation following display method provided by an embodiment of the present application is shown;

[0040] Figure 5a-5b A user interface display diagram of the annotation following process provided by an embodiment of the present application is shown;

[0041] Figure 6 One of the structural schematic diagrams of the annotation following display device provided in an embodiment of the present application is shown;

[0042] Figure 7 The second structural schematic diagram of the annotation following display device provided in the embodiment of the present application is shown;

[0043] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0045] Figure 1 An architecture diagram of a video conferencing system is shown. As shown in the figure, the transmitter sends auxiliary stream data to the server, and the server forwards the auxiliary stream data to all receiving terminals participating in the remote video conference to realize remote conference sharing. During the remote conference sharing process, both the transmitter and the receiving terminal can annotate the content in the auxiliary stream data. For example, after the transmitter completes the annotation, it sends the annotation data to the server, and the server forwards the annotation data to receiving terminals 1, 2, and 3 to synchronize the annotation data and realize multi-terminal interactive annotation of the video conference. However, after the annotation is completed, the display position of the annotation is relatively fixed. When the auxiliary stream content is displaced, scaled, etc., the annotation cannot change with the changes in the auxiliary stream content, resulting in errors in the annotation area and the annotated content, and the problem of insufficient annotation accuracy.

[0046] Based on this, an embodiment of the present application provides an annotation follow display method, which is used to solve the pain point of misalignment between annotations and annotated content when the auxiliary stream content changes, and improve the accuracy of annotations.

[0047] Figure 2The flowchart of the annotation follow-up display method provided in the embodiment of the present application is shown. The execution subject of the method may be a terminal device, wherein the terminal device may be a device such as a personal computer, or a mobile terminal device such as a mobile phone or a tablet computer, and the terminal device may be a terminal device used by a user. In the embodiment of the present application, the execution subject is taken as an example of a terminal device, as shown in the figure, the annotation follow-up display method 200 includes the following steps:

[0048] S201: receiving auxiliary stream data and annotation data sent by a server, wherein the annotation data includes an annotation area image and annotation attribute information;

[0049] S202: Determine a target region matching the annotated region image in the target frame image corresponding to the auxiliary stream data;

[0050] S203: determining a display area of ​​the annotation attribute information according to position information of the target area in the target frame image;

[0051] S204: Displaying the annotation attribute information in the display area.

[0052] In a specific implementation, the terminal device realizes remote conference sharing by receiving auxiliary stream data sent by the server. During the conference sharing process, the terminal device also receives annotation data sent by the server, and the annotation data includes an annotation area image and annotation attribute information. Among them, the annotation area image can be understood as an image containing the annotation area, and the annotation area image can include the annotation area and the background content area. The background content area includes the annotated area, and the area of ​​the background content area is larger than the area of ​​the annotation area. In practical applications, in order to ensure the reliability of subsequent image matching results, the background content area and the annotation area usually have the same geometric center point, and the area of ​​the background content area is set to twice the area of ​​the annotation area. The annotation area can be the circumscribed rectangular area of ​​the figure represented by the annotation attribute information. For example, if the annotated figure is an ellipse, the annotation area is the circumscribed rectangular area of ​​the ellipse circle; if the annotated figure is a wavy line, the annotation area is the circumscribed rectangular area of ​​the wavy line. The annotation attribute information may include attribute information such as brush color, line thickness, and annotation figure.

[0053] Furthermore, the terminal device obtains the target frame image corresponding to the auxiliary stream data, determines the target area in the target frame image that matches the annotation area image, determines the display area of ​​the annotation attribute information based on the position information of the target area in the target frame image, and displays the annotation attribute information in the display area.

[0054] Through the above steps, since the annotation data received by the terminal device includes the annotation area image, the display area of ​​the annotation attribute information is determined according to the target area in the target frame image that matches the annotation area image. In this way, the annotation can be displayed following the annotated content in the target image frame. When the annotated content in the target frame image corresponding to the auxiliary stream data undergoes displacement, scaling, and other changes, the problem of misalignment between the annotation area and the annotated content can be avoided, thereby improving the accuracy of the annotation.

[0055] In a possible implementation manner, in the above step S201, before receiving the auxiliary stream data and annotation data sent by the server, the following step is further included:

[0056] Displaying an annotation interface on the auxiliary stream interface corresponding to the auxiliary stream data, wherein the annotation interface has a transparent color;

[0057] In response to a user's touch operation on the annotation interface, obtaining annotation attribute information corresponding to the touch operation;

[0058] Determine the annotation area image according to the frame image and annotation attribute information currently displayed on the auxiliary stream interface;

[0059] The annotation area image and annotation attribute information are sent to the server.

[0060] Among them, the auxiliary stream interface is a display window of the auxiliary stream data on the terminal device, and the display window can be used to display the auxiliary stream information of PPT, Word or other audio and video materials.

[0061] In a specific implementation, a transparent annotation interface is displayed on the auxiliary stream interface corresponding to the auxiliary stream data, and the user can draw annotation graphics on the annotation interface. In response to the user's touch operation on the annotation interface, the brush color, line thickness and other information selected by the user are obtained, and the annotation trajectory and shape are calculated based on the drawing algorithms such as Bresenham, and the annotation attribute information is determined. According to the frame image and annotation attribute information currently displayed on the auxiliary stream interface, the annotation area image is determined, and then the annotation area image and annotation attribute information are sent to the server, and the annotation data is forwarded to other terminal devices through the server to synchronize the annotation data in remote conference sharing.

[0062] Furthermore, after obtaining the annotation attribute information corresponding to the touch operation, it also includes:

[0063] Display annotation attribute information on the annotation interface.

[0064] In a specific implementation, after the annotation attribute information is acquired, the annotation attribute information is processed such as annotation drawing, rendering, caching and synchronization, and displayed on the annotation interface of the terminal device.

[0065] In a possible implementation manner, in the above step S202, determining the target region matching the annotated region image in the target frame image corresponding to the auxiliary stream data includes:

[0066] Dividing the target frame image into a plurality of regions;

[0067] extracting first feature information of each of the plurality of regions and second feature information of the annotated region image;

[0068] According to the feature matching result of the first feature information and the second feature information, a target area among the multiple areas that matches the annotated area image is determined.

[0069] In a specific implementation, the target frame image corresponding to the auxiliary stream data can be divided into multiple regions, and the first feature information of each region in the multiple regions and the second feature information of the annotated region image are feature matched. When the feature matching degree is higher than the preset matching degree threshold, it is determined that the region matches the annotated region image, and the image matching is successful. In practical applications, the image matching method supports scaling matching with a scaling ratio of 0.5 to 2, that is, the scaling ratio between the region image in the current frame image and the background image of the annotated region shall not be less than 0.5 and shall not be greater than 2, otherwise the matching will definitely fail.

[0070] The annotation attribute information is displayed in the display area, including:

[0071] Determining a scaling ratio of the target frame image according to the first feature information and the second feature information;

[0072] The annotation attribute information is updated according to the zoom ratio, and the updated annotation attribute information is displayed in the display area.

[0073] In a specific implementation, the scaling ratio of the target frame image can be determined according to the first characteristic information and the second characteristic information, and then the annotation attribute information can be updated according to the scaling ratio, for example, the color resolution, line thickness, shape size and other parameters of the annotation can be updated, and then the updated annotation attribute information can be displayed in the display area. In this way, when the auxiliary stream content is scaled, the annotation attribute information can be adjusted accordingly to match the annotation with the annotated content, ensuring the accuracy of the annotation.

[0074] The following description is made with reference to a specific example. The annotation following display method provided in the embodiment of the present application is applied to a client, such as Figure 3 As shown, the client includes: a conference control module, a network module, a video module, an annotation module and an image processing module, wherein:

[0075] Conference control module, used for user login, joining video conference, user identity verification and other operations;

[0076] The network module is used to establish a communication connection between the client and the server, and realize the interaction of auxiliary stream data and annotation data between the client and the server;

[0077] The video module is used to encode and decode the auxiliary stream data and render the frame image;

[0078] Annotation module, used for drawing annotations, caching and synchronization of annotation data;

[0079] The image processing module is used to process the frame image, combine it with the annotation data, and output the relevant processed data. Specifically, it may include the following functions:

[0080] (1) Calculate and segment the annotation area in the current frame image based on the annotation data;

[0081] (2) Cache the image data of the annotation area and background content area;

[0082] (3) Match the target frame image with the annotated area;

[0083] (4) Output the successfully matched target area and the corresponding annotation attribute information.

[0084] Further, such as Figure 4 As shown, in combination with the above-mentioned client, the annotation following display method may include the following steps:

[0085] Step 401: The user starts sharing auxiliary streams, and the video module performs image encoding, decoding, rendering and synchronization operations, and sends the current frame image data to the image processing module for caching;

[0086] Step 402: the annotation module superimposes a transparent annotation interface onto the auxiliary stream interface;

[0087] Step 403: After the user starts to annotate, he draws on the annotation interface, and the annotation module calculates the annotation trajectory and shape based on the Bresenham and other drawing algorithms to determine the annotation attribute information;

[0088] Step 404: after the annotation calculation is completed, the annotation attribute information is annotated, rendered, cached and synchronized, and displayed on the annotation interface;

[0089] Step 405: the annotation module sends the annotation data to the image processing module, and the image processing module obtains the current frame image from the video module and performs image processing;

[0090] Step 406: The image processing module combines the current frame image and the annotation data to calculate and segment the annotation area image in the current frame image;

[0091] Step 407: the image processing module caches the annotated area image and sends it to the server for annotation data synchronization;

[0092] Step 408: When the user starts sharing the auxiliary stream, a thread is started inside the image processing module, and the image matching operation is executed cyclically until the auxiliary stream sharing is stopped, at which time the thread ends;

[0093] Step 409: The image processing module matches the cached target frame image with the annotated area image frame by frame based on image recognition technology. When part of the current frame image of the auxiliary stream matches the background image of the annotated area, the match is considered successful;

[0094] Step 410: After the image processing module successfully matches the image data, the image processing module outputs the successfully matched target area in the current frame image and the corresponding annotation data to the annotation module;

[0095] Step 411: The annotation module determines the display area of ​​the annotation attribute information based on the successfully matched target area, updates the annotation attribute information according to the zoom ratio of the target frame image, moves the annotation attribute information to the display area, and re-draws, renders, caches and synchronizes the information to display it on the annotation interface.

[0096] Through the above steps, if Figure 5a-5b As shown in the figure, when the annotated area in the auxiliary stream image changes from Figure 5a The position shown changes to Figure 5b The comment box can move along with the annotated area to the position shown in Figure 5b The position of the annotation and the annotated area remains unchanged.

[0097] The embodiment of the present application provides an annotation follow display method, which receives auxiliary stream data and annotation data sent by a server; determines a target area in a target frame image corresponding to the auxiliary stream data that matches the annotation area image; determines a display area for the annotation attribute information based on the position information of the target area in the target frame image; and displays the annotation attribute information in the display area. Through the above method, the annotation can be displayed following the annotated content in the target image frame. When the annotated content in the target frame image corresponding to the auxiliary stream data undergoes displacement, scaling, and other changes, the problem of misalignment between the annotation area and the annotated content can be avoided, thereby improving the accuracy of the annotation.

[0098] Figure 6 One of the structural schematic diagrams of the annotation following display device provided in the embodiment of the present application is shown. As shown in the figure, the annotation following display device 600 includes:

[0099] The data receiving module 610 is used to receive auxiliary stream data and annotation data sent by the server, wherein the annotation data includes an annotation area image and annotation attribute information;

[0100] An image processing module 620 is used to determine a target area in a target frame image corresponding to the auxiliary stream data that matches the annotated area image;

[0101] Determining a display area of ​​the annotation attribute information according to position information of the target area in the target frame image;

[0102] The annotation display module 630 is used to display the annotation attribute information in the display area.

[0103] Among them, the annotation area image includes the annotation area and the background content area; the annotation area is the circumscribed rectangular area of ​​the figure represented by the annotation attribute information; the background content area includes the annotation area, and the area of ​​the background content area is larger than the area of ​​the annotation area.

[0104] In a possible implementation manner, when the image processing module 620 is used to determine the target area matching the annotated area image in the target frame image corresponding to the auxiliary stream data, it is specifically used to:

[0105] Dividing the target frame image into a plurality of regions;

[0106] extracting first feature information of each of the plurality of regions and second feature information of the annotated region image;

[0107] According to the feature matching result of the first feature information and the second feature information, a target area among the multiple areas that matches the annotated area image is determined.

[0108] When the annotation display module 630 is used to display the annotation attribute information in the display area, it is specifically used to:

[0109] Determining a scaling ratio of the target frame image according to the first feature information and the second feature information;

[0110] The annotation attribute information is updated according to the zoom ratio, and the updated annotation attribute information is displayed in the display area.

[0111] In a possible implementation, Figure 7 As shown, the annotation following display device 600 further includes:

[0112] An annotation module 640, used to display an annotation interface on the auxiliary stream interface corresponding to the auxiliary stream data, wherein the annotation interface has a transparent color;

[0113] In response to a user's touch operation on the annotation interface, obtaining annotation attribute information corresponding to the touch operation;

[0114] Determine the annotation area image according to the frame image and annotation attribute information currently displayed on the auxiliary stream interface;

[0115] The annotation area image and annotation attribute information are sent to the server.

[0116] In a possible implementation manner, after being used to obtain the annotation attribute information corresponding to the touch operation, the annotation module 640 is further used to:

[0117] The annotation attribute information is displayed on the annotation interface.

[0118] The embodiment of the present application provides an annotation follow display device, including a data receiving module, an image processing module and an annotation display module; the auxiliary stream data and annotation data sent by the server are received by the data receiving module, and the annotation data includes an annotation area image and annotation attribute information; the image processing module determines a target area that matches the annotation area image in a target frame image corresponding to the auxiliary stream data; according to the position information of the target area in the target frame image, the display area of ​​the annotation attribute information is determined; and then, the annotation display module displays the annotation attribute information in the display area. Through the above-mentioned method, the annotation can be displayed following the annotated content in the target image frame. When the annotated content in the target frame image corresponding to the auxiliary stream data undergoes displacement, scaling and other changes, the problem of misalignment between the annotation area and the annotated content can be avoided, thereby improving the accuracy of the annotation.

[0119] Figure 8 A schematic diagram of the hardware structure of an electronic device that implements the embodiments of the present application is shown. Referring to the figure, at the hardware level, the electronic device includes a processor, and optionally, an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the computer device may also include hardware required for other services.

[0120] The processor, the network interface and the memory may be interconnected through an internal bus, which may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0121] The memory stores the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0122] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a device for locating the target user at the logical level. The processor executes the program stored in the memory and specifically performs: Figure 2 or Figure 4 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.

[0123] The above application Figure 2 or Figure 4The method disclosed in the illustrated embodiment may be implemented in a processor or by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0124] The computer device can also execute the methods described in the foregoing method embodiments, and realize the functions and beneficial effects of the methods described in the foregoing method embodiments, which will not be repeated here.

[0125] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0126] The embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable medium stores one or more programs, and when the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes Figure 2 or Figure 4 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.

[0127] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0128] Furthermore, an embodiment of the present application also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the following process is implemented: Figure 2 or Figure 4 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.

[0129] In short, the above is only a preferred embodiment of the present application and does not limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0130] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0131] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0132] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0133] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A method for displaying annotations, characterized in that: include: Receiving auxiliary stream data and annotation data sent by a server, wherein the annotation data includes an annotation area image and annotation attribute information; Determine a target region matching the annotated region image in a target frame image corresponding to the auxiliary stream data; Determining a display area of ​​the annotation attribute information according to position information of the target area in the target frame image; The annotation attribute information is displayed in the display area.

2. The method according to claim 1, characterized in that: Before receiving the auxiliary stream data and annotation data sent by the server, the method further includes: Displaying an annotation interface on the auxiliary stream interface corresponding to the auxiliary stream data, wherein the annotation interface has a transparent color; In response to a user's touch operation on the annotation interface, obtaining annotation attribute information corresponding to the touch operation; Determine the annotation area image according to the frame image and annotation attribute information currently displayed on the auxiliary stream interface; The annotation area image and annotation attribute information are sent to the server.

3. The method according to claim 2, characterized in that After acquiring the annotation attribute information corresponding to the touch operation, the method further includes: The annotation attribute information is displayed on the annotation interface.

4. The method according to claim 1, characterized in that The determining of a target region matching the annotated region image in a target frame image corresponding to the auxiliary stream data comprises: Dividing the target frame image into a plurality of regions; extracting first feature information of each of the plurality of regions and second feature information of the annotated region image; According to the feature matching result of the first feature information and the second feature information, a target area among the multiple areas that matches the annotated area image is determined.

5. The method according to claim 4, characterized in that Displaying the annotation attribute information in the display area includes: Determining a scaling ratio of the target frame image according to the first feature information and the second feature information; The annotation attribute information is updated according to the zoom ratio, and the updated annotation attribute information is displayed in the display area.

6. The method according to any one of claims 1 to 5, characterized in that: The annotation area image includes an annotation area and a background content area; the annotation area is a circumscribed rectangular area of ​​a graphic represented by annotation attribute information; the background content area includes the annotation area, and the area of ​​the background content area is larger than that of the annotation area.

7. An annotation following display device, characterized in that: include: A data receiving module, used for receiving auxiliary stream data and annotation data sent by a server, wherein the annotation data includes an annotation area image and annotation attribute information; An image processing module, used for determining a target area in a target frame image corresponding to the auxiliary stream data that matches the annotated area image; Determining a display area of ​​the annotation attribute information according to position information of the target area in the target frame image; The annotation display module is used to display the annotation attribute information in the display area.

8. The device according to claim 7, characterized in that Also includes: An annotation module, used for displaying an annotation interface on the auxiliary stream interface corresponding to the auxiliary stream data, wherein the annotation interface has a transparent color; In response to a user's touch operation on the annotation interface, obtaining annotation attribute information corresponding to the touch operation; Determine the annotation area image according to the frame image and annotation attribute information currently displayed on the auxiliary stream interface; The annotation area image and annotation attribute information are sent to the server.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images