Method, device and computer storage medium for encapsulating labeling information

Abstract

This application discloses a method, apparatus, and computer storage medium for encapsulating annotation information, belonging to the field of multimedia technology. In this method, descriptive information carrying annotation information for video frames is extended into the source video file. Thus, after annotating a video frame, the annotation information can be written into this descriptive information, thereby encapsulating the annotation information within the source video file. Since the annotation information is encapsulated within the source video file, as long as the source video file is properly stored, the annotation information encapsulated within it will not be lost. Considering the inherent durability of source video files, the technical solution provided by this application can prevent the loss of annotation information.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular to a method, apparatus, and computer storage medium for encapsulating annotation information. Background Technology

[0002] In the field of multimedia technology, video annotation is often required. For example, when training an algorithm model, video frames need to be annotated beforehand, and then the annotated video frames are used as training samples to train the algorithm model. After annotating the video frames, how the annotation information is packaged will affect the subsequent use of the annotation information to some extent.

[0003] In related technologies, after decoding the bitstream from the source video file to obtain video frames, the video frames are labeled, and then the decoded video frames are encapsulated into media data packets, with the labeling information of each video frame encapsulated into label data packets. The media data packets and label data packets are then written into a condensed video file defined based on a proprietary protocol, thereby achieving the encapsulation of labeling information. However, this method of encapsulating labeling information is prone to loss due to the susceptibility of condensed files. Summary of the Invention

[0004] This application provides a method, apparatus, and computer storage medium for encapsulating annotation information, which can reduce the risk of annotation information loss. The technical solution is as follows:

[0005] On the one hand, a method for encapsulating annotation information is provided, the method comprising:

[0006] Obtain a source video file, the source video file including the bitstream of video frames and descriptive information for the bitstream, the descriptive information being used to carry annotation information for the video frames;

[0007] Display the video frames based on the bitstream;

[0008] In response to the annotation instruction for the video frame, the annotation information corresponding to the annotation instruction is obtained, and the annotation information corresponding to the annotation instruction is written into the description information to obtain the updated source video file.

[0009] Optionally, the description information of the source video file does not carry historical annotation information of the video frames;

[0010] The step of obtaining the annotation information corresponding to the annotation instruction and writing the annotation information corresponding to the annotation instruction into the description information includes:

[0011] Add annotation information corresponding to the annotation instruction to the description information.

[0012] Optionally, the description information of the source video file carries historical annotation information of the video frames;

[0013] The step of obtaining the annotation information corresponding to the annotation instruction and writing the annotation information corresponding to the annotation instruction into the description information includes:

[0014] While retaining the historical annotation information in the description information, annotation information corresponding to the annotation instruction is added to the description information.

[0015] Optionally, after displaying the video frame based on the bitstream, the method further includes:

[0016] Display the historical annotations of the video frames based on the historical annotation information;

[0017] The annotation instructions include one or more of the following: modification instructions for the historical annotations, deletion instructions for the historical annotations, and annotation addition instructions.

[0018] Optionally, after obtaining the source video file, the method further includes:

[0019] Determine the starting offset position of the description information in the source video file;

[0020] The step of writing the annotation information corresponding to the annotation instruction into the description information includes:

[0021] Based on the starting offset position, the annotation information corresponding to the annotation instruction is written into the description information to obtain the updated description information;

[0022] Based on the length of the updated description information, update the starting offset position of other data recorded in the source video file.

[0023] Optionally, the source video file is a source video file encapsulated in Program Stream (PS) format. The source video file includes a PS packet, which contains the bitstreams of multiple video frames and a Program Stream Map (PSM). The PSM is used to store description information of the bitstreams of the multiple videos.

[0024] Optionally, the PSM includes description information fields corresponding to the plurality of video frames respectively. The description information fields include one or more descriptor fields, and each descriptor field is used to store annotation information corresponding to one annotation operation.

[0025] Optionally, the description information field includes a labeling indicator field, which indicates that the content following the labeling indicator field is the labeling information of the corresponding video.

[0026] Optionally, the description information field includes a label type field, which indicates the label type corresponding to the label information of the corresponding video frame.

[0027] On the other hand, an apparatus for encapsulating annotation information is provided, the apparatus comprising:

[0028] An acquisition module is used to acquire a source video file, the source video file including the bitstream of video frames and descriptive information for the bitstream, the descriptive information being used to carry annotation information of the video frames;

[0029] A display module is used to display the video frames based on the bitstream;

[0030] The encapsulation module is used to respond to the annotation instruction for the video frame, obtain the annotation information corresponding to the annotation instruction, write the annotation information corresponding to the annotation instruction into the description information, and obtain the updated source video file.

[0031] Optionally, the description information of the source video file does not carry historical annotation information of the video frames;

[0032] The encapsulation module is used for:

[0033] Add annotation information corresponding to the annotation instruction to the description information.

[0034] Optionally, the description information of the source video file carries historical annotation information of the video frames;

[0035] The encapsulation module is used for:

[0036] While retaining the historical annotation information in the description information, annotation information corresponding to the annotation instruction is added to the description information.

[0037] Optionally, the display module is further configured to:

[0038] After displaying the video frame based on the bitstream, display the historical annotations of the video frame based on the historical annotation information;

[0039] The annotation instructions include one or more of the following: modification instructions for the historical annotations, deletion instructions for the historical annotations, and annotation addition instructions.

[0040] Optionally, the acquisition module is further configured to:

[0041] Determine the starting offset position of the description information in the source video file;

[0042] The encapsulation module is used for:

[0043] Based on the starting offset position, the annotation information corresponding to the annotation instruction is written into the description information to obtain the updated description information;

[0044] Based on the length of the updated description information, update the starting offset position of other data recorded in the source video file.

[0045] Optionally, the source video file is a source video file encapsulated in Program Stream (PS) format. The source video file includes a PS packet, which contains the bitstreams of multiple video frames and a Program Stream Map (PSM). The PSM is used to store description information of the bitstreams of the multiple videos.

[0046] Optionally, the PSM includes description information fields corresponding to the plurality of video frames respectively. The description information fields include one or more descriptor fields, and each descriptor field is used to store annotation information corresponding to one annotation operation.

[0047] Optionally, the description information field includes a labeling indicator field, which indicates that the content following the labeling indicator field is the labeling information of the corresponding video.

[0048] Optionally, the description information field includes a label type field, which indicates the label type corresponding to the label information of the corresponding video frame.

[0049] On the other hand, an apparatus for encapsulating annotation information is provided, the apparatus comprising:

[0050] processor;

[0051] Memory used to store processor-executable instructions;

[0052] The processor is configured to perform any step of the method for providing the encapsulation annotation information described above.

[0053] On the other hand, a computer-readable storage medium is provided, on which instructions are stored, which, when executed by a processor, implement any step of the method for encapsulating annotation information provided above.

[0054] On the other hand, a computer program product containing instructions is provided that, when run on a computer, causes the computer to perform any step of the method for encapsulating annotation information provided above.

[0055] In this embodiment, descriptive information carrying annotation information for video frames is extended into the source video file. Thus, after annotating a video frame, the annotation information can be written into this descriptive information, thereby encapsulating the annotation information within the source video file. This technical solution can achieve at least the following technical effects:

[0056] (1) Since the annotation information is encapsulated in the source video file, as long as the source video file is properly stored, the annotation information encapsulated in the source video file will not be lost. Considering the fact that source video files are not easily lost, the technical solution provided by the embodiments of this application can avoid the loss of annotation information.

[0057] (2) The source video file is extended with descriptive information used to carry annotation information for video frames. This allows the descriptive information to be extended based on the standard syntax corresponding to the source video file. This achieves non-intrusive encapsulation of annotation information while ensuring the standard syntax of the source video file's encapsulation layer and normal playback of video frames. It eliminates the need to separately encapsulate annotation information based on proprietary protocol definitions, thus improving the convenience of the technical solution provided in this application.

[0058] (3) Since the annotation information is encapsulated in the source video file, there is no need to encapsulate a separate annotation data packet independent of the source video file, which can also reduce the space occupied by the encapsulated annotation information. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram of the architecture of an annotation system provided in an embodiment of this application.

[0061] Figure 2 This is a schematic diagram of the data format of a source video file encapsulated in PS format, provided in an embodiment of this application.

[0062] Figure 3 This is a schematic diagram of the syntax rules of a PSM provided in an embodiment of this application.

[0063] Figure 4 This is a flowchart of a method for encapsulating annotation information provided in an embodiment of this application.

[0064] Figure 5 This is a schematic diagram of a device for encapsulating labeling information provided in an embodiment of this application.

[0065] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application.

[0066] Figure 7 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0068] Before providing a detailed explanation of the embodiments of this application, the application scenarios of the embodiments of this application will be explained first.

[0069] Currently, AI (artificial intelligence) technology based on multimedia data is developing rapidly, and AI applications based on this technology are also increasing. Numerous AI applications are quickly integrating into our work and lives, such as facial recognition attendance, self-driving cars, and unmanned supermarkets. Video data, as the most prevalent form of multimedia data, will serve as the main source of data for future AI technologies. Furthermore, current AI applications rely heavily on algorithm model training, which in turn requires labeled training samples. These labeled training samples include image data and annotation information. Therefore, how to annotate source video files, and how to encapsulate, save, and parse the annotated information, are prerequisites for the successful application of AI technology.

[0070] The method for encapsulating annotation information provided in this application embodiment is applied to the scenario of training the model of the above-mentioned AI technology. Optionally, the method for encapsulating annotation information provided in this application embodiment can also be applied to other scenarios that require annotation of video frames. Examples will not be provided here.

[0071] To implement the method for encapsulating annotation information provided in the embodiments of this application, an annotation system is provided. The technical solutions of the embodiments of this application can be implemented based on this annotation system. The annotation system provided in the embodiments of this application will be explained below.

[0072] Figure 1 This is a schematic diagram of the architecture of an annotation system provided in an embodiment of this application. For example... Figure 1 As shown, the annotation system 100 includes a parsing module 101, an annotation module 102, and an encapsulation module 103.

[0073] The parsing module 101 is used to perform operations such as decapsulation and decoding on the source video file to obtain each video frame and / or historical annotation information, and to send each video frame and / or historical annotation information to the annotation module 102.

[0074] The annotation module 102 is used to display each video frame on the annotation interface, and also to display historical annotations based on historical annotation information on the annotation interface. Users can perform manual annotations on the annotation interface. After the user performs manual annotation, the annotation module can respond to the annotation command triggered by the user based on the annotation interface and obtain the annotation information corresponding to the current annotation command. Then, the annotation module 102 sends the annotation information corresponding to the current annotation command to the encapsulation module 103.

[0075] The encapsulation module 103 is used to encapsulate the annotation information into the source video file, thereby enabling the updating of the source video file.

[0076] The annotation information involved in the embodiments of this application can be customized by each algorithm. That is, the embodiments of this application do not limit the specific content of the annotation information. When applying the embodiments of this application, the annotation information can be defined based on the algorithm to implement the method provided by the embodiments of this application. In one possible implementation, the annotation information may include information such as the annotator, annotation time, and annotation sequence number, so as to facilitate more detailed annotation trajectory tracing. The annotation trajectory indicates the relevant information of each annotation for a video frame.

[0077] It should be noted that, Figure 1 The annotation system shown can be centrally deployed in a terminal or a server. Optionally, the various modules in the annotation system can also be distributed and deployed on different devices. This application embodiment does not limit this.

[0078] also, Figure 1 Each module in the annotation system is a software module, and the naming of each module is based on its function. When applying the embodiments of this application, different names can be used based on requirements. For example, the parsing module can be named the first module, the annotation module the second module, and the encapsulation module the third module, etc. The embodiments of this application do not limit the naming of the modules as described above.

[0079] In this embodiment, to avoid the loss of annotation information, fields in the source video file can be expanded to enable the source video file to carry annotation information of video frames. Currently, there are various types of source video file encapsulation formats, such as PS (program stream) format, MP4 (media portable 4) format, and formats such as AVI (audio video interleaved) and RMVB (real-media variable bitrate). For any type of encapsulation format, fields capable of carrying annotation information of video frames can be added to the source video file.

[0080] In one possible implementation, to avoid modifying the original format of the source video file, existing fields in the source video file can be extended, thereby non-intrusively encapsulating the annotation information within the source video file. Optionally, new fields can be added to the source video file to carry the annotation information; this application embodiment does not limit this, as long as the extended source video file can carry the annotation information of the video frames.

[0081] The following example uses a source video file packaged in PS format to illustrate how to include video frame annotation information in the source video file.

[0082] When the source video file is encapsulated in PS format, the source video file includes a PS packet, which contains the bitstreams of multiple video frames and a PSM (program stream map). The PSM stores descriptive information about the multiple video bitstreams. In other words, for a PS-encapsulated source video file, the PSM is expanded to allow it to carry annotation information for the video frames.

[0083] Figure 2 This is a schematic diagram illustrating the data format of a source video file encapsulated in PS format, as provided in an embodiment of this application. For example... Figure 2 As shown, the source video file includes multiple PS packets. Each PS packet includes a PSH (PS header), a PSM, and multiple PES (packetized elementary streams). The PSH carries information such as system time. The PSM carries information describing the relationships between the various PES. The PES is the basic unit of the bitstream, carrying the bitstream of each video frame.

[0084] Furthermore, each PES carries a bitstream corresponding to one I-frame and several B / P frames. For example, the first PES carries the bitstream of an I-frame, and subsequent PES carry the bitstreams of several B / P frames in sequence. Here, an I-frame is an internally coded frame (also called a keyframe), a P-frame is a forward-predicted frame (also called a forward reference frame), and a B-frame is a bidirectional interpolated frame (also called a bidirectional reference frame). Simply put, an I-frame is a complete picture, while P-frames and B-frames record changes relative to the I-frame. Without an I-frame, P-frames and B-frames cannot be decoded.

[0085] for Figure 2 In one possible implementation of the source video file shown, the PSM includes descriptive information fields corresponding to multiple video frames. These descriptive information fields include one or more descriptor fields, each of which is expanded to store annotation information corresponding to a single annotation operation. This encapsulates the annotation information within existing fields of the source video file.

[0086] Figure 3 This is a schematic diagram of the syntax rules for PSM provided in an embodiment of this application. The syntax rules are the PSM syntax rules defined in the RFC-13818 document (request for comments-13818, which is a series of numbered documents). Figure 3 The solid-line box shows the descriptive information fields mentioned above. The first loop in the solid-line box (corresponding to...) Figure 3 The first line of code in the solid box indicates the description information field for each video frame. Therefore, the PSM includes a description information field corresponding to each video frame. The second loop field in the dashed box (corresponding to...) Figure 3 The second-to-last line of code within the dashed box corresponds to the descriptor fields of a video frame. Each descriptor field stores the annotation information corresponding to a single annotation operation. Thus, any annotation performed on each video frame in PSM can be recorded, thereby recording the annotation trajectory. The annotation trajectory indicates the relevant information for each annotation performed on a video frame.

[0087] It should be noted that, Figure 3 The interpretations of other codes in the PSM syntax rules are exactly the same as those of the same codes defined in the annotation protocol, so they will not be repeated here.

[0088] Alternatively, the description information field can be extended to include a labeling indicator field, which indicates that the content following the labeling indicator field contains the labeling information for the corresponding video. That is, in another possible implementation, the description information field may also include a labeling indicator field. This allows for quick identification of which fields are used to indicate labeling information during subsequent parsing of the PS packet.

[0089] like Figure 3 As shown, it can be Figure 3 The "stream_type" field within the solid box is expanded to indicate the aforementioned annotation indication field. It should be noted that the "stream_type" field in the PS annotation protocol indicates the stream type, with a value range of 0x80-0xFF; different values ​​can represent different stream types. In this embodiment, the reserved value 0xBD can be selected as the annotation indication field. That is, when the "stream_type" field is 0xBD, it indicates that the content following the "stream_type" field is the annotation information for the corresponding video.

[0090] Figure 3 This example uses the existing "stream_type" field in PSM to illustrate how to implement the aforementioned annotation indicator field. Optionally, a new field can also be added to PSM to implement the aforementioned annotation indicator field, which will not be explained in detail here.

[0091] Alternatively, a label type field can be added to the description information field. This label type field indicates the label type corresponding to the label information of the relevant video frame. That is, in one possible implementation, the description information field includes a label type field. This allows for filtering of label information based on the label type field during subsequent parsing of the PS packet.

[0092] Thus, the annotation type can be animals, people, buildings, roads, etc. This annotation type field allows for quick identification of the type of annotation recorded in the video frame's annotation information, facilitating further related operations by administrators.

[0093] like Figure 3 As shown, it can be Figure 3The "elementary_stream_id" field within the solid box is expanded to indicate the aforementioned annotation type field. Different values ​​for the "elementary_stream_id" field correspond to different annotation types. For example, a value of 0xBD indicates one annotation type, while a value of 0xBF indicates another.

[0094] In addition, Figure 3 In the PSM shown, the descriptor field also includes a descriptor_tag field and a descriptor_length field. Figure 3 (Not shown in the image). The `descriptor_tag` field ranges from 0x40 to 0xFF, representing the frame number (starting from 0x40) in the video frame sequence. 0x40 represents frame 0, 0x41 represents frame 1, and so on. The `descriptor_length` field within the `descriptor` field indicates the length of the subsequent annotation information.

[0095] in addition, Figure 3 The `elementary_stream_info_length` field represents the total length of all descriptor fields in a single video frame. Furthermore, Figure 3 The `elementary_stream_map_length` field represents the total length of all descriptor fields for all video frames. Further explanation of these two fields is unnecessary.

[0096] for Figure 3 The extended syntax rules shown add a descriptor field for each additional annotation, and update the value of the elementary_stream_map_length field accordingly.

[0097] It should be noted that for source video files encapsulated in PS format, Figure 3 This involves recording all annotation information for each video frame individually. Alternatively, instead of recording information frame by frame, all annotation information for all video frames can be directly included in the source video file. This simply requires adding the corresponding video frame identifier and timestamp to each annotation.

[0098] also, Figure 3This example uses a PS-formatted source video file to illustrate how to expand a source video file to include descriptive information carrying annotation information, thus enabling the source video file to carry annotation information. When applying the embodiments of this application, other methods can also be used to expand the source video file to carry annotation information. For example, fields can be added to the source video file so that these new fields can carry annotation information. All possible expansion methods will not be illustrated here.

[0099] Based on the above Figure 1 The annotation system shown in this application provides a method for encapsulating annotation information. This method will be explained in detail below. Furthermore, based on... Figure 1 As can be seen from the annotation system shown, the subject executing this method is not limited. For ease of subsequent explanation, the following embodiment uses the example of the annotation system being centrally deployed on a terminal.

[0100] Figure 4 This is a flowchart illustrating a method for encapsulating annotation information provided in an embodiment of this application. For example... Figure 4 As shown, the method includes the following steps.

[0101] Step 401: The terminal obtains the source video file, which includes the bitstream of the video frames and the descriptive information for the bitstream. The descriptive information is used to carry the annotation information of the video frames.

[0102] Based on the above-mentioned content regarding the expansion of the source video file, it can be seen that the expanded source video file in this embodiment includes descriptive information, which is used to carry the annotation information of the video frames. Therefore, when annotating the video, the source video file can be directly obtained so that the source video file can be updated subsequently based on the current annotation process.

[0103] Step 402: The terminal displays the video frame based on the bitstream.

[0104] It should be noted that the bitstream typically includes multiple video frames. Therefore, in step 402, the terminal can display each video frame based on the bitstream. Specifically, displaying each video frame based on the bitstream means decoding the bitstream to obtain each video frame, and then displaying each video frame. The decoding process will not be described in detail in this embodiment.

[0105] In addition, before displaying video frames based on the bitstream, the source video file needs to be parsed to obtain the bitstream and description information.

[0106] For example, in Figure 3In the extended method shown, if a PES structure is identified during the parsing process, the bitstream of each video frame is identified. Then, based on step 402, decapsulation and decoding are performed to obtain video frames in YUV (an image encoding method). Finally, image encoding is performed based on the YUV format video frames to achieve image display. This image can be in JPEG (joint photographic experts group) format.

[0107] In addition, Figure 3 In the extended method shown, if the PSM structure is identified during the parsing process, it can also be based on... Figure 3 The syntax rules are used to parse the historical annotation information of the video frame. This historical annotation information refers to the annotations made to the video frame before the current time. In this case, after displaying the video frame based on the bitstream in step 4020, the historical annotations of the video frame can also be displayed based on the historical annotation information. Historical annotations refer to the annotations made to the video frame before the current time.

[0108] It should be noted that, since a user may have added a label in one annotation process and deleted a label in another before the current time, the aforementioned historical annotations refer to the annotation results of video frames annotated before the current time.

[0109] Furthermore, for any video frame in the bitstream, the description information corresponding to that video frame may or may not contain historical annotation information. Therefore, in the above implementation of displaying historical annotations, after parsing the source video file, for any video frame, if the video frame corresponds to historical annotation information, both the video frame and the historical annotation are displayed simultaneously, so that subsequent annotation operations can be performed based on the historical annotations, thereby updating the historical annotations. If the video frame does not have corresponding historical annotation information, only the video frame itself is displayed.

[0110] For example, for Figure 3 The extended method shown searches the description field for historical annotation information for each video frame based on the frame number (cycled in GOPs, starting from 0x40, where 0x40 represents frame 0, 0x41 represents frame 1, and so on). If no historical annotation information is found for any given video frame, it indicates that the subsequent annotation process is the first annotation, and only a JPEG image is sent. Figure 1 The annotation module shown is for initial annotation later. If found, historical annotation information exists for that video frame; in this case, the historical annotation information along with the JPEG image is sent to... Figure 1 The annotation module shown is for subsequent iterative annotation.

[0111] The above embodiment illustrates the use of displaying historical annotations based on existing historical annotation information. Optionally, historical annotations may not be displayed when they exist. Whether or not historical annotations are displayed depends on whether the algorithm needs to update them; therefore, this embodiment does not limit this.

[0112] In addition, Figure 3 In the extended method shown, if the PSM structure is identified during the parsing process, the starting offset position of the PSM structure in the entire PS packet can also be parsed. The relevant role of the parsed starting offset position will be explained in subsequent embodiments, and will not be elaborated here.

[0113] Step 403: The terminal responds to the annotation instruction for the video frame, obtains the annotation information corresponding to the annotation instruction, writes the annotation information corresponding to the annotation instruction into the description information, and obtains the updated source video file.

[0114] As can be seen from step 402, the terminal may only display video frames, or it may display both video frames and historical annotations. Therefore, in this embodiment, the annotation instructions include one or more of the following: modification instructions for historical annotations, deletion instructions for historical annotations, and annotation addition instructions.

[0115] Specifically, when the terminal only displays video frames, the annotation command is limited to "Add Annotation". When the terminal displays both video frames and historical annotations, if the user needs to modify historical annotations, the annotation command is for modifying historical annotations. If the user needs to delete historical annotations, the annotation command is for deleting historical annotations. If the user needs to add other annotations to the video frames, the annotation command is for adding annotations.

[0116] In step 403, the terminal responds to the annotation instruction for the video frame and obtains the annotation information corresponding to the annotation instruction in the following ways: the terminal displays an annotation confirmation control on the display interface. When the user triggers the annotation confirmation control, the terminal can detect the annotation instruction, thereby obtaining the various operations triggered by the user during the annotation process, and then generating annotation information.

[0117] In one possible implementation, the annotation information carried in the source video file consists of annotation information generated during all annotation processes, facilitating subsequent tracing of the annotation trajectory. Therefore, after obtaining the annotation information, if the description information of the source video file does not carry historical annotation information for video frames, the process of writing the annotation information corresponding to the annotation instruction into the description information can be: adding the annotation information corresponding to the annotation instruction to the description information. If the description information of the source video file carries historical annotation information for video frames, the process of writing the annotation information corresponding to the annotation instruction into the description information can be: retaining the historical annotation information in the description information while continuing to add the annotation information corresponding to the annotation instruction to the description information.

[0118] based on Figure 1 As can be seen from the system shown, this step can be performed by... Figure 1 This is achieved through the annotation module. In this implementation, if it's the first time annotating, the annotation module only displays that video frame, and the user annotates using conventional methods. For example, the user draws a rectangle on the video frame and labels the content of the rectangle as "person." After completing the annotation and clicking "OK," the annotation information for that video frame is converted into... Figure 3 The required syntax structure is sent to the packaging module so that the packaging module can update the source video file. This annotation information includes the annotator, annotation time, frame number, clockwise coordinates of the four corners of the rectangle, and the "person" category, etc.

[0119] If this is not the first time annotation is performed, the annotation module will draw all historical annotations on the video frames based on historical annotation information. Users can choose to add new annotations, delete some historical annotations, or modify some historical annotations. After completing the annotation and clicking the "OK" control, all annotation information generated during this annotation process will be converted into... Figure 3 The required syntax structure is sent to the packaging module so that the packaging module can update the source video file.

[0120] It's important to note that in the above implementation, users can choose to annotate or not annotate each video frame during the annotation process, and the annotation information is updated each time an annotation is performed. Therefore, the annotation information carried in the source video file is a cumulative record of each annotation operation; that is, the more annotations performed, the longer the annotation information becomes, and it is not the final annotation result. For example, if a user's first annotation is "add information A," and the second annotation is "delete information A," then there will be two annotation records, one for adding information A and one for deleting information A, used to trace the annotation trajectory.

[0121] The above implementation is illustrated using the example where the annotation information carried in the source video file represents all annotation information generated during the annotation process. Optionally, the annotation information carried in the source video file can also be only the annotation information generated in the most recent annotation process at the current time. In this case, after obtaining the annotation information, it is only necessary to replace the annotation information in the description information with the obtained annotation information. In this implementation, the annotation information carried in the source video file only represents the final annotation result and cannot achieve the tracing of the annotation trajectory. This will not be explained in detail here.

[0122] Furthermore, since the source video file also includes other data packets, to avoid other data packets becoming inaccessible due to updates to the description information, in this embodiment, after writing the annotation information corresponding to the annotation instruction into the description information, it is also necessary to update the starting offset positions of other data based on the changes in the starting offset positions of the description information, so that other data can be accessed subsequently based on the updated starting offset positions.

[0123] Therefore, in one possible implementation, after obtaining the source video file in step 401, the terminal can also determine the starting offset position of the description information in the source video file. In this case, the process of writing the annotation information corresponding to the annotation instruction into the description information in step 403 is as follows: based on the starting offset position, the annotation information corresponding to the annotation instruction is written into the description information to obtain the updated description information; based on the length of the updated description information, the starting offset positions of other data recorded in the source video file are updated.

[0124] For example, for Figure 3 The syntax rules shown indicate that when the parsing module encounters a PSM structure while parsing the source video file, it first records the starting offset position of the PSM structure. After updating the PSM structure based on the annotation information from this annotation process, it first updates the starting offset position of that PSM structure, and then updates the starting offset positions of other data such as PSH and PES based on the updated starting offset position of the PSM structure. Thus, after annotating each video frame, the actual update is to the content of the corresponding PSM structure for that video frame; other data in the PS packet remains unchanged, only the starting offset position is extended.

[0125] It should be noted that steps 401 to 403 above can be applied to source video files encapsulated in any format. Because PS-encapsulated source video files are highly fault-tolerant, scalable, and support both file and streaming data formats, steps 401 to 403 will be further explained below using PS-encapsulated source video files to clarify the technical solution of this application.

[0126] based on Figure 1The system shown includes the following steps in its specific annotation scheme.

[0127] 1. The source video file encapsulated in PS format is sent to the parsing module. When the parsing module encounters a PSM structure, it records the starting offset position of the current PSM structure and the historical annotation information in the PSM structure.

[0128] 2. The parsing module performs decapsulation and decoding operations on the frame data, and sends the parsed complete YUV frame data and corresponding historical annotation information to the annotation module for annotation.

[0129] 3. The annotation module displays YUV frame data on the annotation interface for manual annotation. Users can choose to update (including appending and deleting) a historical annotation, or re-annotate the video frame. The returned annotation information is sent to the encapsulation module.

[0130] 4. The encapsulation module writes the annotation information into the PSM structure and updates the starting offset position of each data recorded in the source video file, thereby completing the annotation work of a video frame.

[0131] In summary, the technical solution for encapsulating annotation information provided in this application has the following characteristics and advantages.

[0132] 1. Based on the standard syntax of the source video file's encapsulation layer, an extension is made to achieve non-intrusive encapsulation of annotation information while ensuring the standard syntax of the encapsulation layer and normal playback of media data. Since the annotation information is encapsulated in the extended fields of the PSM structure, it has no impact on the syntactic standard of the source video file.

[0133] 2. Supports duplicate annotation of source video files. For duplicate annotation of any video frame, simply repeat steps 401 to 403 above.

[0134] 3. Supports the traceability of annotation trajectories. Since the annotation information for each annotation operation in each video frame can include the annotator, annotation time, annotation count, and annotation content, the annotation trajectory can also be traced.

[0135] 4. The annotation process is simple, requiring only the parsing module, annotation module, and encapsulation module.

[0136] 5. Adding annotation information to the source video file will not double the space consumption; theoretically, it will only increase the amount of annotation information data.

[0137] 6. As long as the source video file is properly stored, all historical annotation information based on that file will not be lost. Therefore, as long as the source video file is complete, the annotation information will never be lost, thus resolving the risk of loss associated with storing annotation information separately.

[0138] 7. Simple matching relationship between video frames and annotation information. Based on Figure 3 The syntax rules shown can achieve this technical effect.

[0139] 8. Supports updating annotation information, including appending and deleting. This technical effect can be achieved by updating based on historical annotations.

[0140] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this application, and the embodiments of this application will not be described in detail one by one.

[0141] Figure 5 This is a schematic diagram of a device for encapsulating labeling information provided in an embodiment of this application. This device can be implemented by software, hardware, or a combination of both. Figure 5 As shown, the device 500 for encapsulating labeling information includes the following modules.

[0142] The acquisition module 501 is used to acquire the source video file, which includes the bitstream of video frames and the descriptive information for the bitstream. The descriptive information is used to carry the annotation information of the video frames.

[0143] Display module 502 is used to display video frames based on the bitstream;

[0144] The encapsulation module 503 is used to respond to the annotation instructions for the video frame, obtain the annotation information corresponding to the annotation instructions, write the annotation information corresponding to the annotation instructions into the description information, and obtain the updated source video file.

[0145] Optionally, the description information of the source video file does not carry historical annotation information of the video frames;

[0146] The encapsulation module is used for:

[0147] Add annotation information corresponding to the annotation instructions to the description information.

[0148] Optionally, the description information of the source video file carries historical annotation information of the video frames;

[0149] The encapsulation module is used for:

[0150] While retaining the historical annotation information in the description information, continue to add annotation information corresponding to the annotation instructions to the description information.

[0151] Optionally, the display module is also used for:

[0152] After displaying video frames based on the bitstream, display the historical annotations of the video frames based on the historical annotation information.

[0153] The annotation instructions include one or more of the following: modification instructions for historical annotations, deletion instructions for historical annotations, and annotation addition instructions.

[0154] Optionally, the acquisition module is also used for:

[0155] Determine the starting offset position of the description information in the source video file;

[0156] The encapsulation module is used for:

[0157] Based on the starting offset position, the annotation information corresponding to the annotation instruction is written into the description information to obtain the updated description information;

[0158] Based on the length of the updated description information, update the starting offset position of other data recorded in the source video file.

[0159] Optionally, the source video file is a source video file encapsulated in the program stream PS format. The source video file includes a PS packet, which contains the bitstreams of multiple video frames and a program stream mapping PSM. The PSM is used to store descriptive information of the bitstreams of multiple videos.

[0160] Optionally, the PSM includes description information fields corresponding to multiple video frames respectively. The description information fields include one or more descriptor fields, and each descriptor field is used to store the annotation information corresponding to one annotation operation.

[0161] Optionally, the description information field includes a labeling indicator field, which indicates that the content following the labeling indicator field is the labeling information for the corresponding video.

[0162] Optionally, the description information field includes a label type field, which indicates the label type corresponding to the label information of the corresponding video frame.

[0163] In this embodiment, descriptive information carrying annotation information for video frames is extended into the source video file. Thus, after annotating a video frame, the annotation information can be written into this descriptive information, thereby encapsulating the annotation information within the source video file. This technical solution can achieve at least the following technical effects:

[0164] (1) Since the annotation information is encapsulated in the source video file, as long as the source video file is properly stored, the annotation information encapsulated in the source video file will not be lost. Considering the fact that source video files are not easily lost, the technical solution provided by the embodiments of this application can avoid the loss of annotation information.

[0165] (2) The source video file is extended with descriptive information used to carry annotation information for video frames. This allows the descriptive information to be extended based on the standard syntax corresponding to the source video file. This achieves non-intrusive encapsulation of annotation information while ensuring the standard syntax of the source video file's encapsulation layer and normal playback of video frames. It eliminates the need to separately encapsulate annotation information based on proprietary protocol definitions, thus improving the convenience of the technical solution provided in this application.

[0166] (3) Since the annotation information is encapsulated in the source video file, there is no need to encapsulate a separate annotation data packet independent of the source video file, which can also reduce the space occupied by the encapsulated annotation information.

[0167] It should be noted that the device for encapsulating labeling information provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device for encapsulating labeling information and the method for encapsulating labeling information provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0168] Figure 6 This is a schematic diagram of the structure of a terminal 600 provided in an embodiment of this application. Figure 1 Any module in the system shown can be accessed via Figure 6 The terminal 600 shown is used for implementation. This terminal 600 can be: a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 600 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0169] Typically, terminal 600 includes a processor 601 and a memory 602.

[0170] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0171] Memory 602 may include one or more computer-readable storage media, which may be non-transitory. Memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 602 is used to store at least one instruction, which is executed by processor 601 to implement the method of encapsulating annotation information provided in the method embodiments of this application.

[0172] In some embodiments, the terminal 600 may also optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608, and a power supply 609.

[0173] Peripheral interface 603 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 601 and memory 602. In some embodiments, processor 601, memory 602 and peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 601, memory 602 and peripheral interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0174] The radio frequency (RF) circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 604 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 604 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0175] Display screen 605 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 601 for processing. In this case, display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 605, which serves as the front panel of terminal 600; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of terminal 600 or in a folded design; in still other embodiments, display screen 605 may be a flexible display screen, disposed on a curved or folded surface of terminal 600. Furthermore, display screen 605 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 605 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0176] The camera assembly 606 is used to acquire images or videos. Optionally, the camera assembly 606 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0177] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 601 for processing, or input to the radio frequency circuit 604 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 600. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0178] The positioning component 608 is used to determine the current geographic location of the terminal 600 in order to enable navigation or LBS (Location Based Service). The positioning component 608 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, Russia's Granas system, or the European Union's Galileo system.

[0179] Power supply 609 is used to supply power to the various components in terminal 600. Power supply 609 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 609 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0180] In some embodiments, the terminal 600 further includes one or more sensors 610. The one or more sensors 610 include, but are not limited to: an accelerometer 611, a gyroscope 612, a pressure sensor 613, a fingerprint sensor 614, an optical sensor 615, and a proximity sensor 616.

[0181] Accelerometer 611 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 600. For example, accelerometer 611 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 601 can control display screen 605 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 611. Accelerometer 611 can also be used for games or for acquiring user motion data.

[0182] The gyroscope sensor 612 can detect the orientation and rotation angle of the terminal 600. The gyroscope sensor 612, in conjunction with the accelerometer sensor 611, can collect 3D motion data from the user on the terminal 600. Based on the data collected by the gyroscope sensor 612, the processor 601 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0183] The pressure sensor 613 can be disposed on the side bezel of the terminal 600 and / or on the lower layer of the display screen 605. When the pressure sensor 613 is disposed on the side bezel of the terminal 600, it can detect the user's grip signal on the terminal 600, and the processor 601 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 613. When the pressure sensor 613 is disposed on the lower layer of the display screen 605, the processor 601 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 605. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0184] The fingerprint sensor 614 is used to collect a user's fingerprint. The processor 601 identifies the user based on the fingerprint collected by the fingerprint sensor 614, or vice versa. When the user's identity is identified as trusted, the processor 601 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 614 can be located on the front, back, or side of the terminal 600. When the terminal 600 has physical buttons or a manufacturer's logo, the fingerprint sensor 614 can be integrated with the physical buttons or manufacturer's logo.

[0185] An optical sensor 615 is used to collect ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 based on the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is decreased. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 based on the ambient light intensity collected by the optical sensor 615.

[0186] The proximity sensor 616, also known as a distance sensor, is typically mounted on the front panel of the terminal 600. The proximity sensor 616 is used to detect the distance between the user and the front of the terminal 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from a screen-on state to a screen-off state; when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from a screen-off state to a screen-on state.

[0187] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on terminal 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0188] This application also provides a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of a terminal, enables the terminal to execute the method for encapsulating annotation information provided in the above embodiment.

[0189] This application also provides a computer program product containing instructions that, when run on a terminal, cause the terminal to execute the method for encapsulating annotation information provided in the above embodiments.

[0190] Figure 7 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Figure 1 Any module in the system shown can be accessed via Figure 7 This is implemented using the server shown. This server can be a server within a backend server cluster. Specifically:

[0191] Server 700 includes a central processing unit (CPU) 701, a system memory 704 including random access memory (RAM) 702 and read-only memory (ROM) 703, and a system bus 705 connecting the system memory 704 and the CPU 701. Server 700 also includes a basic input / output system (I / O system) 706 that facilitates the transfer of information between various devices within the computer, and a mass storage device 707 for storing the operating system 713, application programs 714, and other program modules 715.

[0192] The basic input / output system 706 includes a display 708 for displaying information and an input device 709 for user input, such as a mouse or keyboard. Both the display 708 and the input device 709 are connected to the central processing unit 701 via an input / output controller 710 connected to the system bus 705. The basic input / output system 706 may also include the input / output controller 710 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 710 also provides output to a display screen, printer, or other types of output devices.

[0193] Mass storage device 707 is connected to central processing unit 701 via a mass storage controller (not shown) connected to system bus 705. Mass storage device 707 and its associated computer-readable media provide non-volatile storage for server 700. That is, mass storage device 707 may include computer-readable media (not shown) such as hard disk or CD-ROM drive.

[0194] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 704 and mass storage device 707 described above can be collectively referred to as memory.

[0195] According to various embodiments of this application, server 700 can also be connected to a remote computer on a network, such as the Internet. That is, server 700 can be connected to network 712 via network interface unit 711 connected to system bus 705, or it can use network interface unit 711 to connect to other types of networks or remote computer systems (not shown).

[0196] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU. The one or more programs include a method for performing the encapsulation annotation information provided in the embodiments of this application.

[0197] This application also provides a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of a server, enables the server to execute the method for encapsulating annotation information provided in the above embodiments.

[0198] This application also provides a computer program product containing instructions that, when run on a server, cause the server to execute the method for encapsulating annotation information provided in the above embodiments.

[0199] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0200] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A method of encapsulating annotation information, characterized by, The method comprises: obtaining a source video file, the source video file comprising a code stream of video frames and description information for the code stream, the description information being used to carry annotation information of the video frames, the source video file being a source video file packaged in a program stream (PS) format, the source video file comprising a PS packet, the PS packet comprising a code stream of a plurality of video frames and a program stream map (PSM), the PSM being used to store description information of the code stream of the plurality of video frames, the PSM comprising description information fields corresponding to the plurality of video frames respectively, each description information field comprising one or more descriptor fields, each descriptor field being used to store annotation information corresponding to one annotation operation, the description information of the source video file carrying historical annotation information of the video frames; displaying the video frames based on the code stream and displaying historical annotations of the video frames based on the historical annotation information; in response to an annotation instruction for the video frames, obtaining annotation information corresponding to the annotation instruction, and writing the annotation information corresponding to the annotation instruction into the description information to obtain an updated source video file; wherein the obtaining of the annotation information corresponding to the annotation instruction and the writing of the annotation information corresponding to the annotation instruction into the description information comprise: continuously adding the annotation information corresponding to the annotation instruction into the description information while retaining the historical annotation information in the description information; wherein the annotation instruction comprises one or more of a modification instruction for the historical annotation, a deletion instruction for the historical annotation, and an annotation addition instruction.

2. The method of claim 1, wherein, The description information of the source video file does not carry the historical annotation information of the video frames. The obtaining of the annotation information corresponding to the annotation instruction and the writing of the annotation information corresponding to the annotation instruction into the description information comprise: adding the annotation information corresponding to the annotation instruction into the description information.

3. The method of claim 1, wherein, After the obtaining of the source video file, the method further comprises: determining a starting offset position of the description information in the source video file; the writing of the annotation information corresponding to the annotation instruction into the description information comprises: based on the starting offset position, writing the annotation information corresponding to the annotation instruction into the description information to obtain updated description information; based on a length of the updated description information, updating starting offset positions of other data recorded in the source video file.

4. The method of claim 1, wherein, The description information field further comprises an annotation indication field, the annotation indication field indicating that content after the annotation indication field is annotation information of a corresponding video.

5. The method of claim 1, wherein, The description information field further comprises an annotation type field, the annotation type field indicating an annotation type corresponding to annotation information of a corresponding video frame.

6. An apparatus for encapsulating annotation information, the apparatus comprising: The apparatus comprises: An acquisition module is configured to acquire a source video file, the source video file including a code stream of video frames and description information for the code stream, the description information being used to carry annotation information of the video frames, the source video file being a source video file in a program stream (PS) format, the source video file including a PS packet, the PS packet including the code stream of a plurality of video frames and a program stream map (PSM), the PSM being used to store description information of the code stream of the plurality of video frames, the PSM including description information fields corresponding to the plurality of video frames respectively, each description information field including one or more descriptor fields, each descriptor field being used to store annotation information corresponding to one annotation operation, and the description information of the source video file carrying historical annotation information of the video frames; A display module is configured to display the video frames based on the code stream and display historical annotations of the video frames based on the historical annotation information. An encapsulation module is configured to, in response to an annotation instruction for the video frames, acquire annotation information corresponding to the annotation instruction, write the annotation information corresponding to the annotation instruction into the description information, and obtain an updated source video file. The acquisition of the annotation information corresponding to the annotation instruction and the writing of the annotation information corresponding to the annotation instruction into the description information include: in a case where the historical annotation information in the description information is retained, the annotation information corresponding to the annotation instruction is continuously added into the description information. The annotation instruction includes one or more of a modification instruction for the historical annotation, a deletion instruction for the historical annotation, and an annotation addition instruction.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores instructions, and the instructions are executed by the processor to implement the steps of the method in any one of claims 1 to 5.

Images