A video annotation method, apparatus, electronic device, storage medium, and product.

Abstract

This application provides a video annotation method, apparatus, electronic device, storage medium, and product, relating to the field of video processing technology. The method includes: acquiring a video to be annotated and an identification result obtained by recognizing the video to be annotated; combining the video to be annotated and the corresponding identification result, using a multimodal large model and prompt words, generating attribute description text for each object in the video to be annotated and scene description text for the video to be annotated, obtaining a description text to be corrected; acquiring a correction result obtained by correcting the description text to be corrected, as the final annotation result of the video to be annotated; acquiring new prompt words obtained by correcting the prompt words, and using the difference between the final annotation result of the video to be annotated and the description text to be corrected, adjusting the parameters of the multimodal large model to obtain a new multimodal large model. This enables fast and accurate video annotation.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video annotation method, apparatus, electronic device, storage medium, and product. Background Technology

[0002] With the development of video processing technology, video data is increasingly widely used in various scenarios. When utilizing video data, annotation is often necessary. For example, annotation can provide training data support for machine learning and artificial intelligence. In autonomous driving scenarios, annotating targets such as vehicles can improve the system's environmental perception capabilities; in the security field, annotating abnormal behavior helps achieve real-time monitoring. Annotation can also enable efficient video retrieval and intelligent processing. Therefore, how to quickly and accurately perform video annotation has become an urgent problem to be solved. Summary of the Invention

[0003] The purpose of this application is to provide a video annotation method, apparatus, electronic device, storage medium, and product to quickly and accurately achieve video annotation. The specific technical solution is as follows:

[0004] A first aspect of this application provides a video annotation method, the method comprising:

[0005] The process involves acquiring the current video to be labeled and the recognition results obtained by identifying the current video to be labeled; wherein, the recognition results include: the scene of the current video to be labeled, the position of the same object in different video frames in the current video to be labeled, and the behavior of each object;

[0006] By combining the current video to be labeled and the corresponding recognition results, and using the current multimodal large model and prompt words, the attribute description text of each object in the current video to be labeled and the scene description text of the current video to be labeled are generated, thus obtaining the current description text to be corrected.

[0007] Obtain the correction result obtained by correcting the current description text to be corrected, and use it as the final annotation result of the current video to be annotated;

[0008] The system obtains new prompt words by correcting the current prompt words, and adjusts the parameters of the current multimodal large model by using the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected, thus obtaining a new multimodal large model. The new prompt words are determined based on the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected.

[0009] Optionally, by combining the current video to be labeled and the corresponding recognition results, and utilizing the current multimodal large model and prompt words, attribute description text for each object in the current video to be labeled and scene description text for the current video to be labeled are generated to obtain the current description text to be corrected, including:

[0010] The correction result obtained by manually correcting the recognition result corresponding to the current video to be labeled is used as the current labeling result to be utilized.

[0011] Input the current prompt words, the current annotation results to be utilized, and the current video to be annotated into the current multimodal large model to generate attribute description text for each object in the current video to be annotated and scene description text for the current video to be annotated, thus obtaining the current description text to be corrected.

[0012] Optionally, before obtaining the correction result obtained by manually correcting the recognition result corresponding to the current video to be labeled, and using it as the current labeling result to be utilized, the method further includes:

[0013] The video frames in the current video to be labeled are stitched together in chronological order to obtain a stitched image;

[0014] The stitched image is displayed, and the recognition result corresponding to the current video to be labeled is shown in the stitched image, so that the user can correct the recognition result corresponding to the current video to be labeled based on the displayed content.

[0015] Optionally, before obtaining the current video to be labeled and the recognition result obtained by recognizing the current video to be labeled, the method further includes:

[0016] Acquire multiple videos to be processed;

[0017] For each video to be processed, identify the scene of the video, the position of the same object in different video frames, and the behavior of each object to obtain the recognition result corresponding to the video to be processed.

[0018] Based on the recognition results corresponding to each video to be processed, the videos to be processed are sorted according to the preset scene priority order and / or preset behavior priority order.

[0019] The process of acquiring the current video to be labeled and the recognition result obtained by recognizing the current video to be labeled includes:

[0020] Based on the obtained sorting results, select the next video to be processed that has not yet obtained the final annotation result as the current video to be annotated, and obtain the recognition result corresponding to the current video to be annotated.

[0021] Optionally, acquiring multiple videos to be processed includes:

[0022] Acquire multiple raw videos;

[0023] For each original video, keyframes are extracted to obtain the corresponding video to be processed.

[0024] A second aspect of this application also provides a video annotation apparatus, the apparatus comprising:

[0025] The unlabeled video acquisition module is used to acquire the current unlabeled video and the recognition results obtained by recognizing the current unlabeled video; wherein, the recognition results include: the scene of the current unlabeled video, the position of the same object in different video frames in the current unlabeled video, and the behavior of each object;

[0026] The description text generation module is used to combine the current video to be labeled and the corresponding recognition results, and use the current multimodal large model and prompt words to generate attribute description text for each object in the current video to be labeled and scene description text for the current video to be labeled, so as to obtain the current description text to be corrected.

[0027] The annotation result acquisition module is used to obtain the correction result obtained by correcting the current descriptive text to be corrected, and use it as the final annotation result of the current video to be annotated;

[0028] The data acquisition module is optimized to acquire new prompt words obtained by correcting the current prompt words, and to adjust the parameters of the current multimodal large model by using the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected, so as to obtain a new multimodal large model. The new prompt words are determined according to the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected.

[0029] Optionally, the descriptive text generation module includes:

[0030] The result correction submodule is used to obtain the correction result obtained by manually correcting the recognition result corresponding to the current video to be labeled, and use it as the current labeling result to be utilized.

[0031] The description generation submodule is used to input the current prompt words, the current annotation results to be utilized, and the current video to be annotated into the current multimodal large model, generate the attribute description text of each object in the current video to be annotated and the scene description text of the current video to be annotated, and obtain the current description text to be corrected.

[0032] Optionally, the device further includes:

[0033] The image stitching module is used to stitch the video frames in the current video to be labeled in chronological order before obtaining the correction result obtained by manually correcting the recognition result corresponding to the current video to be labeled, and using it as the current labeling result to be used;

[0034] The image display module is used to display the obtained stitched image and show the recognition result corresponding to the current video to be labeled in the obtained stitched image, so that the user can correct the recognition result corresponding to the current video to be labeled based on the displayed content.

[0035] Optionally, the device further includes:

[0036] The video acquisition module is used to acquire multiple videos to be processed before acquiring the current video to be labeled and the recognition result obtained by recognizing the current video to be labeled;

[0037] The video recognition module is used to identify the scene of each video to be processed, the position of the same object in different video frames, and the behavior of each object, so as to obtain the recognition result corresponding to the video to be processed.

[0038] The video sorting module is used to sort the videos to be processed according to the recognition results corresponding to each video, in a preset scene priority order and / or a preset behavior priority order.

[0039] The video to be labeled acquisition module is specifically used to select the next video to be processed that has not yet obtained a final labeling result according to the obtained sorting result, as the current video to be labeled, and to obtain the recognition result corresponding to the current video to be labeled.

[0040] Optionally, the video acquisition module is specifically used to acquire multiple original videos; for each original video, key frames are extracted from the original video to obtain the corresponding video to be processed.

[0041] A third aspect of this application also provides an electronic device, comprising:

[0042] Memory, used to store computer programs;

[0043] The processor, when executing a program stored in memory, implements any of the video annotation methods described above.

[0044] A fourth aspect of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the video annotations described above.

[0045] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the video annotation methods described above.

[0046] Beneficial effects of the embodiments in this application:

[0047] The video annotation method provided in this application can obtain the video to be annotated and the corresponding recognition results. The recognition results for the video to be annotated include the scene of the video, the position of the same object in different video frames, and the behavior of each object, which can help the multimodal large model understand the attributes of each object in the video and the scene content of the video. Accordingly, by combining the video to be annotated and the corresponding recognition results, the multimodal large model and prompt words can be used to generate attribute description text for each object in the video and scene description text for the video (i.e., the description text to be corrected). Furthermore, the correction result obtained by correcting the current description text to be corrected (i.e., the final annotation result of the video to be annotated) can be obtained, which can further improve the accuracy of the final annotation result.

[0048] The new prompts are obtained by correcting the current prompts based on the differences between the final annotation results of the video to be annotated and the descriptive text to be corrected. Consequently, using the new prompts, more accurate descriptive text can be generated subsequently. Furthermore, the differences between the final annotation results of the video to be annotated and the descriptive text to be corrected can be used to optimize the current multimodal model. Consequently, using the new multimodal model, more accurate descriptive text can be generated subsequently. Thus, while ensuring high accuracy in the annotation results of the current video to be annotated, the prompts and multimodal model can be further optimized to improve the accuracy of the generated descriptive text, further ensuring the accuracy of subsequent annotations. Moreover, annotation using the multimodal model requires minimal manual intervention, enabling fast and accurate video annotation.

[0049] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0051] Figure 1 This is a schematic diagram of the first flowchart of the video annotation method provided in the embodiments of this application;

[0052] Figure 2 This is a schematic diagram of a second process for the video annotation method provided in the embodiments of this application;

[0053] Figure 3 This is a schematic diagram of a third process for the video annotation method provided in the embodiments of this application;

[0054] Figure 4 This is a schematic diagram of the structure of a video annotation device provided in an embodiment of this application;

[0055] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0057] With the development of video processing technology, video data is increasingly widely used in various scenarios. When utilizing video data, annotation is often required. To achieve fast and accurate video annotation, this application provides a video annotation method that can be applied to electronic devices. The type of electronic device is not specifically limited, as long as it can implement the video annotation method provided in this application. For example, the electronic device can be a personal computer or a server. See also... Figure 1 , Figure 1 This is a schematic diagram of a first flowchart of a video annotation method provided in an embodiment of this application. The video annotation method may include:

[0058] Step S101: Obtain the current video to be labeled, and the recognition result obtained by recognizing the current video to be labeled;

[0059] The identification results include: the scene of the current video to be labeled, the position of the same object in different video frames in the current video to be labeled, and the behavior of each object;

[0060] Step S102: Combining the current video to be labeled and the corresponding recognition results, using the current multimodal large model and prompt words, generate attribute description text for each object in the current video to be labeled and scene description text for the current video to be labeled, and obtain the current description text to be corrected.

[0061] Step S103: Obtain the correction result obtained by correcting the current description text to be corrected, and use it as the final annotation result of the current video to be annotated;

[0062] Step S104: Obtain new prompt words obtained by correcting the current prompt words, and adjust the parameters of the current multimodal large model by using the difference between the final annotation result of the current video to be annotated and the current description text to be corrected, so as to obtain a new multimodal large model.

[0063] The new prompt words are determined based on the difference between the final annotation results of the current video to be annotated and the current descriptive text to be corrected.

[0064] Based on the above processing, the video to be labeled and its corresponding recognition results can be obtained. The recognition results for the video to be labeled include the scene of the video, the position of the same object in different video frames, and the behavior of each object. This helps the multimodal large model understand the attributes of each object in the video and the scene content of the video. Accordingly, by combining the video to be labeled and its corresponding recognition results, the multimodal large model and prompt words can be used to generate attribute description text for each object in the video and scene description text for the video (i.e., the description text to be corrected). Furthermore, the correction result obtained by correcting the current description text to be corrected (i.e., the final labeling result of the video to be labeled) can be obtained, which can further improve the accuracy of the final labeling result.

[0065] The new prompts are obtained by correcting the current prompts based on the differences between the final annotation results of the video to be annotated and the descriptive text to be corrected. Consequently, using the new prompts, more accurate descriptive text can be generated subsequently. Furthermore, the differences between the final annotation results of the video to be annotated and the descriptive text to be corrected can be used to optimize the current multimodal model. Consequently, using the new multimodal model, more accurate descriptive text can be generated subsequently. Thus, while ensuring high accuracy in the annotation results of the current video to be annotated, the prompts and multimodal model can be further optimized to improve the accuracy of the generated descriptive text, further ensuring the accuracy of subsequent annotations. Moreover, annotation using the multimodal model requires minimal manual intervention, enabling fast and accurate video annotation.

[0066] Regarding step S101, the video currently requiring annotation can be referred to as the current unannotated video. For example, the unannotated video can be captured by a video capture device set in a preset scene. The electronic device can pre-identify the video currently requiring annotation (i.e., the current unannotated video) to obtain the corresponding identification result. For example, scene recognition can be performed on the current unannotated video to obtain the scene of the current unannotated video. For instance, a pre-trained deep learning model or large language model for scene recognition can be used for scene recognition. The scene of the unannotated video can be: a hospital, a school, a park, or a forest.

[0067] Object detection can be performed on the current video to be labeled, obtaining the location of objects in each video frame. Furthermore, the locations of the same object in different video frames can be correlated to obtain the position of the same object in different video frames. For example, pre-trained deep learning models or large language models for object detection and correlation can be used. Objects in the video to be labeled can include people or vehicles. The location of an object in a video frame can be marked with a bounding box, and the bounding boxes for different objects can be displayed in different ways, such as using different colors for the bounding boxes of different objects. Taking a person as an example, an object detection model can be used for initial object localization, and then combined with a ReID (Person Re-identification) model to correlate the identity of the same object across frames, that is, to correlate the location of the same object in different video frames.

[0068] For each object, behavior recognition can be performed using video frames containing that object in the current video to be labeled, thus obtaining the object's behavior. For example, pre-trained deep learning models or large language models for behavior recognition can be used. Taking a person as an example, the object's behavior could be running, smoking, fighting, or eating. The ReID model can be used to associate and match the bounding boxes (also called person boxes) of each person, and the large language model can then be used to identify the person's behavior. In other words, behavior pre-screening can be performed using the large language model based on the person boxes matched by the ReID model.

[0069] In one implementation, the large language model used for scene recognition, the large language model used for behavior recognition, and the subsequent multimodal large model can be the same model. The multimodal large model can be a VLM (Visual Language Model).

[0070] When it is necessary to annotate the current video to be annotated, the current video to be annotated and the recognition results obtained by recognizing the current video to be annotated can be obtained.

[0071] Regarding step S102, the recognition results corresponding to the current video to be labeled may include: the scene of the current video to be labeled, the position of the same object in different video frames in the current video to be labeled, and the behavior of each object. Correspondingly, the recognition results corresponding to the current video to be labeled can help the multimodal large model understand the content of the video to be labeled, so as to accurately generate descriptive text. Furthermore, combining the current video to be labeled and the recognition results corresponding to the current video to be labeled, using the multimodal large model and prompt words (i.e., prompts), attribute description text for each object in the video to be labeled and scene description text for the video to be labeled (i.e., description text to be corrected) can be generated. That is, fine-grained attribute recognition can be performed on each object in the video to be labeled to obtain attribute description text for each object, and scene description text for the video to be labeled can be generated. Users can set the current prompt words according to their desired generation requirements. For example, the current prompt words could be "Please describe the scene content of the video to be labeled, and describe the attributes of each object in the video, such as the object's behavior and posture."

[0072] Regarding step S103, since the multimodal large model may not accurately understand the current expected generation requirement indicated by the current prompt word, and the generation results of the multimodal large model have a certain degree of randomness and diversity, the current description text to be corrected obtained by the multimodal large model may not meet the current expected generation requirement. To improve the accuracy of the description text, the user can correct the current description text to be corrected according to the current expected generation requirement indicated by the current prompt word, and obtain the correction result.

[0073] For example, the current prompt is "Please describe the scene content of the video to be labeled, and describe the attributes of each object in the video, such as the object's behavior and posture." The expected generation requirement is to generate a detailed scene description of the video to be labeled, including descriptions of each area in the scene, and descriptions of the behavior and posture of each object in the video. The posture description of each object needs to be specific to its local movements. However, the generated description text to be corrected might only describe part of the scene. For example, in a classroom scene, it might only describe the desk area and not the podium area. In this case, the corrected description text can add a description of the podium area. Or, the generated description text to be corrected might only describe the overall posture of each object and not its local movements. For example, if the object is a person, it might only describe the person as sitting or standing, without describing the person's hand and leg movements. In this case, the corrected description text can add descriptions of the person's hand and leg movements.

[0074] Correspondingly, the electronic device can obtain the correction result obtained by correcting the current descriptive text to be corrected, and use it as the final annotation result for the current video to be annotated. Since the final annotation result is obtained by correcting according to the expected generation requirements, the final annotation result obtained after correction is more in line with the expected generation requirements. In this way, the electronic device can obtain a more accurate final annotation result.

[0075] Regarding step S104, the user can use the difference between the final annotation result of the video to be annotated and the description text to be corrected to correct the current prompt word and obtain a new prompt word. For example, the current prompt word is "Please describe the scene content of the video to be annotated, and describe the attributes of each object in the video, such as the object's behavior and posture." The current expected generation requirement is: generate a detailed scene description of the video to be annotated, including descriptions of each area in the scene, and generate behavior and posture descriptions of each object in the video, with the posture description of each object needing to be specific to local actions. However, the generated description text to be corrected may only describe part of the scene. For example, in a classroom scene, it may only describe the desk area and not the podium area. In this case, the final annotation result of the corrected video to be annotated includes a description of the podium area. Correspondingly, the difference between the final annotation result of the video to be annotated and the description text to be corrected is the description of the podium area. Therefore, the user can add a statement to the current prompt word instructing the generation of a description of the podium area. For example, the new prompt could be "Please describe the scene content of the video to be labeled, specifically the different areas of the scene in the video, including but not limited to the desk area, the podium area, and the doorway; and describe the attributes of each object in the video, such as the object's behavior and posture."

[0076] Alternatively, the generated description text to be corrected might only describe the overall posture of each object, without describing its local movements. For example, if the object is a person, it might only describe the person as sitting or standing, without describing their hand and leg movements. In this case, the final annotation result of the corrected video will include descriptions of the person's hand and leg movements. Correspondingly, the difference between the final annotation result of the video and the description text to be corrected is the description of the person's hand and leg movements. Therefore, the user can add a statement to the current prompt to generate descriptions of the person's hand and leg movements. For example, the new prompt could be: "Please describe the scene content of the video to be annotated, and describe the attributes of each object in the video, such as the object's behavior and posture. The posture of each object includes, but is not limited to, the overall posture, and the object's local posture, such as the object's hand and leg movements."

[0077] Correspondingly, electronic devices can acquire new prompt words, and subsequently use these new prompt words to generate descriptive text that better meets the expected generation requirements, thereby improving the accuracy of the generated descriptive text.

[0078] Electronic devices can also utilize the differences between the final annotation results of the video to be annotated and the descriptive text to be corrected to adjust the parameters of the current multimodal large model, resulting in a new multimodal large model—that is, optimizing the current multimodal large model. The new multimodal large model can then learn to generate descriptive text that better meets the desired generation requirements. Consequently, using the new multimodal large model subsequently can generate descriptive text that better meets the desired generation requirements, further improving the accuracy of subsequent generated descriptive text.

[0079] In this way, while ensuring the accuracy of the current annotation results for the videos to be annotated, the prompt words and multimodal large model can be further optimized. Subsequently, the optimized prompt words and multimodal large model can be used to generate the final annotation results for new videos to be annotated, and the prompt words and multimodal large model can be further optimized using the annotation results of the new videos. This enables a multimodal annotation generation framework driven by the multimodal large model and a human-computer interactive multi-round correction and self-learning optimization process, that is, it enables a multi-round iterative optimization mechanism in collaboration between humans and machines. This continuously improves the accuracy of the descriptive text of the videos generated by the multimodal large model, further ensuring that the subsequent video annotation results that better meet the expected generation requirements are obtained, improving the accuracy of video annotation, and through annotation by the multimodal large model, video annotation can be achieved quickly and accurately without much human intervention.

[0080] In this application, the current description text to be corrected can be generated using a multimodal large model in the following two ways:

[0081] In one implementation, the current prompt word, the current video to be labeled, and the corresponding recognition result can be directly input into the current multimodal large model to generate attribute description text for each object in the current video to be labeled and scene description text for the current video to be labeled, thus obtaining the current description text to be corrected.

[0082] In another implementation, step S102 includes:

[0083] Step 1: Obtain the correction result obtained by manually correcting the recognition result corresponding to the current video to be labeled, and use it as the current labeling result to be utilized;

[0084] Step 2: Input the current prompt words, the current annotation results to be used, and the current video to be annotated into the current multimodal large model to generate the attribute description text of each object in the current video to be annotated and the scene description text of the current video to be annotated, and obtain the current description text to be corrected.

[0085] In this implementation, users can correct the recognition results corresponding to the current video to be labeled. For example, users can determine whether the scene of the current video to be labeled, the position of each object in each video frame, and the behavior of each object are accurate. If inaccurate, users can make adjustments, such as adjusting the position or size of the label boxes if there are offset object bounding boxes or incomplete objects in the video frames. Alternatively, users can determine whether there are unrecognized objects (i.e., missed detection objects). If so, users can supplement them, such as adding label boxes corresponding to the missed detection objects in the video frames. In other words, users can perform bounding box correction and missed detection completion.

[0086] In one implementation, a user can sequentially browse each frame of the current video to be labeled in chronological order to correct the recognition result corresponding to the current video. In this method, the user needs to browse each video frame individually and perform correction for each frame. After completing the correction for one video frame, a switching operation is required to correct the next video frame.

[0087] In another implementation, prior to step 1, the video annotation method may also include:

[0088] Step 1: Stitch the video frames in the current video to be labeled in chronological order to obtain a stitched image;

[0089] Step 2: Display the obtained stitched image and show the recognition result corresponding to the current video to be labeled in the obtained stitched image, so that the user can correct the recognition result corresponding to the current video to be labeled based on the displayed content.

[0090] In this implementation, the electronic device stitches together video frames from the current video to be labeled in chronological order to obtain a stitched image (also known as a two-dimensional panoramic image). The electronic device can then display the stitched image and show the recognition results corresponding to the current video to be labeled. For example, the stitched image can display the scene of the current video to be labeled, and each video frame's corresponding portion can display the bounding boxes of objects within that frame. Different objects have different colored bounding boxes, and the behavior of each object can be displayed. Accordingly, the user can browse the displayed content and correct the recognition results corresponding to the current video to be labeled based on the displayed content. In this way, the user can intuitively browse all video frames without switching video frames to perform individual corrections for each frame. That is, discrete temporal labeling tasks can be integrated into a unified spatial labeling task, and consequently, the user can conveniently and quickly correct the recognition results corresponding to the video to be labeled.

[0091] After the user completes the correction of the recognition result corresponding to the current video to be labeled, the electronic device can obtain the correction result obtained by the human to correct the recognition result corresponding to the current video to be labeled, and use it as the current labeling result to be utilized. Subsequently, the electronic device can input the current prompt words, the current labeling result to be utilized, and the current video to be labeled into the current multimodal large model to generate attribute description text for each object in the current video to be labeled and scene description text for the current video to be labeled, thus obtaining the current description text to be corrected.

[0092] Based on the above processing, users can correct the recognition results corresponding to the current video to be labeled, thereby obtaining a more accurate labeling result. Then, the electronic device can use the more accurate labeling result to generate descriptive text, which can further improve the accuracy of the generated descriptive text. This improves the accuracy of the final labeling result of the video to be labeled obtained subsequently based on the generated descriptive text, further ensuring that video labeling can be achieved quickly and accurately.

[0093] In one embodiment, see Figure 2 , Figure 2 This is a schematic diagram of a second flowchart of the video annotation method provided in this application embodiment. Before step S101, the video annotation method further includes:

[0094] Step S105: Obtain multiple videos to be processed;

[0095] Step S106: For each video to be processed, identify the scene of the video to be processed, the position of the same object in different video frames in the video to be processed, and the behavior of each object, and obtain the recognition result corresponding to the video to be processed.

[0096] Step S107: Based on the recognition results corresponding to each video to be processed, sort each video to be processed according to the preset scene priority order and / or preset behavior priority order;

[0097] Step S101 includes:

[0098] Step S1011: Based on the obtained sorting results, select the next video to be processed that has not yet obtained the final annotation result as the current video to be annotated, and obtain the recognition result corresponding to the current video to be annotated.

[0099] In this embodiment of the application, there can be multiple videos that need to be labeled (i.e., videos to be processed). In one implementation, the electronic device can acquire multiple raw videos captured by video acquisition devices set up in different scenes as videos to be processed.

[0100] In another implementation, step S105 includes: acquiring multiple original videos; for each original video, extracting keyframes from the original video to obtain the video to be processed corresponding to the original video.

[0101] In this implementation, electronic devices can acquire multiple raw videos captured by video capture devices set up in different scenarios. A preset keyframe extraction algorithm can then be used to extract keyframes from each raw video, resulting in the corresponding video to be processed. For example, the preset keyframe extraction algorithm could be: inter-frame difference method, which determines whether a frame is a keyframe by calculating the difference between consecutive frames (such as pixel difference or histogram difference). When the difference exceeds a set threshold, the current frame is selected as a keyframe; clustering analysis method, which extracts feature vectors (such as color and texture) from all frames and groups them using clustering algorithms such as K-means, with the center frame of each group serving as a keyframe; motion analysis method, which detects moments of drastic motion change in the video and extracts frames before and after the motion change as keyframes. In this way, frames can be extracted from each raw video, retaining the keyframe sequence with significant motion, resulting in the corresponding video to be processed. This preserves key data, reduces the amount of video data to be processed subsequently, and improves the efficiency of video annotation.

[0102] The electronic device can acquire multiple videos to be processed. For each video, it identifies the scene of the video, the position of the same object in different video frames, and the behavior of each object to obtain the recognition result corresponding to the video. For specific recognition methods, please refer to the relevant description of step S101 in the above embodiments.

[0103] Users can pre-set the scene priority order (i.e., preset scene priority order). For example, in the preset scene priority order, scenes with fewer occurrences (which can be called rare scenes) or scenes that require special attention (such as schools, hospitals, etc.) have higher priority. Users can also pre-set the behavior priority order (i.e., preset behavior priority order). For example, in the preset behavior priority order, behaviors with fewer occurrences (which can be called rare behaviors) or behaviors that require special attention (such as fighting, smoking, etc.) have higher priority.

[0104] The electronic device can sort the videos to be processed according to the recognition results corresponding to each video, based on a preset scene priority order and / or a preset behavior priority order. For example, the videos to be processed can first be sorted according to the preset scene priority order, based on the scene of each video; for videos to be processed belonging to the same scene, they can be sorted according to the preset behavior priority order, based on the behaviors present in each video. For any behavior, if at least one object in a video exhibits that behavior, then that behavior is present in the video.

[0105] Then, the electronic device can select the next video to be processed that has not yet obtained a final annotation result according to the obtained sorting result, and use it as the current video to be annotated, and obtain the recognition result corresponding to the current video to be annotated. After completing step S104, the electronic device can return to execute step S1011 until all videos to be processed have been selected.

[0106] Based on the above processing, videos with specific scenes or specific behaviors can be labeled first. The prompt words and multimodal large model can be optimized based on the labeling results of videos with specific scenes or specific behaviors. This will improve the ability of the multimodal large model to label videos with specific scenes or specific behaviors, and further ensure that video labeling is achieved quickly and accurately.

[0107] In one embodiment, see Figure 3 , Figure 3 This is a schematic diagram of a third process for the video annotation method provided in this application embodiment. The video annotation method may include:

[0108] Step S301: Obtain raw video data. That is, obtain multiple raw videos.

[0109] Step S302: Intelligent frame extraction to obtain key information frames. That is, for each original video, key frames are extracted from the original video to obtain the corresponding video to be processed.

[0110] Step S303: Scene sorting, prioritizing rare scenes. That is, for each video to be processed, identify the scene of the video to be processed; and sort the videos to be processed according to the preset scene priority order based on the scene of each video to be processed.

[0111] Step S304: Person re-identification model matches human bounding boxes. That is, for each video to be processed, the position of the same object in different video frames of the video to be processed is identified.

[0112] Step S305: Behavior ranking, prioritizing rare behaviors. That is, for each video to be processed, identify the behavior of each object in that video. For videos belonging to the same scene, they can be ranked according to a preset behavior priority order, based on the behaviors present in each video.

[0113] Step S306: Keyframes are spliced ​​in time sequence. That is, step one in the above embodiment.

[0114] Step S307: Manually modify the human-shaped frame. That is, step two in the above embodiment.

[0115] Step S308: The visual language model pre-annotates the person's attributes and image descriptions. That is, step S102 in the above embodiment.

[0116] Step S309: Manually correct the character attributes and image descriptions to obtain the processed data. That is, step S103 in the above embodiment.

[0117] Step S310: Is the prompt word setting reasonable? If yes, proceed to step S312; if no, proceed to step S311. That is, the user determines whether the prompt word setting is reasonable based on the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected.

[0118] Step S311: Optimize prompt words. That is, obtain new prompt words by correcting the current prompt words. You can also use the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected to adjust the parameters of the current multimodal large model to obtain a new multimodal large model.

[0119] Step S312: Obtain prompts that meet specific requirements. That is, obtain new prompts.

[0120] Based on the above processing, while achieving video annotation, it is possible to perform human-machine collaborative iterative optimization of the multimodal large model. That is, the model's generalization ability can be optimized through multiple rounds of feedback, thereby improving the quality of the annotated data. This can significantly improve the efficiency and accuracy of video annotation, reduce the workload of manual annotation, and at the same time, the multimodal large model also supports multi-object and multi-attribute annotation in complex scenes.

[0121] In the technical solution of this application, the operations of obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0122] It should be noted that the deep learning model or multimodal large model used for identification in this embodiment is not a model for a specific user and cannot reflect the personal information of a specific user.

[0123] Based on the same inventive concept, this application also provides a video annotation device, see [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of a video annotation device provided in an embodiment of this application. The device includes:

[0124] The video to be labeled acquisition module 401 is used to acquire the current video to be labeled and the recognition result obtained by recognizing the current video to be labeled; wherein, the recognition result includes: the scene of the current video to be labeled, the position of the same object in different video frames in the current video to be labeled, and the behavior of each object;

[0125] The description text generation module 402 is used to combine the current video to be labeled and the corresponding recognition results, and use the current multimodal large model and prompt words to generate attribute description text for each object in the current video to be labeled and scene description text for the current video to be labeled, so as to obtain the current description text to be corrected.

[0126] The annotation result acquisition module 403 is used to acquire the correction result obtained by correcting the current description text to be corrected, and use it as the final annotation result of the current video to be annotated.

[0127] The optimized data acquisition module 404 is used to acquire new prompt words obtained by correcting the current prompt words, and to adjust the parameters of the current multimodal large model by using the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected, so as to obtain a new multimodal large model; wherein, the new prompt words are determined according to the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected.

[0128] Based on the video annotation apparatus provided in this application embodiment, it is possible to obtain the video to be annotated and the corresponding recognition results. The recognition results corresponding to the video to be annotated include the scene of the video to be annotated, the position of the same object in different video frames, and the behavior of each object, which can help the multimodal large model understand the attributes of each object in the video to be annotated and the scene content of the video to be annotated. Accordingly, by combining the video to be annotated and the corresponding recognition results, the multimodal large model and prompt words can be used to generate attribute description text for each object in the video to be annotated and scene description text for the video to be annotated (i.e., the description text to be corrected). Furthermore, the correction result obtained by correcting the current description text to be corrected (i.e., the final annotation result of the video to be annotated) can be obtained, which can further improve the accuracy of the final annotation result.

[0129] The new prompts are obtained by correcting the current prompts based on the differences between the final annotation results of the video to be annotated and the descriptive text to be corrected. Consequently, using the new prompts, more accurate descriptive text can be generated subsequently. Furthermore, the differences between the final annotation results of the video to be annotated and the descriptive text to be corrected can be used to optimize the current multimodal model. Consequently, using the new multimodal model, more accurate descriptive text can be generated subsequently. Thus, while ensuring high accuracy in the annotation results of the current video to be annotated, the prompts and multimodal model can be further optimized to improve the accuracy of the generated descriptive text, further ensuring the accuracy of subsequent annotations. Moreover, annotation using the multimodal model requires minimal manual intervention, enabling fast and accurate video annotation.

[0130] In one embodiment, the descriptive text generation module 402 includes:

[0131] The result correction submodule is used to obtain the correction result obtained by manually correcting the recognition result corresponding to the current video to be labeled, and use it as the current labeling result to be utilized.

[0132] The description generation submodule is used to input the current prompt words, the current annotation results to be utilized, and the current video to be annotated into the current multimodal large model, generate the attribute description text of each object in the current video to be annotated and the scene description text of the current video to be annotated, and obtain the current description text to be corrected.

[0133] In one embodiment, the apparatus further includes:

[0134] The image stitching module is used to stitch the video frames in the current video to be labeled in chronological order before obtaining the correction result obtained by manually correcting the recognition result corresponding to the current video to be labeled, and using it as the current labeling result to be used;

[0135] The image display module is used to display the obtained stitched image and show the recognition result corresponding to the current video to be labeled in the obtained stitched image, so that the user can correct the recognition result corresponding to the current video to be labeled based on the displayed content.

[0136] In one embodiment, the apparatus further includes:

[0137] The video acquisition module is used to acquire multiple videos to be processed before acquiring the current video to be labeled and the recognition result obtained by recognizing the current video to be labeled;

[0138] The video recognition module is used to identify the scene of each video to be processed, the position of the same object in different video frames, and the behavior of each object, so as to obtain the recognition result corresponding to the video to be processed.

[0139] The video sorting module is used to sort the videos to be processed according to the recognition results corresponding to each video, in a preset scene priority order and / or a preset behavior priority order.

[0140] The video to be labeled acquisition module 401 is specifically used to select the next video to be processed that has not yet obtained a final labeling result according to the obtained sorting result, as the current video to be labeled, and to obtain the recognition result corresponding to the current video to be labeled.

[0141] In one embodiment, the video acquisition module is specifically used to acquire multiple original videos; for each original video, key frames are extracted from the original video to obtain the video to be processed corresponding to the original video.

[0142] This application also provides an electronic device, such as... Figure 5 As shown, it includes:

[0143] Memory 501 is used to store computer programs;

[0144] When processor 502 executes the program stored in memory 501, it performs the following steps:

[0145] The process involves acquiring the current video to be labeled and the recognition results obtained by identifying the current video to be labeled; wherein, the recognition results include: the scene of the current video to be labeled, the position of the same object in different video frames in the current video to be labeled, and the behavior of each object;

[0146] By combining the current video to be labeled and the corresponding recognition results, and using the current multimodal large model and prompt words, the attribute description text of each object in the current video to be labeled and the scene description text of the current video to be labeled are generated, thus obtaining the current description text to be corrected.

[0147] Obtain the correction result obtained by correcting the current description text to be corrected, and use it as the final annotation result of the current video to be annotated;

[0148] The system obtains new prompt words by correcting the current prompt words, and adjusts the parameters of the current multimodal large model by using the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected, thus obtaining a new multimodal large model. The new prompt words are determined based on the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected.

[0149] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 502, communication interface, and memory 501 communicating with each other via the communication bus.

[0150] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0151] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0152] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0153] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0154] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described video annotation methods.

[0155] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the video annotation methods described above.

[0156] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0157] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, storage media, and program products are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0159] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A video annotation method, characterized in that, The method includes: The process involves acquiring the current video to be labeled and the recognition results obtained by identifying the current video to be labeled; wherein, the recognition results include: the scene of the current video to be labeled, the position of the same object in different video frames in the current video to be labeled, and the behavior of each object; the current video to be labeled is obtained by extracting keyframes from the original video; The correction result obtained by manually correcting the recognition result corresponding to the current video to be labeled is used as the current labeling result to be utilized. Input the current prompt words, the current annotation results to be used, and the current video to be annotated into the current multimodal large model to generate the attribute description text of each object in the current video to be annotated and the scene description text of the current video to be annotated, and obtain the current description text to be corrected. Obtain the correction result obtained by correcting the current description text to be corrected, and use it as the final annotation result of the current video to be annotated; The system obtains new prompt words by correcting the current prompt words, and adjusts the parameters of the current multimodal large model by using the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected, thus obtaining a new multimodal large model. The new prompt words are determined based on the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected.

2. The method according to claim 1, characterized in that, Before obtaining the correction result obtained by manually correcting the recognition result corresponding to the current video to be labeled, and using it as the current labeling result to be utilized, the method further includes: The video frames in the current video to be labeled are stitched together in chronological order to obtain a stitched image; The stitched image is displayed, and the recognition result corresponding to the current video to be labeled is shown in the stitched image, so that the user can correct the recognition result corresponding to the current video to be labeled based on the displayed content.

3. The method according to claim 1 or 2, characterized in that, Before obtaining the current video to be labeled and the recognition result obtained by recognizing the current video to be labeled, the method further includes: Acquire multiple videos to be processed; For each video to be processed, identify the scene of the video, the position of the same object in different video frames, and the behavior of each object to obtain the recognition result corresponding to the video to be processed. Based on the recognition results corresponding to each video to be processed, the videos to be processed are sorted according to the preset scene priority order and / or preset behavior priority order. The process of acquiring the current video to be labeled and the recognition result obtained by recognizing the current video to be labeled includes: Based on the obtained sorting results, select the next video to be processed that has not yet obtained the final annotation result as the current video to be annotated, and obtain the recognition result corresponding to the current video to be annotated.

4. The method according to claim 3, characterized in that, The acquisition of multiple videos to be processed includes: Acquire multiple raw videos; For each original video, keyframes are extracted to obtain the corresponding video to be processed.

5. A video annotation device, characterized in that, The device includes: The unlabeled video acquisition module is used to acquire the current unlabeled video and the recognition results obtained by recognizing the current unlabeled video; wherein, the recognition results include: the scene of the current unlabeled video, the position of the same object in different video frames in the current unlabeled video, and the behavior of each object; the current unlabeled video is obtained by extracting keyframes from the original video; The result correction submodule is used to obtain the correction result obtained by manually correcting the recognition result corresponding to the current video to be labeled, and use it as the current labeling result to be utilized. The description generation submodule is used to input the current prompt words, the current annotation results to be utilized, and the current video to be annotated into the current multimodal large model, generate the attribute description text of each object in the current video to be annotated and the scene description text of the current video to be annotated, and obtain the current description text to be corrected. The annotation result acquisition module is used to obtain the correction result obtained by correcting the current descriptive text to be corrected, and use it as the final annotation result of the current video to be annotated; The data acquisition module is optimized to acquire new prompt words obtained by correcting the current prompt words, and to adjust the parameters of the current multimodal large model by using the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected, so as to obtain a new multimodal large model. The new prompt words are determined according to the difference between the final annotation result of the current video to be annotated and the current descriptive text to be corrected.

6. The apparatus according to claim 5, characterized in that, The device further includes: The image stitching module is used to stitch the video frames in the current video to be labeled in chronological order before obtaining the correction result obtained by manually correcting the recognition result corresponding to the current video to be labeled, and using it as the current labeling result to be used; The image display module is used to display the obtained stitched image and show the recognition result corresponding to the current video to be labeled in the obtained stitched image, so that the user can correct the recognition result corresponding to the current video to be labeled based on the displayed content; And / or, The device further includes: The video acquisition module is used to acquire multiple videos to be processed before acquiring the current video to be labeled and the recognition result obtained by recognizing the current video to be labeled; The video recognition module is used to identify the scene of each video to be processed, the position of the same object in different video frames, and the behavior of each object, so as to obtain the recognition result corresponding to the video to be processed. The video sorting module is used to sort the videos to be processed according to the recognition results corresponding to each video, in a preset scene priority order and / or a preset behavior priority order. The video to be labeled acquisition module is specifically used to select the next video to be processed that has not yet obtained a final labeling result according to the obtained sorting result, as the current video to be labeled, and to obtain the recognition result corresponding to the current video to be labeled. And / or, The video acquisition module is specifically used to acquire multiple original videos; for each original video, key frames are extracted to obtain the corresponding video to be processed.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-4.

9. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method according to any one of claims 1-4.

Images