A method and apparatus for identifying empty mirror tags

Abstract

The application provides a method and device for identifying empty mirror tags, which comprises the following steps: extracting a key frame image and audio data corresponding to the key frame image; processing the key frame image by using a first model and a second model to obtain first empty mirror tags matched by the key frame image in each preset classification dimension; inputting the key frame image and the audio data corresponding to the key frame image into an empty mirror corpus large language model to obtain second empty mirror tags matched by the key frame image in each preset classification dimension; and fusing the first empty mirror tags and the second empty mirror tags matched by the key frame image in each preset classification dimension respectively to obtain final empty mirror tags matched by the key frame image in each preset classification dimension. According to the scheme, the empty mirror tags of the image are automatically identified, and the empty mirror materials no longer need to be classified manually by business personnel, so that the screening cost is reduced and the data classification efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method and apparatus for identifying empty mirror labels. Background Technology

[0002] During video shooting, to facilitate data reuse, it is necessary to summarize and classify the establishing footage generated during the shooting process. Currently, this mainly relies on business personnel to manually summarize and classify the establishing footage. However, the aforementioned method requires business personnel to spend a lot of time and energy browsing videos and filtering out establishing footage for summarization and classification, resulting in high screening costs and low data summarization and classification efficiency. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method and apparatus for identifying empty shot tags, in order to solve the problems of high screening costs and low data summarization and classification efficiency that exist in the method of manually summarizing and classifying empty shot materials by business personnel.

[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0005] The first aspect of this invention discloses a method for identifying empty camera tags, the method comprising:

[0006] Extract keyframe images and the corresponding audio data of the keyframe images;

[0007] The first empty shot label is processed using a preset first model and a second model. Multiple empty shot labels are set under each preset classification dimension. The first model and the second model are trained based on sample data. The first model and the second model are different models.

[0008] The keyframe image and its corresponding audio data are input into a preset empty scene corpus large language model to obtain the second empty scene label under each preset classification dimension matched by the keyframe image.

[0009] The first empty shot label and the second empty shot label matched by the keyframe image under each preset classification dimension are fused to obtain the final empty shot label matched by the keyframe image under each preset classification dimension.

[0010] Preferably, the keyframe image is processed using a preset first model and a preset second model to obtain the first empty shot label under each preset classification dimension matched by the keyframe image, including:

[0011] The keyframe image is input into a preset first model for empty shot label recognition to obtain the empty shot label recognition result corresponding to the keyframe image. The empty shot label recognition result includes: the label score of the empty shot label set under each preset classification dimension.

[0012] Based on the empty shot label recognition results, empty shot labels that match the keyframe images under each of the preset classification dimensions are initially selected;

[0013] The keyframe image and the initially selected empty shot labels are input into a preset second model to determine the degree of label matching, and the label matching result is obtained. The label matching result includes the matching score between the keyframe image and the initially selected empty shot labels.

[0014] Based on the tag matching results, the first empty shot tags under each preset classification dimension matched by the keyframe image are selected.

[0015] Preferably, the keyframe image and its corresponding audio data are input into a preset empty-scene corpus large language model to obtain the second empty-scene label under each preset classification dimension matched by the keyframe image, including:

[0016] The keyframe image and its corresponding audio data are input into a preset empty-scene corpus large language model. The empty-scene corpus large language model is then questioned using a preset first question template, causing the empty-scene corpus large language model to output the second empty-scene label under each preset classification dimension matched by the keyframe image.

[0017] Preferably, the first empty shot labels and second empty shot labels matched by the keyframe image under each preset classification dimension are fused to obtain the final empty shot labels matched by the keyframe image under each preset classification dimension, including:

[0018] For each preset classification dimension, the first empty shot label and the second empty shot label matched by the keyframe image under the preset classification dimension are input into the empty shot corpus large language model. The empty shot corpus large language model is then asked a question in combination with the preset second question template, so that the empty shot corpus large language model fuses the first empty shot label and the second empty shot label and outputs the final empty shot label matched by the keyframe image under the preset classification dimension.

[0019] Preferably, based on the empty shot label recognition results, empty shot labels that match the keyframe images under each of the preset classification dimensions are initially selected, including:

[0020] For each preset classification dimension, based on the empty shot label recognition results, the empty shot labels with the highest scores in the preset classification dimension are selected to obtain the initially selected empty shot labels that match the keyframe images in the preset classification dimension.

[0021] Preferably, based on the tag matching results, the first empty shot tags under each preset classification dimension matched by the keyframe image are filtered out, including:

[0022] For each preset classification dimension, based on the tag matching results, the empty shot tags with the highest matching scores in the preset classification dimension are selected to obtain the first empty shot tags for each preset classification dimension matched by the keyframe image.

[0023] A second aspect of this invention discloses an apparatus for identifying empty mirror labels, the apparatus comprising:

[0024] An extraction unit is used to extract keyframe images and audio data corresponding to the keyframe images;

[0025] The processing unit is used to process the keyframe image using a preset first model and a second model to obtain the first empty shot label of the keyframe image under each preset classification dimension matched. Multiple empty shot labels are set under each preset classification dimension. The first model and the second model are trained based on sample data. The first model and the second model are different models.

[0026] The acquisition unit is used to input the keyframe image and its corresponding audio data into a preset empty shot corpus large language model to obtain the second empty shot label under each preset classification dimension matched by the keyframe image.

[0027] The fusion unit is used to fuse the first empty shot label and the second empty shot label under each preset classification dimension matched by the keyframe image to obtain the final empty shot label under each preset classification dimension matched by the keyframe image.

[0028] Preferably, the processing unit includes:

[0029] The recognition module is used to input the keyframe image into a preset first model to perform empty shot label recognition, and obtain the empty shot label recognition result corresponding to the keyframe image. The empty shot label recognition result includes: the label score of the empty shot label set under each preset classification dimension.

[0030] The first filtering module is used to initially filter out empty shot labels that match the keyframe image under each preset classification dimension based on the empty shot label recognition results.

[0031] The matching module is used to input the keyframe image and the initially selected empty shot labels into a preset second model to determine the degree of label matching and obtain the label matching result. The label matching result includes the matching score between the keyframe image and the initially selected empty shot labels.

[0032] The second filtering module is used to filter out the first empty shot labels under each preset classification dimension matched by the keyframe image based on the label matching results.

[0033] Preferably, the acquisition unit is specifically used to: input the keyframe image and its corresponding audio data into a preset empty-scene corpus large language model, and ask questions to the empty-scene corpus large language model in combination with a preset first question template, so that the empty-scene corpus large language model outputs the second empty-scene label under each preset classification dimension matched by the keyframe image.

[0034] Preferably, the fusion unit is specifically used to: for each preset classification dimension, input the first empty shot label and the second empty shot label matched by the keyframe image under the preset classification dimension into the empty shot corpus large language model, and ask questions to the empty shot corpus large language model in combination with the preset second question template, so that the empty shot corpus large language model fuses the first empty shot label and the second empty shot label and outputs the final empty shot label matched by the keyframe image under the preset classification dimension.

[0035] Based on the above embodiments of the present invention, a method and apparatus for identifying empty shot tags are provided. The method includes: extracting keyframe images and corresponding audio data; processing the keyframe images using a first model and a second model to obtain first empty shot tags for each preset classification dimension matched by the keyframe images; inputting the keyframe images and their corresponding audio data into an empty shot corpus large language model to obtain second empty shot tags for each preset classification dimension matched by the keyframe images; and fusing the first and second empty shot tags for each preset classification dimension matched by the keyframe images to obtain final empty shot tags for each preset classification dimension matched by the keyframe images. This solution processes keyframe images using a first model and a second model to obtain first empty shot tags for each preset classification dimension matched by the keyframe images; and processes keyframe images and their corresponding audio data using an empty shot corpus large language model to obtain second empty shot tags for each preset classification dimension matched by the keyframe images. The first and second empty shot labels matched by the keyframe images under each preset classification dimension are fused to obtain the final empty shot labels matched by the keyframe images under each preset classification dimension. This eliminates the need for business personnel to manually summarize and classify the empty shot materials, reducing screening costs and improving data summarization and classification efficiency. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 A flowchart illustrating a method for identifying empty mirror labels provided in an embodiment of the present invention;

[0038] Figure 2 This is a flowchart of obtaining the first empty mirror label provided in an embodiment of the present invention;

[0039] Figure 3 Another flowchart of a method for identifying empty mirror labels provided in an embodiment of the present invention;

[0040] Figure 4 This is a structural block diagram of a device for identifying empty mirror labels provided in an embodiment of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0043] As can be seen from the background technology, during video shooting, it is necessary to summarize and classify the establishing shot footage generated during the shooting process for data reuse. Currently, this still relies on business personnel to manually summarize and classify the establishing shot footage. The aforementioned method requires business personnel to spend a lot of time and energy browsing videos and filtering out establishing shot footage for summarization and classification, which is costly and inefficient.

[0044] Therefore, this solution provides a method and apparatus for identifying empty shot tags. It processes keyframe images using a first model and a second model to obtain first empty shot tags for each preset classification dimension matched by the keyframe images. It then processes the keyframe images and their corresponding audio data using a large language model of the empty shot corpus to obtain second empty shot tags for each preset classification dimension matched by the keyframe images. The first and second empty shot tags for each preset classification dimension matched by the keyframe images are then fused to obtain the final empty shot tags for each preset classification dimension matched by the keyframe images. This eliminates the need for manual classification of empty shot materials by business personnel, thereby reducing screening costs and improving data classification efficiency.

[0045] It should be noted that the establishing shot material (also known as establishing shot or landscape shot) involved in this plan specifically refers to: similar landscape, mountains and rivers, and tall buildings generated during the filming process of film and television variety shows.

[0046] See Figure 1 The flowchart illustrates a method for identifying empty scene labels provided by an embodiment of the present invention. The method includes:

[0047] Step S101: Extract keyframe images and the corresponding audio data.

[0048] It should be noted that the video includes images and the corresponding audio data at the image locations.

[0049] In the specific implementation step S101, the video footage of the empty shot to be processed is acquired, and frame extraction processing is performed based on the video footage of the empty shot to be processed, thereby extracting the key frame images and the audio data (i.e., audio features) corresponding to the key frame images in the video footage of the empty shot to be processed.

[0050] Specifically, the video footage of the empty shot can be processed by frame extraction using N-fps, thereby extracting the keyframe images from the video footage of the empty shot.

[0051] Step S102: Process the keyframe image using the preset first model and second model to obtain the first empty shot label under each preset classification dimension matched by the keyframe image.

[0052] It should be noted that the first model and the second model are different models. The first model can be called a multi-label model, and the second model can be called an image-text comparison model. Multiple preset classification dimensions are pre-defined, and each preset classification dimension has multiple empty shot labels (or contains multiple empty shot labels). For example, there are four preset classification dimensions: time, space, weather, and video content. The time dimension includes empty shot labels such as dawn, morning, forenoon, noon, afternoon, dusk, and evening; the space dimension includes empty shot labels such as indoor and outdoor; the weather dimension includes empty shot labels such as sunny, cloudy, rainy, and snowy; and the video content dimension includes empty shot labels such as people, no people, natural scenery (e.g., trees, streams), and architecture (e.g., pavilions, bridges, skyscrapers). In short, the empty shot labels set under each preset classification dimension are a refined classification of that preset classification dimension.

[0053] The first model (multi-label model) and the second model (image-text comparison model) are trained based on sample data. To better understand the subsequent content of this solution, the process of training the first and second models will be explained first.

[0054] Explanation regarding the training to obtain the first model:

[0055] Establishment footage is collected based on multiple preset classification dimensions for business needs (such as time dimension, space dimension, weather dimension and video content dimension). The establishment footage tags are refined based on each preset classification dimension to complete data annotation, thereby obtaining sample data. The sample data includes at least sample images labeled with establishment footage tags and the corresponding audio data of the sample images.

[0056] For example: in the time dimension, sample images in the collected empty shot footage are labeled with empty shot tags such as dawn, morning, morning, noon, afternoon, dusk, and evening; in the space dimension, sample images in the collected empty shot footage are labeled with empty shot tags such as indoor and outdoor; in the weather dimension, sample images in the collected empty shot footage are labeled with empty shot tags such as sunny, cloudy, rainy, and snowy; in the video content dimension, sample images in the collected empty shot footage are labeled with empty shot tags such as people, no people, natural scenery (such as trees, streams, etc.), and architecture (such as pavilions, bridges, and tall buildings).

[0057] Based on the sample images labeled with empty shot tags in the above sample data, the first model (multi-label model) is trained. During the training process, an attention mechanism is introduced to improve the model's feature representation, thereby improving the accuracy of the first model in outputting empty shot tags.

[0058] Explanation regarding the training to obtain the second model:

[0059] Based on the sample data collected in the above "Explanation on Training the First Model", a second model is trained. The input of the second model is an image, and the output information of the second model contains empty image labels under multiple preset classification dimensions.

[0060] The above content explains how the first and second models were obtained through training.

[0061] In the specific implementation step S102, the keyframe image is processed using the first model and the second model to obtain the first empty shot label of the keyframe image under each preset classification dimension (or obtain the first empty shot label of the keyframe image under each preset classification dimension).

[0062] For example: obtain the first empty shot label in the time dimension matched by the keyframe image, obtain the first empty shot label in the spatial dimension matched by the keyframe image, obtain the first empty shot label in the spatial dimension matched by the keyframe image, and obtain the first empty shot label in the video content dimension matched by the keyframe image.

[0063] Step S103: Input the keyframe image and its corresponding audio data into the preset empty shot corpus large language model to obtain the second empty shot label under each preset classification dimension matched by the keyframe image.

[0064] It should be noted that the empty shot corpus large language model is obtained by adjusting the large language model (LLM) based on sample data. Specifically, the empty shot material tag corpus information, sample images, and audio data corresponding to the sample images in the "sample data" collected in step S102 above are used to adjust the large language model, thereby completing the training of the empty shot corpus large language model. The empty shot material tag corpus information includes empty shot tags used to annotate the sample images.

[0065] In the specific implementation step S103, the keyframe image and its corresponding audio data are input into the preset empty shot corpus large language model. The empty shot corpus large language model is then questioned in conjunction with the preset first question template, so that the empty shot corpus large language model outputs the second empty shot label matched by the keyframe image under each preset classification dimension (or, the empty shot corpus large language model outputs the second empty shot label matched by the keyframe image under each preset classification dimension).

[0066] It should be noted that the question template is the prompt used in the large language model application; that is, the desired answer is obtained by asking the large language model a question through the prompt. In the process of applying the empty-speech corpus large language model, each preset classification dimension has a corresponding first question template. This solution uses a "question + rule + example" approach to optimize the first question template for each preset classification dimension, thereby making the generated results of the empty-speech corpus large language model more stable and reliable.

[0067] For example, the process of optimizing the first question template using the "question + rule + example" approach is shown in the following process A1-A4.

[0068] A1. Add rules to make the output of the large language model of empty mirror corpus more standardized and avoid outputting unnecessary redundant information.

[0069] A2. Add rules to make the large language model of empty corpus only need to output concise results, without needing to output the analysis process.

[0070] A3. Provide the Empty Mirror Corpus Large Language Model with some examples of questions and answers, so that the Empty Mirror Corpus Large Language Model can answer the questions in the manner given in the examples.

[0071] A4. Optimize the questioning methods for different preset classification dimensions to reduce the inference time of the large language model for empty-speech corpus.

[0072] In other words, first question templates are written for multiple preset classification dimensions and optimized to the best state; after inputting the keyframe image and its corresponding audio data into the empty shot corpus language model, questions are asked to the empty shot corpus language model in combination with the first question templates corresponding to the preset classification dimensions, so that the empty shot corpus language model outputs in each preset classification dimension, thereby obtaining the second empty shot label matched by the keyframe image in each preset classification dimension.

[0073] For example, after inputting the keyframe images and their corresponding audio data of the video footage of the empty shot into the empty shot corpus language model, the empty shot corpus language model is asked a question using the first question template "Is this video indoors or outdoors? Please answer". This causes the empty shot corpus language model to output the second empty shot label in the spatial dimension matched by the keyframe images.

[0074] Step S104: Merge the first empty shot label and the second empty shot label matched by the keyframe image under each preset classification dimension to obtain the final empty shot label matched by the keyframe image under each preset classification dimension.

[0075] In the specific implementation step S104, for each preset classification dimension, the first empty shot label and the second empty shot label matched by the keyframe image under the preset classification dimension are input into the empty shot corpus large language model. The empty shot corpus large language model is then asked a question in combination with the preset second question template, so that the empty shot corpus large language model can fuse the first empty shot label and the second empty shot label and output the final empty shot label matched by the keyframe image under the preset classification dimension.

[0076] It should be noted that each preset category dimension has a corresponding second question template, and the second question templates for each preset category dimension are pre-optimized in the form of "question + rule + example"; the first question template and the second question template mentioned above are different question templates.

[0077] In other words, for each preset classification dimension, the first and second empty shot labels matched by the keyframe image under that preset classification dimension are input into the empty shot corpus language model. Then, the second question template corresponding to that preset classification dimension is used to ask the empty shot corpus language model a question. The empty shot corpus language model then fuses the first and second empty shot labels (duplicate empty shot labels can be removed during the fusion process). Finally, the empty shot corpus language model outputs the final empty shot label matched by the keyframe image under that preset classification dimension.

[0078] By using the second question template corresponding to each preset classification dimension, questions are asked separately on the large language model of the empty shot corpus for each preset classification dimension, thereby obtaining the final empty shot labels under each preset classification dimension matched by the keyframe image.

[0079] For example, suppose the second question template corresponding to a certain preset classification dimension is "The above are the empty shot label annotation results obtained by two annotators for the same video. Please combine these results to obtain the final description for this video". By asking the empty shot corpus big language model through this second question template, the final empty shot label under the preset classification dimension matched by the keyframe image can be obtained.

[0080] In this embodiment of the invention, keyframe images of the video footage containing empty shots are processed using a first model and a second model to obtain first empty shot tags matched by the keyframe images under various preset classification dimensions. The keyframe images and their corresponding audio data are then processed using a large language model of the empty shot corpus to obtain second empty shot tags matched by the keyframe images under various preset classification dimensions. The first and second empty shot tags matched by the keyframe images under various preset classification dimensions are then fused to obtain the final empty shot tags matched by the keyframe images under various preset classification dimensions. This eliminates the need for manual classification of empty shot footage by business personnel, reducing screening costs and improving data classification efficiency.

[0081] Regarding the above embodiments of the present invention Figure 1 Step S102, which involves "obtaining the first empty shot labels under each preset classification dimension matched by the keyframe image," refers to [the relevant section]. Figure 2 The flowchart illustrating the process of obtaining the first empty mirror label provided in an embodiment of the present invention is shown. Figure 2 Includes the following steps:

[0082] Step S201: Input the keyframe image into the preset first model to perform empty shot label recognition, and obtain the empty shot label recognition result corresponding to the keyframe image.

[0083] In the specific implementation step S201, the keyframe image is input into the first model for feature extraction and feature enhancement through an attention mechanism, thereby obtaining the empty shot label recognition result corresponding to the keyframe image output by the first model. The empty shot label recognition result includes the label score of the empty shot label set under each preset classification dimension.

[0084] Step S202: Based on the empty shot label recognition results, initially filter out the empty shot labels that match the keyframe images under each preset classification dimension.

[0085] In the specific implementation step S202, for each preset classification dimension, based on the empty shot label recognition result corresponding to the keyframe image, the empty shot labels with the top x positions of the label score under the preset classification dimension are selected to obtain the initially selected empty shot labels that match the keyframe image under the preset classification dimension; that is, the empty shot labels that match the keyframe image under a certain preset classification dimension are: the empty shot labels with the top-x positions of the label score under the preset classification dimension.

[0086] Specifically, based on the recognition results of the empty shot labels corresponding to the keyframe images, for each preset classification dimension, the empty shot labels included in that preset classification dimension are sorted according to their label scores. The empty shot labels with the highest label scores in that preset classification dimension are taken as: the initially selected empty shot labels that match the keyframe image in that preset classification dimension. For each preset classification dimension, the top x empty shot labels (top-x empty shot labels) are taken out, thus obtaining the initially selected empty shot labels that match the keyframe image in each preset classification dimension.

[0087] Step S203: Input the keyframe image and the initially selected empty shot labels into the preset second model to determine the degree of label matching and obtain the label matching result.

[0088] It should be noted that the empty shot labels initially selected are the empty shot labels that match the keyframe images under each preset classification dimension initially selected in step S202 above.

[0089] It should be further explained that the empty shot labels initially selected in step S202 are empty shot labels that match the keyframe images. However, the degree of matching between the keyframe images and these initially selected empty shot labels is different. Therefore, a second model (i.e., the image-text comparison model) is needed to calculate the degree of matching between the keyframe images and the initially selected empty shot labels. The degree of matching between the keyframe images and the empty shot labels is measured by the matching score between the keyframe images and the empty shot labels.

[0090] In the specific implementation step S203, the keyframe image and the initially selected empty shot labels are input into the second model to determine the degree of label matching, and the label matching result output by the second model is obtained. The label matching result includes the matching score between the keyframe image and the initially selected empty shot labels.

[0091] In other words, the keyframe image and the initially selected empty shot labels (i.e., the empty shot labels that match the keyframe image under each preset classification dimension) are input into the second model, and the second model calculates the matching score between the keyframe image and the initially selected empty shot labels to obtain the label matching result.

[0092] Step S204: Based on the label matching results, filter out the first empty shot labels under each preset classification dimension matched by the keyframe image.

[0093] In the specific implementation step S204, for each preset classification dimension, based on the label matching results, the empty shot labels ranked in the top y positions under that preset classification dimension are selected to obtain the first empty shot label matched by the keyframe image under that preset classification dimension. That is, the first empty shot label matched by the keyframe image under a certain preset classification dimension is: the empty shot labels with matching scores in the top-y positions under that preset classification dimension (the initially selected empty shot labels).

[0094] Specifically, based on the label matching results, for each preset classification dimension, the empty shot labels (preliminarily selected empty shot labels) included in that preset classification dimension are sorted according to their matching scores. The empty shot label with the highest matching score in that preset classification dimension is taken as the first empty shot label matched by the keyframe image in that preset classification dimension. For each preset classification dimension, the top y empty shot labels (top-y empty shot labels) are selected, thus obtaining the first empty shot label matched by the keyframe image in each preset classification dimension.

[0095] For example: For a certain preset classification dimension, the top 10 empty shot labels (top-10, x=10) with the highest label scores are taken from the preset classification dimension as the initial empty shot labels under the preset classification dimension; then the keyframe image and the 10 initially selected empty shot labels under the preset classification dimension are input into the second model to determine the label matching degree, and the matching score between the keyframe image and these 10 empty shot labels is obtained; the 10 empty shot labels are sorted according to the matching score, and the empty shot labels with the top 5 matching scores (top-5, y=5) are taken as the first empty shot label matched by the keyframe image under the preset classification dimension.

[0096] In other words, for each preset classification dimension, from the empty shot labels (the empty shot labels with the highest labels in the top 10) under the preset classification dimension initially selected by step S202, the top-y empty shot labels with higher matching scores are selected as the first empty shot labels under the preset classification dimension matched by the keyframe image.

[0097] It should be noted that the process of selecting the first empty shot label from the initially screened empty shot labels is equivalent to the process of roughly selecting the top-x empty shot labels and then finely selecting the top-y empty shot labels. This ensures that the keyframe image can be matched with the accurate empty shot label (that is, the first empty shot label) in each preset classification dimension.

[0098] The above embodiments of the present invention Figure 2 This is a description of the content related to "obtaining the first empty shot label under each preset classification dimension matched by the keyframe image".

[0099] To better understand the above embodiments of the present invention Figure 1 and Figure 2 The content, through Figure 3 Another flowchart illustrating a method for identifying empty camera labels is shown as an example. Figure 3 Includes the following steps:

[0100] Step S301: Input the video footage of the empty shot to be processed.

[0101] Step S302: Extract keyframe images from the video footage of the empty shot to be processed, and extract the audio data corresponding to the keyframe images from the video footage of the empty shot to be processed.

[0102] It should be noted that after extracting the keyframe image and its corresponding audio data, steps S303-S304 are executed to obtain the first empty shot label, and steps S305-S306 are executed to obtain the second empty shot label.

[0103] Step S303: Input the keyframe image into the first model. The first model outputs the label scores of each empty shot label in four preset classification dimensions. Take the top-x empty shot labels with higher label scores in each preset classification dimension as the initially selected empty shot labels.

[0104] It should be noted that the four preset classification dimensions are: time dimension, space dimension, weather dimension, and video content dimension.

[0105] In the specific implementation, the keyframe image is input into the first model for feature extraction to obtain feature V; then, self-attention feature enhancement is performed based on the extracted feature V to obtain attention-V, where the query, key, and value parameters involved in the feature enhancement process all come from feature V; the first model outputs the label scores of each empty shot label in four preset classification dimensions; the top-x empty shot labels with higher label scores in each preset classification dimension are taken as the initially selected empty shot labels.

[0106] Step S304: Input the keyframe image and the initially selected empty shot labels into the second model, calculate the matching score between the keyframe image and the initially selected empty shot labels; from the initially selected empty shot labels, take the top-y empty shot labels with higher matching scores for each preset classification dimension as the first empty shot labels.

[0107] It should be noted that after taking the top-y empty shot labels with the highest matching scores for each preset classification dimension as the first empty shot labels, we can obtain the multimodal empty shot material output results based on the first model and the second model. The multimodal empty shot material output results contain the first empty shot labels matched by each keyframe image under each preset classification dimension.

[0108] Since the output of multimodal empty shot footage can only contain the labeled empty shot tags and lacks diversity, and the matching results between the keyframe images and the first empty shot tags may be incorrect, simply obtaining the output of multimodal empty shot footage is not enough to complete the task of identifying empty shot tags. Therefore, steps S305-S306 need to be executed to obtain the second empty shot tags.

[0109] Step S305: Input the keyframe images and their corresponding audio data into the empty shot corpus large language model.

[0110] Step S306: Write a first question template for each of the four preset classification dimensions; ask questions to the empty shot corpus language model using the first question template, so that the empty shot corpus language model outputs the second empty shot label matched by the keyframe image under the four preset classification dimensions.

[0111] It should be noted that after writing the first question template, the first question template can be optimized by using the "question + example + result" method, so that the result LLM_result generated by the empty shot corpus large language model is more accurate. The LLM_result contains the second empty shot label matched by the keyframe image under the four preset classification dimensions.

[0112] Step S307: Input the first empty shot label and the second empty shot label matched by the keyframe image under the preset classification dimension into the empty shot corpus large language model.

[0113] Step S308: Use the second question template to ask questions to the empty scene corpus large language model, so that the empty scene corpus large language model can fuse the first empty scene label and the second empty scene label and output the final empty scene label matched by the key frame image under the preset classification dimension.

[0114] It should be noted that after writing the second question template, it can be optimized using the "question + example + result" approach.

[0115] The above embodiments of the present invention Figure 3 This is an example illustrating the proposed solution.

[0116] As can be seen from the above embodiments, this solution designs subcategories of empty shot tags for multiple preset classification dimensions (time, space, weather, video content, etc.) based on business needs, with corresponding subcategories of empty shot tags designed for each dimension. After applying this solution, the input video of the empty shot material to be processed generates the final empty shot tag corresponding to the video of the empty shot material to be processed. The final empty shot tag is output according to multiple preset classification dimensions, which makes it convenient for the business to classify the empty shot material generated during the shooting process, thereby reducing the cost of manual screening and improving the efficiency of data summarization and classification, and facilitating data reuse.

[0117] It should be noted that in practical applications, for the first model (i.e., the multi-label model) in this solution, audio features can also be incorporated, and cross-attention enhancement can be performed using these audio features to improve the accuracy of the first model's output. Furthermore, the scale of the "sample data" can be expanded based on massive amounts of data fed back from business stakeholders, and then the empty-scene corpus large language model can be further optimized to enhance the richness of its output; alternatively, the output of GPT can be integrated, and the final empty-scene label can be obtained by fusing the first empty-scene label, the second empty-scene label, and the GPT output.

[0118] Corresponding to the method for identifying empty mirror labels provided in the above embodiments of the present invention, see also... Figure 4 The present invention also provides a structural block diagram of a device for identifying empty mirror labels, the device including: an extraction unit 401, a processing unit 402, an acquisition unit 403, and a fusion unit 404;

[0119] Extraction unit 401 is used to extract keyframe images and audio data corresponding to the keyframe images.

[0120] The processing unit 402 is used to process the keyframe image using a preset first model and a second model to obtain the first empty shot label under each preset classification dimension matched by the keyframe image. Multiple empty shot labels are set under each preset classification dimension. The first model and the second model are trained based on sample data and are different models.

[0121] The acquisition unit 403 is used to input the keyframe image and its corresponding audio data into a preset empty shot corpus large language model to obtain the second empty shot label under each preset classification dimension matched by the keyframe image.

[0122] In the specific implementation, the acquisition unit 403 is specifically used to: input the key frame image and its corresponding audio data into the preset empty shot corpus large language model, and ask questions to the empty shot corpus large language model in combination with the preset first question template, so that the empty shot corpus large language model outputs the second empty shot label under each preset classification dimension matched by the key frame image.

[0123] The fusion unit 404 is used to fuse the first empty shot label and the second empty shot label under each preset classification dimension matched by the keyframe image to obtain the final empty shot label under each preset classification dimension matched by the keyframe image.

[0124] In a specific implementation, the fusion unit 404 is specifically used to: for each preset classification dimension, input the first empty shot label and the second empty shot label matched by the keyframe image under the preset classification dimension into the empty shot corpus large language model, and ask questions to the empty shot corpus large language model in combination with the preset second question template, so that the empty shot corpus large language model fuses the first empty shot label and the second empty shot label and outputs the final empty shot label matched by the keyframe image under the preset classification dimension.

[0125] In this embodiment of the invention, keyframe images are processed using a first model and a second model to obtain first empty shot tags matched by the keyframe images under various preset classification dimensions; the keyframe images and their corresponding audio data are processed using an empty shot corpus large language model to obtain second empty shot tags matched by the keyframe images under various preset classification dimensions. The first and second empty shot tags matched by the keyframe images under various preset classification dimensions are fused to obtain the final empty shot tags matched by the keyframe images under various preset classification dimensions. This eliminates the need for business personnel to manually summarize and classify empty shot materials, reducing screening costs and improving data summarization and classification efficiency.

[0126] Preferred, combined Figure 4 The processing unit 402, as shown, includes an identification module, a first filtering module, a matching module, and a second filtering module; the execution principle of each module is as follows:

[0127] The recognition module is used to input keyframe images into a preset first model to perform empty shot label recognition, and obtain the empty shot label recognition result corresponding to the keyframe image. The empty shot label recognition result includes: the label score of the empty shot label set under each preset classification dimension.

[0128] The first filtering module is used to initially filter out empty shot labels that match the keyframe images under each preset classification dimension based on the empty shot label recognition results.

[0129] In the specific implementation, the first filtering module is used to: for each preset classification dimension, based on the empty shot label recognition result, filter out the empty shot labels with the top x positions of the label score under the preset classification dimension, so as to obtain the empty shot labels that match the key frame image under the preset classification dimension.

[0130] The matching module is used to input the keyframe image and the initially selected empty shot labels into a preset second model to determine the degree of label matching and obtain the label matching result. The label matching result includes the matching score between the keyframe image and the initially selected empty shot labels.

[0131] The second filtering module is used to filter out the first empty shot labels under each preset classification dimension matched by the keyframe image based on the label matching results.

[0132] In the specific implementation, the second filtering module is used to: for each preset classification dimension, based on the label matching results, filter out the empty shot labels that rank in the top y positions under the preset classification dimension, so as to obtain the first empty shot labels under each preset classification dimension matched by the keyframe image.

[0133] In summary, this invention provides a method and apparatus for identifying empty shot tags. It processes keyframe images using a first model and a second model to obtain first empty shot tags matched by the keyframe images under various preset classification dimensions. It then processes the keyframe images and their corresponding audio data using a large language model of the empty shot corpus to obtain second empty shot tags matched by the keyframe images under various preset classification dimensions. By fusing the first and second empty shot tags matched by the keyframe images under various preset classification dimensions, the final empty shot tags matched by the keyframe images under various preset classification dimensions are obtained. This eliminates the reliance on manual classification of empty shot materials by business personnel, reducing screening costs and improving data classification efficiency.

[0134] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0135] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0136] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying empty mirror labels, characterized in that, The method includes: Extract keyframe images and the corresponding audio data of the keyframe images; The first empty shot label is processed using a preset first model and a second model. Multiple empty shot labels are set under each preset classification dimension. The first model and the second model are trained based on sample data. The first model and the second model are different models. The keyframe image and its corresponding audio data are input into a preset empty scene corpus large language model to obtain the second empty scene label under each preset classification dimension matched by the keyframe image. The first empty shot label and the second empty shot label matched by the keyframe image under each preset classification dimension are fused to obtain the final empty shot label matched by the keyframe image under each preset classification dimension. The keyframe image is processed using a preset first model and a second model to obtain the first empty shot label under each preset classification dimension matched by the keyframe image, including: The keyframe image is input into a preset first model for empty shot label recognition to obtain the empty shot label recognition result corresponding to the keyframe image. The empty shot label recognition result includes: the label score of the empty shot label set under each preset classification dimension. Based on the empty shot label recognition results, empty shot labels that match the keyframe images under each of the preset classification dimensions are initially selected; The keyframe image and the initially selected empty shot labels are input into a preset second model to determine the degree of label matching, and the label matching result is obtained. The label matching result includes the matching score between the keyframe image and the initially selected empty shot labels. Based on the tag matching results, the first empty shot tags under each preset classification dimension matched by the keyframe image are selected.

2. The method according to claim 1, characterized in that, The keyframe image and its corresponding audio data are input into a preset empty-scene corpus large language model to obtain the second empty-scene label under each preset classification dimension matched by the keyframe image, including: The keyframe image and its corresponding audio data are input into a preset empty-scene corpus large language model. The empty-scene corpus large language model is then questioned using a preset first question template, causing the empty-scene corpus large language model to output the second empty-scene label under each preset classification dimension matched by the keyframe image.

3. The method according to claim 1, characterized in that, The first and second empty shot labels matched by the keyframe images under each preset classification dimension are fused to obtain the final empty shot labels matched by the keyframe images under each preset classification dimension, including: For each preset classification dimension, the first empty shot label and the second empty shot label matched by the keyframe image under the preset classification dimension are input into the empty shot corpus large language model. The empty shot corpus large language model is then asked a question in combination with the preset second question template, so that the empty shot corpus large language model fuses the first empty shot label and the second empty shot label and outputs the final empty shot label matched by the keyframe image under the preset classification dimension.

4. The method according to claim 1, characterized in that, Based on the empty shot label recognition results, empty shot labels that match the keyframe images under each of the preset classification dimensions are initially selected, including: For each preset classification dimension, based on the empty shot label recognition results, the empty shot labels with the highest scores in the preset classification dimension are selected to obtain the initially selected empty shot labels that match the keyframe images in the preset classification dimension.

5. The method according to claim 1, characterized in that, Based on the tag matching results, the first empty shot tags under each preset classification dimension matched by the keyframe image are filtered out, including: For each preset classification dimension, based on the tag matching results, the empty shot tags with the highest matching scores in the preset classification dimension are selected to obtain the first empty shot tags for each preset classification dimension matched by the keyframe image.

6. A device for identifying empty mirror labels, characterized in that, The device includes: An extraction unit is used to extract keyframe images and the audio data corresponding to the keyframe images; The processing unit is used to process the keyframe image using a preset first model and a second model to obtain the first empty shot label of the keyframe image under each preset classification dimension matched. Multiple empty shot labels are set under each preset classification dimension. The first model and the second model are trained based on sample data. The first model and the second model are different models. The acquisition unit is used to input the keyframe image and its corresponding audio data into a preset empty shot corpus large language model to obtain the second empty shot label under each preset classification dimension matched by the keyframe image. The fusion unit is used to fuse the first empty shot label and the second empty shot label under each preset classification dimension matched by the keyframe image to obtain the final empty shot label under each preset classification dimension matched by the keyframe image. The recognition module is used to input the keyframe image into a preset first model to perform empty shot label recognition, and obtain the empty shot label recognition result corresponding to the keyframe image. The empty shot label recognition result includes: the label score of the empty shot label set under each preset classification dimension. The first filtering module is used to initially filter out empty shot labels that match the keyframe image under each preset classification dimension based on the empty shot label recognition results. The matching module is used to input the keyframe image and the initially selected empty shot labels into a preset second model to determine the degree of label matching and obtain the label matching result. The label matching result includes the matching score between the keyframe image and the initially selected empty shot labels. The second filtering module is used to filter out the first empty shot labels under each preset classification dimension matched by the keyframe image based on the label matching results.

7. The apparatus according to claim 6, characterized in that, The acquisition unit is specifically used to: input the keyframe image and its corresponding audio data into a preset empty-scene corpus large language model, and ask questions to the empty-scene corpus large language model in combination with a preset first question template, so that the empty-scene corpus large language model outputs the second empty-scene label under each preset classification dimension matched by the keyframe image.

8. The apparatus according to claim 6, characterized in that, The fusion unit is specifically used to: for each preset classification dimension, input the first empty shot label and the second empty shot label matched by the keyframe image under the preset classification dimension into the empty shot corpus large language model, and ask questions to the empty shot corpus large language model in combination with the preset second question template, so that the empty shot corpus large language model fuses the first empty shot label and the second empty shot label and outputs the final empty shot label matched by the keyframe image under the preset classification dimension.

Images