A method, device, equipment and medium for identifying a person
By combining convolutional neural network and VGG model and combining name lists in cast list to filter the recognition results, the problem of reduced recognition accuracy caused by 'face-like' in film and television works is solved, and the accuracy of character recognition is improved, especially in the long-term view Under the lens.
Patent Information
- Application Number
- CN202111227657.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2018-07-19
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2038-07-19
AI Technical Summary
In film and television works, due to the frequent "face-like" situations between actors, the accuracy of character recognition is reduced, especially when the character avatar is blurred under the long-range lens, misidentification and unrecognition are more common.
A character recognition method based on the cast is adopted, a convolutional neural network and a VGG model is combined to identify facial images, and the name list in the cast is used to filter the recognition results to improve the recognition accuracy. The method includes identifying cast lists in videos, text detection and optical character recognition, obtaining a list of characters' names, and combining image and text information for identity confirmation.
Effectively distinguishing actors with similar appearances improves the accuracy of character recognition and reduces the situation of misidentification and inability to recognize, especially when the character avatar is blurred under the long-range lens.
Smart Images

Figure CN113869281B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video image processing, and in particular to a method, device, equipment and medium for character recognition. Background Art
[0002] When identifying characters in film and television works, the usual practice is to compare the head portrait of the character appearing in the video frame with the photos in the sample set of the database, so as to identify the identity of the character and mark the character. However, with the increase in the number of people in the entertainment industry and the number of film and television works, actors often have "bumped faces". Since some actors look very similar and have similar styles, misidentification often occurs when identifying characters. For long-range shots, misidentification is more common, resulting in reduced accuracy in character recognition. Summary of the invention
[0003] The purpose of this application is to overcome the above problems or at least partially solve or alleviate the above problems.
[0004] According to one aspect of the present application, a method for character recognition based on a cast list is provided, comprising:
[0005] Person identification step: identifying a facial image of a person appearing in the video, determining the identity information of the person based on the facial image, and obtaining a first identity information set, wherein the first identity set includes at least one identity information;
[0006] Identity information filtering step: based on the list of character names in the cast of the video, the first identity information set is filtered to obtain a second identity information set.
[0007] Through this method, character portrait recognition and text recognition can be combined, and the information in the cast list can be used to assist in identifying the identity of the character, so that actors with similar appearances can be distinguished. In the case of blurred character portraits in long-range shots, the accuracy of recognition can be improved and the number of cases where no recognition or erroneous recognition is possible can be reduced.
[0008] Optionally, the actor list is obtained by the following steps:
[0009] A cast identification step: identifying a portion of a video frame in the video where a cast is located;
[0010] Actor content recognition step: performing text detection on the video frame portion to obtain screenshots with character name attributes for each video in the video frame portion, performing optical character recognition on the screenshots to obtain a list of character names appearing in the actor list.
[0011] This step can identify the cast list in the video. Even if the cast list cannot provide directly usable text information, a list of character names can be obtained based on the video frame image through text detection and optical character recognition. This method can obtain batches of character name data, greatly improving data processing capabilities.
[0012] Optionally, in the actor content recognition step: a target detection network model is used to perform text detection on the video frame portion to obtain attributes of the text in the video frame portion.
[0013] Optionally, in the character identification step, for each frame in the video, a facial image of a character appearing in the video frame is identified through a convolutional neural network; based on the facial image, the identity information and confidence of the character are determined through a trained VGG model to obtain a first identity information set, wherein the first identity set includes at least one identity information and the confidence of the identity information.
[0014] By combining the convolutional neural network and the VGG model, this method can fully utilize the advantages of the two models to recognize the facial images of people in pictures, and further identify the identities of people in facial images. In this way, it can identify people's images in the content-rich data of video frames and obtain identity information. Compared with manual recognition, this method is more efficient and convenient. For large videos, the processing speed is faster and the advantages are more obvious.
[0015] Optionally, the identity information filtering step includes: sorting the identity information in the first identity information set from high to low according to confidence, comparing the identity information in the first identity information set with the list of character names in turn, and if the identity information appears in the list of character names, using the identity information as an element in the second identity information set.
[0016] This step can use the actor information to filter and confirm the recognition results of the facial image, reducing the difficulty of improving the recognition accuracy of the model from the perspective of the graphic recognition algorithm. It can find ideas and solutions to the problem from a new perspective based on the characteristics of the complete video, thereby achieving the technical effect of improving the recognition accuracy.
[0017] According to another aspect of the present application, a character recognition device based on a cast list is also provided, comprising:
[0018] a person identification module configured to identify a facial image of a person appearing in a video, determine the identity information of the person based on the facial image, and obtain a first identity information set, wherein the first identity set includes at least one identity information; and
[0019] The identity information filtering module is configured to filter the first identity information set based on a list of character names in a cast list of the video to obtain a second identity information set.
[0020] Through this device, character portrait recognition and text recognition can be combined, and the information in the cast list can be used to assist in identifying the identity of the character, so that actors with similar appearances can be distinguished. In the case of blurred character portraits in long-range shots, the accuracy of recognition can be improved and the situations of unrecognition and erroneous recognition can be reduced.
[0021] Optionally, the actor list is obtained through the following modules:
[0022] a cast identification module configured to identify a portion of a video frame in the video where a cast is located;
[0023] The actor content recognition module is configured to perform text detection on the video frame portion, obtain screenshots with character name attributes for each video in the video frame portion, perform optical character recognition on the screenshots, and obtain a list of character names appearing in the actor list.
[0024] Optionally, the identity information filtering module is used to: sort the identity information in the first identity information set from high to low according to confidence, compare the identity information in the first identity information set with the list of character names in turn, and if the identity information appears in the list of character names, use the identity information as an element in the second identity information set.
[0025] According to another aspect of the present application, a computing device is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.
[0026] According to another aspect of the present application, a computer-readable storage medium is provided, preferably a non-volatile readable storage medium, which stores a computer program, and the computer program implements the method described above when executed by a processor.
[0027] Based on the detailed description of the specific embodiments of the present application in combination with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:
[0029] Figure 1 is a schematic flow chart of an embodiment of a method for character recognition based on a cast list according to the present application;
[0030] Figure 2 is a schematic flow chart of an embodiment of a cast list obtaining step according to the present application;
[0031] Figure 3 is a schematic flow chart of another embodiment of the step of obtaining a cast list according to the present application;
[0032] Figure 4 is a schematic block diagram of an embodiment of a character recognition device based on a cast list according to the present application;
[0033] Figure 5 is a schematic block diagram of an embodiment of a cast list obtaining module according to the present application;
[0034] Figure 6 is a schematic block diagram of another embodiment of a cast list obtaining module according to the present application;
[0035] Figure 7 is a block diagram of one embodiment of a computing device of the present application;
[0036] Figure 8 is a block diagram of one embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION
[0037] Based on the detailed description of the specific embodiments of the present application in combination with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application.
[0038] One embodiment of the present application provides a method for character recognition based on a cast list. Figure 1 is a schematic flow chart of an embodiment of a method for character recognition based on a cast list according to the present application. The method may include:
[0039] S100 person identification step: identifying a facial image of a person appearing in the video, determining the identity information of the person based on the facial image, and obtaining a first identity information set, wherein the first identity set includes at least one identity information;
[0040] S200 Identity information filtering step: Based on the list of character names in the cast of the video, the first identity information set is filtered to obtain a second identity information set.
[0041] Through this method, character portrait recognition and text recognition can be combined, and the information in the cast list can be used to assist in identifying the identity of the character, so that actors with similar appearances can be distinguished. In the case of blurred character portraits in long-range shots, the accuracy of recognition can be improved and the number of cases where no recognition or erroneous recognition is possible can be reduced.
[0042] The video of the present application includes data in storage formats related to various dynamic images, including but not limited to movies, TV series, documentaries, advertisements, variety shows, etc. The processing object of the present method is a complete video, for example, a complete movie or an episode of a TV series. The video includes a cast list, which refers to the actors appearing in a video, or a comparison table between actors and roles in the play. The cast list of a TV series or movie usually appears at the end of the credits.
[0043] Figure 2 is a schematic flow chart of an embodiment of the step of obtaining a cast list according to the present application. The method may further include a step of obtaining a cast list. In a video, the cast list may be obtained by the following step of obtaining a cast list, which may include:
[0044] S010 Actor identification step: identifying a portion of a video frame in the video where a cast is located;
[0045] S030 Actor content recognition step: performing text detection on the video frame portion to obtain screenshots with character name attributes for each video in the video frame portion, performing optical character recognition on the screenshots to obtain a list of character names appearing in the actor list.
[0046] This step can identify the cast list in the video. Even if the cast list cannot provide directly usable text information, a list of character names can be obtained based on the video frame image through text detection and optical character recognition. This method can obtain batches of character name data, greatly improving data processing capabilities.
[0047] Optionally, in the actor identification step, a deep learning network is used to identify a portion of a video frame where the actor list is located in the video to obtain a video frame sequence.
[0048] Optionally, the deep learning network can be a ResNeXt network, an Xception network, or a DenseNet network. Taking Xception as an example, the network model is a deep classifiable convolutional network that can classify scenes. For the two video frames before and after the video, binary classification training of the picture is performed. In the training stage, the part of the video with the cast is used as a positive example, and the corresponding non-cast part is used as a counter example. Each time a picture is input, classification training is performed according to the picture features and the label of the picture, and the output is the result of whether the picture is a positive example or a counter example. The training is stopped when the results of the test set used basically converge. In the use stage, the picture sequence composed of each video frame of the video to be analyzed is sequentially input into the deep learning network. The position of the discrimination result where a large number of continuous positive examples appear is the target position, that is, the part of the video frame where the cast is located, thereby obtaining a video frame sequence.
[0049] Through this method, the types of pictures in the video can be distinguished, and the cast part in the video can be found, so as to analyze the names of the characters from the cast.
[0050] Optionally, in the actor list content recognition step, a composite neural network may be used to perform text detection and obtain a list of character names, wherein the composite neural network may include a text detection network and a text recognition component.
[0051] The text detection network can be a YOLOv3 network. In the training stage, the annotated cast text information is used as training data. In the training data, all the characters in the cast are annotated, rather than just the names. In the use stage, the input of the text detection network is each video frame image in the video frame sequence, and the output is a screenshot of the interesting part of the video frame, thereby obtaining a screenshot set of the video frame sequence.
[0052] The text recognition component can be an optical character recognition (OCR) component, such as a Tesseract-OCR component. Taking Tesseract-OCR as an example, during training, the image is converted into a tif format to generate a box file. The screenshots in the tif format are corrected and trained using jTessBoxEditor. This step can optimize and adjust the names of the actors. When in use, a screenshot in the screenshot collection is input into the component, and the name of the person in the screenshot can be obtained, and then a list of the names of the characters appearing in the cast can be obtained.
[0053] Optionally, the actor list content recognition step further includes: performing deduplication processing on the character name list to obtain a deduplicated character name list. This step can avoid repeated information comparison when subsequently comparing the first identity information set with the character name list, thereby increasing the comparison speed.
[0054] Figure 3 It is a schematic flow chart of another embodiment of the cast obtaining step according to the present application. Optionally, before the cast content identification step, the cast obtaining step may also include S020 a video frame deduplication step: comparing the similarity between the two video frames before and after the video frame portion, and if the similarity is higher than a first threshold, deleting the latter video frame from the video frame portion. This step can delete redundant video frames after obtaining the cast, thereby reducing the amount of data processing in the cast content identification step. For some videos, the cast portion switches a screen at regular intervals, or scrolls at a slower speed. This step can remove redundant video frames and avoid repeated content identification of the same video frame.
[0055] Optionally, in the S100 character identification step, for each frame in the video, a facial image of a character appearing in the video frame is identified through a convolutional neural network; based on the facial image, the identity information and confidence of the character are determined through a trained VGG model to obtain a first identity information set, wherein the first identity set includes at least one identity information and the confidence of the identity information.
[0056] By combining the convolutional neural network and the VGG model, this method can fully utilize the advantages of the two models to recognize the facial images of people in pictures, and further identify the identities of people in facial images. In this way, it can identify people's images in the content-rich data of video frames and obtain identity information. Compared with manual recognition, this method is more efficient and convenient. For large videos, the processing speed is faster and the advantages are more obvious.
[0057] When using a convolutional neural network (CNN) to identify a person's facial image, a large number of pictures including the person's facial image can be obtained on the Internet based on the person's name. The facial image includes a front view and a side view, and these pictures form a training data set. In the training stage, first build a convolutional neural network, which includes multiple convolutional layers and multiple deconvolutional layers connected in sequence, wherein each convolutional layer is connected to a normalization operation and an excitation operation; initialize the weight value of the convolutional neural network; input the pictures of the pre-established training data set into the initialized convolutional neural network, and iteratively train the convolutional neural network with the goal of minimizing the cost function. The model output is a facial image of the person on the picture, and the facial image is a screenshot of the picture, wherein the weight value is updated once for each iteration until the model converges. In the use stage, a trained convolutional neural network is obtained; each frame in the video is input into the trained convolutional neural network, and the corresponding facial image and position information of the person are output.
[0058] Based on the facial image, the identity information and confidence of the person are determined by the trained VGG model to obtain a first identity information set, wherein the first identity set includes at least one identity information and the confidence of the identity information. During the training phase, face image data of more than 1,000 people are used as training data, with no less than 100 pictures of each person, including multiple angles from the front to the side. The VGG model training results should meet the average accuracy mAP>0.94 for the test set of target video screenshots. It is understandable that models such as VGG can be used for training, or existing face recognition tools can be used for recognition.
[0059] Optionally, the S200 identity information filtering step includes: sorting the identity information in the first identity information set from high to low according to confidence, comparing the identity information in the first identity information set with the list of personal names in turn, and if the identity information appears in the list of personal names, using the identity information as an element in the second identity information set.
[0060] This step can use the actor information to filter and confirm the recognition results of the facial image, reducing the difficulty of improving the recognition accuracy of the model from the perspective of the graphic recognition algorithm. It can find ideas and solutions to the problem from a new perspective based on the characteristics of the complete video, thereby achieving the technical effect of improving the recognition accuracy.
[0061] Optionally, in the case where the identity information does not appear in the character name list, the second identity information is an empty set, indicating that there is no correct recognition result.
[0062] Optionally, when the identity information does not appear in the character name list, identity information in the first identity information set whose confidence level is greater than a second threshold is used as an element in the second identity information set.
[0063] Optionally, when the highest confidence level in the identity information in the first identity information set is less than a second threshold, the second identity information is an empty set, indicating that the identity of the person is not recognized.
[0064] In an optional embodiment, after the identity information filtering step in step S200, the method further includes: a secondary identity information filtering step: based on a second character name list obtained by detecting the audio corresponding to the video, the second identity information set is filtered using the second character name list to obtain a third identity information set.
[0065] Through this method, character portrait recognition, cast list and audio information of the video can be combined, and the name information obtained by voice recognition can be used to assist in identifying the identity of the character, so that actors with similar appearances can be distinguished. In the case of blurred character portraits in long-range shots, the accuracy of recognition can be improved and the number of cases where no recognition or misrecognition occurs can be reduced.
[0066] The second person name list is obtained by the following steps:
[0067] Video speech recognition step: based on the speech word library, performing speech recognition on the audio corresponding to the video to obtain speech recognition text;
[0068] Text detection step: Based on the character name vocabulary, the speech recognition text is detected to obtain a list of character names appearing in the speech recognition text.
[0069] This step can identify the names of people mentioned in the video voice through natural language processing. Even if the video cannot provide directly usable information about the names of people appearing in the video, a list of names of people can be obtained through voice recognition based on the audio of the video, thereby assisting image recognition and greatly improving the recognition accuracy.
[0070] Optionally, in the video speech recognition step, speech recognition can be implemented based on a speech recognition engine, and the speech recognition engine performs speech recognition on the audio corresponding to the video to obtain a speech recognition text. The speech vocabulary of the speech recognition engine can be defined by itself, and the speech vocabulary includes the names of characters and corresponding audio features. Different speech vocabulary can be defined for the type of video to be processed, for example, a competition vocabulary defined for sports competitions, which includes competition terms, athlete names, etc.; a variety show vocabulary defined for variety shows, which includes celebrity names, host names, etc. This step uses a speech recognition engine optimized for character names to improve the accuracy of character name recognition in audio and reduce the probability that names are easily misrecognized.
[0071] Optionally, the character name word library may include one or more of the following data: a character's role name, real name, stage name, English name, former name, etc.
[0072] In an optional embodiment, in the text detection step, based on the character name word library, the character names appearing in the speech recognition text are extracted, and a second character name list is obtained after deduplication processing. Through this method, the names of all characters mentioned in the audio can be quickly obtained, and the method is simple to implement and highly efficient.
[0073] In another optional implementation, in the text detection step, all character names in the speech recognition text are labeled based on a character name vocabulary, and semantic analysis is performed on the character names and the sentiment words nearby to obtain the names of the characters appearing in the video, and a second character name list is obtained after deduplication processing.
[0074] This step can mark and locate the names in the speech recognition text through the character name word library; semantic analysis of the character name and the words around it can detect whether the meaning of the sentence describes the characters appearing in the video, or the characters not appearing in the video but related to the content of the video, for example, characters similar to the characters in the video, or characters in currently popular events, etc. If the video to be processed is a variety show, sports event, etc., and the real names of the characters appear in the audio, then the second character name list is obtained directly based on the character names appearing in the audio. If the video to be processed is a movie, TV series, etc., and the character names appear in the audio, it is necessary to use the character name word library to map the character names to the base words, that is, the real names, and then the second character name list is obtained based on the real names.
[0075] Wherein, the text detection step may include:
[0076] Steps for establishing a character name word library: establish a character name set for each character, the character name set includes: role name, real name, stage name, English name, former name, and characters who look similar to the character, with the real name as the base word and other names as similar words;
[0077] Speech recognition processing steps: Perform speech recognition on the audio to identify the person’s name and related emotional words;
[0078] Semantic analysis steps: cluster analysis, identify semantically related character names and sentiment words, and make semantic judgments;
[0079] Analysis result output step: obtain the names of the characters appearing in the video, and obtain a list of character names after deduplication processing.
[0080] The speech recognition processing step may include:
[0081] Text conversion step: Use the speech recognition engine to perform speech recognition on the audio and convert it into text;
[0082] Text processing steps: complete natural language processing such as word segmentation, part-of-speech tagging, basic semantic recognition, and save to the database;
[0083] Labeling step: Identify and label the character names and related sentiment words.
[0084] Wherein, the semantic analysis step may include:
[0085] Cluster analysis steps: Perform cluster analysis to identify semantically related character names and sentiment words;
[0086] Semantic analysis step: Calculate the difference between the distance vectors of the character's name. If it is less than the preset value, perform a measurement calculation based on the word order of the character's name and the sentiment word to make a semantic judgment.
[0087] Optionally, the deduplication process in the step of obtaining the character name list may include:
[0088] Remove duplicate names;
[0089] The type of the name is determined based on the character name vocabulary. If it is a homonymous word, the name is replaced with a base word of the homonymous word to obtain a character name list.
[0090] This step can avoid repeated information comparison when subsequently comparing the first identity information set with the character name list, thereby increasing the speed of comparison.
[0091] Optionally, the identity information secondary filtering step may include: comparing the identity information of the second identity information set with the second person name list in turn, and if the identity information appears in the second person name list, using the identity information as an element in the third identity information set. This step can filter and confirm the recognition results of the face image using the video voice information, reducing the difficulty of improving the model recognition accuracy from the perspective of the graphic recognition algorithm, and can find ideas and solutions to solve the problem from a completely new perspective based on the characteristics of the complete video, thereby achieving the technical effect of improving the recognition accuracy.
[0092] An embodiment of the present application also provides a character recognition device based on a cast list. Figure 4 is a schematic block diagram of an embodiment of a character recognition device based on a cast list according to the present application. The device may include:
[0093] A person identification module 100 is configured to identify a facial image of a person appearing in a video, determine the identity information of the person based on the facial image, and obtain a first identity information set, wherein the first identity set includes at least one identity information;
[0094] The identity information filtering module 200 is configured to filter the first identity information set based on the character name list in the actor list of the video to obtain a second identity information set.
[0095] Through this device, character portrait recognition and text recognition can be combined, and the information in the cast list can be used to assist in identifying the identity of the character, so that actors with similar appearances can be distinguished. In the case of blurred character portraits in long-range shots, the accuracy of recognition can be improved and the situations of unrecognition and erroneous recognition can be reduced.
[0096] Optionally, the device further comprises a cast obtaining module, and the cast is obtained by the cast obtaining module. Figure 5 is a schematic block diagram of an embodiment of a cast list acquisition module according to the present application. The cast list acquisition module may include:
[0097] A cast identification module 010 configured to identify a portion of a video frame in the video where a cast is located;
[0098] The cast content recognition module 030 is configured to perform text detection on the video frame portion, obtain screenshots with character name attributes for each video in the video frame portion, perform optical character recognition on the screenshots, and obtain a list of character names appearing in the cast.
[0099] Figure 6 020 is a schematic block diagram of another embodiment of the actor list acquisition module according to the present application. Optionally, the actor list acquisition module may also include a video frame deduplication module 020, which is configured to compare the similarity between the two video frames before and after the video frame portion, and if the similarity is higher than a first threshold, the latter video frame is deleted from the video frame portion. After obtaining the actor list, the module can delete redundant video frames to reduce the data processing amount of the actor list content recognition step.
[0100] Optionally, the cast identification module is used to use a deep learning network to identify a portion of a video frame in which a cast is located in the video to obtain a video frame sequence.
[0101] Optionally, the actor content recognition module is used to perform text detection on the video frame portion using an object detection network model to obtain the properties of the text in the video frame portion. Optionally, the actor content recognition module is used to perform text detection using a composite neural network and obtain a list of character names. The composite neural network may include a text detection network and a text recognition component.
[0102] Optionally, the person identification module 100 is used to identify the facial image of a person appearing in the video frame for each frame in the video through a convolutional neural network; determine the identity information and confidence of the person based on the facial image through a trained VGG model to obtain a first identity information set, wherein the first identity set includes at least one identity information and the confidence of the identity information.
[0103] Optionally, the identity information filtering module 200 is used to: sort the identity information in the first identity information set from high to low according to confidence, compare the identity information in the first identity information set with the list of personal names in turn, and if the identity information appears in the list of personal names, use the identity information as an element in the second identity information set.
[0104] In an optional embodiment, after the identity information filtering module step, the device also includes: an identity information secondary filtering module, which is configured to filter the second identity information set using the second character name list based on a second character name list obtained by detecting the audio corresponding to the video to obtain a third identity information set.
[0105] Optionally, the device may further include a module for obtaining a second character name list. The module for obtaining the second character name list may include:
[0106] A video speech recognition module, configured to perform speech recognition on the audio corresponding to the video based on a speech vocabulary to obtain a speech recognition text;
[0107] The text detection module is configured to detect the speech recognition text based on a character name vocabulary to obtain a list of character names appearing in the speech recognition text.
[0108] Optionally, the character name database may include one or more of the following data: the character's real name, stage name, English name, former name, etc.
[0109] In an optional implementation, the text detection module is used to extract the names of the characters appearing in the speech recognition text based on a character name vocabulary, and obtain a character name list after deduplication processing.
[0110] In another optional implementation, the text detection module is used to label all character names in the speech recognition text based on a character name vocabulary, perform semantic analysis on the character names and nearby sentiment words, obtain the names of the characters appearing in the video, and obtain a character name list after deduplication processing.
[0111] The text detection module may include:
[0112] Character name word library building module: used to build a character name set for each character, which may include: real name, stage name, English name, former name, and characters who look similar to the character, with the real name as the base word and other names as similar words;
[0113] Speech recognition processing module: used to perform speech recognition on audio, identify the person's name and related emotional words;
[0114] Semantic analysis module: used to perform cluster analysis, identify semantically related character names and sentiment words, and make semantic judgments;
[0115] Analysis result output module: used to obtain the names of the characters appearing in the video, and obtain a list of character names after deduplication processing.
[0116] Among them, the speech recognition processing module may include:
[0117] Text conversion module: used to use the speech recognition engine to perform speech recognition on the audio and convert it into text;
[0118] Text processing module: used to complete natural language processing such as word segmentation, part-of-speech tagging, basic semantic recognition, etc., and save them to the database;
[0119] Step module: used to identify and label character names and related sentiment words.
[0120] Wherein, the semantic analysis module may include:
[0121] Cluster analysis module: used to perform cluster analysis and identify semantically related person names and sentiment words;
[0122] Semantic analysis module: used to calculate the difference between the distance vectors of the character's name. If it is less than the preset value, a measurement calculation is performed based on the word order of the character's name and the sentiment word to make a semantic judgment.
[0123] Optionally, the de-duplication processing in the character name list acquisition module may include: removing duplicate names, determining the type of the name based on the character name vocabulary, and if it is a similar word, replacing the name with a base word of the similar word to obtain a second character name list.
[0124] The embodiment of the present application also provides a computing device, referring to Figure 7 The computing device includes a memory 1120, a processor 1110, and a computer program stored in the memory 1120 and executable by the processor 1110. The computer program is stored in a space 1130 for program code in the memory 1120. When the computer program is executed by the processor 1110, it is implemented to execute any one of the method steps 1131 according to the present invention.
[0125] The embodiments of the present application also provide a computer-readable storage medium. Figure 8The computer-readable storage medium comprises a storage unit for program code, the storage unit being provided with a program 1131' for executing the method steps according to the present invention, the program being executed by a processor.
[0126] The present application embodiment also provides a computer program product containing instructions, including computer-readable codes, which, when executed by a computing device, cause the computing device to execute the method as described above.
[0127] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer loads and executes the computer program instructions, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0128] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0129] A person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing a processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state drive, a magnetic tape, a floppy disk, an optical disc, and any combination thereof.
[0130] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for character recognition based on a cast, comprising: Character identification step: identifying the facial image of the character appearing in the video, determining the identity information of the character based on the facial image, and obtaining a first identity information set, wherein the first identity information set includes at least one identity information. In the character identification step, for each frame in the video, the facial image of the character appearing in the video frame is identified by a convolutional neural network; based on the facial image, the identity information and confidence of the character are determined by a trained VGG model to obtain a first identity information set, wherein the first identity information set includes at least one identity information and the confidence of the identity information; An identity information filtering step: filtering the first identity information set based on a list of character names in a cast list of the video to obtain a second identity information set; and The second identity information filtering step includes: detecting the audio corresponding to the video, identifying the names of the characters mentioned in the video voice, obtaining a second character name list, and filtering the second identity information set using the second character name list to obtain a third identity information set; In the case that the identity information does not appear in the character name list, taking the identity information in the first identity information set whose confidence level is greater than a second threshold as an element in the second identity information set; The cast is obtained by the following steps: A cast identification step: identifying a portion of a video frame in the video where a cast is located; and A cast content recognition step: performing text detection on the video frame portion to obtain a screenshot with a character name attribute for each video in the video frame portion, performing optical character recognition on the screenshot to obtain a list of character names appearing in the cast; The second list of person names is obtained by following the steps below: Video speech recognition step: based on the speech word library, performing speech recognition on the audio corresponding to the video to obtain speech recognition text; Text detection step: Based on the character name vocabulary, the speech recognition text is detected to obtain a list of character names appearing in the speech recognition text.
2. The method according to claim 1, characterized in that: In the actor content recognition step: using a target detection network model to perform text detection on the video frame portion to obtain attributes of the text in the video frame portion.
3. The method according to claim 1 or 2, characterized in that: The identity information filtering step includes: sorting the identity information in the first identity information set from high to low according to confidence, comparing the identity information in the first identity information set with the character name list in turn, and if the identity information appears in the character name list, using the identity information as an element in the second identity information set.
4. A character recognition device based on a cast list, comprising: A person identification module, configured to identify a facial image of a person appearing in a video, determine the identity information of the person based on the facial image, and obtain a first identity information set, wherein the first identity information set includes at least one identity information, and the person identification module is used to identify the facial image of the person appearing in the video frame through a convolutional neural network for each frame in the video; determine the identity information and confidence of the person based on the facial image through a trained VGG model, and obtain a first identity information set, wherein the first identity information set includes at least one identity information and the confidence of the identity information; an identity information filtering module configured to filter the first identity information set based on a list of character names in a cast of the video to obtain a second identity information set; and an identity information secondary filtering module, configured to detect the audio corresponding to the video, identify the names of the characters mentioned in the video voice, obtain a second character name list, and use the second character name list to filter the second identity information set to obtain a third identity information set; In the case that the identity information does not appear in the character name list, taking the identity information in the first identity information set whose confidence level is greater than a second threshold as an element in the second identity information set; The cast is obtained via the following modules: a cast identification module configured to identify a portion of a video frame of the video where a cast is located; and a cast content recognition module configured to perform text detection on the video frame portion, obtain a screenshot with character name attributes for each video in the video frame portion, perform optical character recognition on the screenshot, and obtain a list of character names appearing in the cast; The module further comprises: a module for obtaining a second character name list, wherein obtaining the second character name list comprises: A video speech recognition module, configured to perform speech recognition on the audio corresponding to the video based on a speech vocabulary to obtain a speech recognition text; The text detection module is configured to detect the speech recognition text based on a character name vocabulary to obtain a list of character names appearing in the speech recognition text.
5. The device according to claim 4, characterized in that The identity information filtering module is used to: sort the identity information in the first identity information set from high to low according to confidence, compare the identity information in the first identity information set with the character name list in turn, and if the identity information appears in the character name list, use the identity information as an element in the second identity information set.
6. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.
7. A computer-readable storage medium, which is a non-volatile readable storage medium, storing a computer program therein, wherein the computer program implements the method according to any one of claims 1 to 3 when executed by a processor.
