Fusion movie and television play content retrieval method and device based on large language model, face recognition, target detection and cross-modal vector, medium and product

Abstract

The invention discloses a fusion movie and television play content retrieval method based on a large language model, face recognition, target detection and a cross-modal vector. The method comprises the steps that videos are preprocessed and stored; performing word segmentation and optimization processing by using a large language model according to a retrieval text of the video content input by a user; performing OCR (optical character recognition) on the staff table of the video content and matching a character role association relationship; performing target detection on the object; for scenes and event contents involved in user retrieval, video frames are converted into vector representation through cross-modal mapping established by a CLIP model, the vector representation is compared with vector representation of a query text, and the similarity between the video frames and the query text is calculated; and finally, carrying out secondary sorting through the hit number and the final weighted score. The invention further discloses electronic equipment, a medium and a product. According to the method, the movie and television play media video material content retrieval capability is improved, comprehensive judgment is carried out through a fusion algorithm, and a high-quality video retrieval scheme is obtained.

Description

Technical Field

[0001] The present invention relates to video retrieval technology, and more specifically, to a method, device, medium and product for retrieval of integrated film and television drama content based on a large language model, face recognition, target detection and cross-modal vectors. Background Art

[0002] With the development of Internet technology and the increasing richness of media content, video, as an important carrier of information dissemination, has been widely used in the field. However, faced with massive video data, how to efficiently and accurately retrieve the information required by users has become an urgent problem to be solved. Traditional video retrieval methods mostly rely on keyword matching or simple visual feature comparison, which is not only inefficient, but also difficult to meet users' requirements for the quality of retrieval results.

[0003] With the development of artificial intelligence, OCR target detection and face recognition technologies are becoming more mature and achieving higher and higher accuracy in various applications; at the same time, cross-modal vector feature retrieval has also been innovatively applied to content retrieval; at the same time, large language models can better optimize and expand user retrieval text and user retrieval interaction. Summary of the invention

[0004] In view of the defects existing in the prior art, the purpose of the present invention is to provide a fusion film and television content retrieval method, equipment, medium and product based on a large language model, face recognition, target detection and cross-modal vector, so as to improve the content retrieval capability of film and television media video materials, focusing on using a large language model to classify user search content, using OCR target detection and face recognition technology to perform target detection on people, objects and other aspects in the material content, and at the same time performing vector detection on content scenes, events, etc., and making comprehensive judgments through fusion algorithms to obtain high-quality video retrieval solutions.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] The present invention provides a fusion film and television drama content retrieval method based on a large language model, face recognition, target detection and cross-modal vector;

[0007] Video preprocessing and storage;

[0008] Based on the search text of the video content input by the user, the large language model is used to perform word segmentation and optimization processing;

[0009] Then, the cast and crew of the video content are identified by OCR and the character relationships are matched;

[0010] Perform target detection on objects;

[0011] For the scenes and events involved in user retrieval, the cross-modal mapping established by the CLIP model is used to convert the video frames into vector representations, and then compared with the vector representations of the query text to calculate the similarity between the video frames and the query text.

[0012] Finally, secondary sorting is performed based on the number of hits and the final weighted score.

[0013] Preferably, the video preprocessing and storage specifically includes the following steps:

[0014] S1, video frame extraction and time recording;

[0015] S2, object detection and classification;

[0016] S3, credits information processing;

[0017] S4, face recognition detection;

[0018] S5, cross-modal vectorization processing.

[0019] Preferably, the step S1 specifically includes:

[0020] Extract image frames from the video stream at set time intervals and record the timestamp corresponding to each frame. The timestamp information will be used for subsequent precise positioning and analysis; and / or

[0021] The step S2 specifically includes:

[0022] Perform object detection on each frame using an object detection algorithm. Once an object is detected in the image, a pre-trained deep learning model is further applied to perform fine-grained object classification. The classification results are annotated on the original image and stored for subsequent use; and / or

[0023] The step S3 specifically includes:

[0024] Using OCR to identify credit information displayed at the beginning or end of the video; and / or

[0025] Combine named entity recognition in natural language processing to identify people's names, and correctly match actors and role names through regular expression matching;

[0026] The credits information will be stored in a structured manner to facilitate quick subsequent lookup and reference; and / or

[0027] The step S4 specifically includes:

[0028] For each frame of the image, a pre-built face database and a trained face recognition model are used to identify the person appearing in the video;

[0029] The facial recognition model extracts facial features and compares them with known person features in a database to determine the person's identity and associate it with the corresponding role; and / or

[0030] The step S5 specifically includes:

[0031] The pre-trained CLIP model is used to extract cross-modal features from video frames and convert images into high-dimensional feature vectors.

[0032] Preferably, based on the search text of the video content input by the user, a large language model is used to perform word segmentation and optimization processing, specifically including:

[0033] When a user makes a video retrieval request, the user's input text is first preprocessed using a large language model to extract key words;

[0034] And then through multiple rounds of dialogue interaction with the user, we can more accurately understand the user's search intention.

[0035] Preferably, the Elasticsearch search engine is used to match the pre-processed object tags according to the user's query keywords, return a list of objects related to the query, and give a confidence score for each object;

[0036] Using face recognition technology, the user can find the closest match in the face database based on the description provided by the user or the photo uploaded directly, and also provide a confidence score for each character;

[0037] The CLIP model is used to convert the text input by the user into a feature vector, which is then compared with the image feature vector generated in the preprocessing stage to find the most similar scenes or events, which also come with a similarity score.

[0038] Preferably, secondary sorting is performed based on the number of hits and the final weighted score, specifically including:

[0039] For each video segment, a preliminary score is obtained by weighted average based on the matching degree of the three dimensions of objects, people, and scene events;

[0040] Additional points are awarded based on the number of consecutive matching frames;

[0041] All video segments are sorted from high to low according to their total scores, and the results that best meet the query requirements are presented to users.

[0042] The present invention further provides an electronic device, comprising:

[0043] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the integrated film and television drama content retrieval method based on a large language model, face recognition, target detection and cross-modal vector of the present invention.

[0044] The present invention also provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement the present invention's integrated film and television content retrieval method based on a large language model, face recognition, target detection and cross-modal vectors when executed.

[0045] The present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the integrated film and television drama content retrieval method based on a large language model, face recognition, target detection and cross-modal vector according to the present invention.

[0046] The present invention provides a method, device, medium and product for retrieval of integrated film and television drama content based on a large language model, face recognition, target detection and cross-modal vector, which has the following beneficial effects:

[0047] (1) Use a large language model to perform text processing such as escape and word segmentation on user text retrieval, optimize user retrieval content through multiple rounds of interaction, clarify key retrieval information, and improve the accuracy and efficiency of subsequent retrieval;

[0048] (2) For character retrieval in film and television drama content, ordinary face recognition cannot locate the role played by the actor. The present invention uses OCR technology to identify the information of the cast and crew in the video, and clarifies the relationship between the actual actor and the plot role, so as to achieve retrieval and location of the role;

[0049] (3) Compared with other traditional searches, the accuracy of video vector retrieval based on semantic features is relatively low, and the accuracy of the results is unstable due to the influence of models, etc. The present invention can greatly improve the accuracy of video content retrieval results by weighting factors with high hit rates such as face and object recognition, and performing secondary sorting based on the number of consecutive hits. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the process of video preprocessing in the method for retrieval of integrated film and television drama content of the present invention;

[0051] Figure 2 It is a schematic diagram of the user search process in the integrated film and television drama content search method of the present invention;

[0052] Figure 3 It is a schematic diagram of the structural framework of the electronic equipment of the present invention. DETAILED DESCRIPTION

[0053] In order to better understand the above technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0054] The present invention provides a fusion film and television drama content retrieval method based on a large language model, face recognition, target detection and cross-modal vector;

[0055] Video preprocessing and storage;

[0056] Based on the search text of the video content entered by the user, the large language model is used for word segmentation and optimization. At the same time, the user's search intention can be clearly located through multiple rounds of interactive dialogues.

[0057] Then, the video content’s cast and crew are identified by OCR and the character relationships are matched. For character retrieval, high-precision face detection and recognition are achieved using methods based on deep learning models and convolutional neural networks (CNNs).

[0058] Perform target detection on objects;

[0059] For scenes, events, and other contents involved in user retrieval, the cross-modal mapping established by the CLIP model is used to convert video frames into vector representations, and then compared with the vector representations of the query text to calculate the similarity between the video frames and the query text.

[0060] Finally, secondary sorting is performed based on the number of hits and the final weighted score.

[0061] The present invention searches for videos by integrating search methods of different dimensions, adopts relatively better search methods for different objects, and obtains more accurate video search results through fusion.

[0062] Combination Figure 1 and Figure 2 As shown, the integrated film and television drama content retrieval method of the present invention mainly includes two parts: video preprocessing storage and user retrieval, which are as follows:

[0063] 1. Video preprocessing and storage specifically include the following steps:

[0064] S1, video frame extraction and time recording

[0065] For video materials that need to be stored, offline preprocessing is first performed. This process includes extracting frames from the video, that is, extracting image frames from the video stream at a certain time interval (for example, one frame per second) and recording the timestamp corresponding to each frame. These timestamp information will be used for subsequent precise positioning and analysis.

[0066] S2, object detection and classification

[0067] Perform object detection on each frame of the image. Here, you can choose to use an efficient object detection algorithm such as YOLO. Once the objects in the image are detected, a pre-trained deep learning model such as ResNet will be further applied to perform fine-grained object classification. The classified results will be annotated on the original image and stored for subsequent use.

[0068] S3, credits information processing

[0069] Use OCR technology to identify the credits information displayed at the beginning or end of the video. In order to improve the recognition accuracy, it can also combine the named entity recognition (NER) technology in natural language processing to accurately identify the names of people, and correctly match the actors and role names through regular expression matching and other methods. This information will be stored in a structured manner for quick search and reference later.

[0070] S4, face recognition detection

[0071] For each frame, a pre-built face database and a trained face recognition model (such as a convolutional neural network (CNN)-based method) are used to identify the person appearing in the video. The model extracts facial features and compares them with known person features in the database to determine the person's identity and associate it with the corresponding role.

[0072] S5, cross-modal vectorization processing

[0073] The pre-trained CLIP model is used to extract cross-modal features from video frames and convert images into high-dimensional feature vectors. This process enables images and text to be represented in the same vector space, thus supporting cross-modal data retrieval and matching tasks.

[0074] Second, user search specifically includes:

[0075] 1) User input processing

[0076] When a user makes a video search request, the system first uses a large language model (LLM) to pre-process the user's input text, including word segmentation and part-of-speech tagging, to extract key words. The system also interacts with the user through multiple rounds of dialogue to more accurately understand the user's search intent, and provides similar recommendations when necessary to help the user refine the query conditions.

[0077] 2) Object Retrieval

[0078] a. Object retrieval: Use the Elasticsearch search engine to match the pre-processed object tags according to the user's query keywords, return a list of objects related to the query, and give a confidence score for each object.

[0079] b. Person retrieval: Using face recognition technology, the closest matching result is found in the face library based on the description provided by the user or the photo uploaded directly, and a confidence score is also provided for each person.

[0080] c. Scene event retrieval: The CLIP model is used to convert the text input by the user into a feature vector, which is then compared with the image feature vector generated in the preprocessing stage to find the most similar scene or event, which also comes with a similarity score.

[0081] 3) Result integration and sorting

[0082] Finally, the system will integrate the search results from different modules and calculate the comprehensive score of each video clip according to the preset weights. Specifically, for each video segment, the system will consider the matching degree of the three dimensions of objects, people, and scene events, and obtain a preliminary score by weighted average; then, additional points will be given according to the number of consecutive matching frames; finally, all video segments are sorted from high to low according to the total score, and the results that best meet the query requirements are presented to the user.

[0083] Combination Figure 3 As shown, the present invention also provides an electronic device, which is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or required herein. The electronic device includes:

[0084] At least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0085] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0086] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the integrated film and television drama content retrieval method.

[0087] The technical and economic quantitative evaluation method of the present invention can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the technical and economic quantitative evaluation method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the fusion film and television drama content retrieval method in any other appropriate manner (for example, by means of firmware).

[0088] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0089] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0090] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0091] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0092] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0093] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0094] Example

[0095] Refer to Figure 1 and Figure 2 As shown, the integrated film and television drama content retrieval method of this embodiment mainly includes two parts: video pre-processing storage and user retrieval, which are as follows:

[0096] 1. Video Preprocessing

[0097] The materials that need to be retrieved are pre-processed offline and stored in the database so that they can be quickly matched and calculated when users perform real-time retrieval later.

[0098] First, extract frames from the video as needed and record the timestamp of each frame for subsequent analysis.

[0099] The preprocessing of video content is divided into the following parallel processing steps:

[0100] S1, perform target detection on each frame of the image. You can use advanced object detection models such as YOLO to detect objects in each frame. For the detected objects, use pre-trained classification models such as ResNet to further refine the object category and annotate and record the objects detected in each frame.

[0101] S2, process the cast and crew information in the video frame. First, use OCR technology to identify the key frames containing the cast and crew information features from the end or beginning of the video. Usually, the names of the actors and characters appear in the same line. Use NER technology to identify the names of people in the text, and use regular expressions or other text matching technologies to identify the names of the actors and characters. Pair the identified actor names with their corresponding roles, and structure the paired information for subsequent processing.

[0102] S3 performs face recognition detection on each frame of the image at the same time. A large number of annotated face images of actors are collected in the early stage as face library data. The pre-trained face recognition model is used to learn the representation of the face using the convolutional neural network CNN. The face matching result is obtained by face matching with the facial feature information in the actor face library, and the images where the characters appear are annotated and recorded.

[0103] S3 performs cross-modal vectorization on the video frames. The pre-trained CLIP cross-modal large model is used to convert the images into cross-modal feature vectors. The CLIP model and its related vectorization processing steps can effectively map images and texts into the same vector space, facilitating subsequent cross-modal data retrieval and matching.

[0104] 2. User Search

[0105] The user inputs the video retrieval text, and the large language model LLM is used to segment the user's text retrieval, extract and classify the characters, objects, scene events, etc. in the user's retrieval, and provide users with multiple rounds of interactions and approximate recommendations to clarify and refine various key information of the retrieval, thereby achieving quality optimization of the user's retrieval intention.

[0106] Next, different types of objects are retrieved using different technical methods:

[0107] (1) For object type searches, Elasticsearch is used for retrieval. The query results are sorted based on the relevance score. The query result relevance score is normalized to obtain confidence indicators a1, a2, etc. in the [0,1] interval. When there are multiple object objects, a list containing multiple objects is output, and each object has a corresponding confidence level. A certain threshold is set to ensure the validity of the hit result.

[0108] (2) For the search of people, a pre-trained face recognition model is used for query. The query results are sorted according to the similarity calculation results between the face to be identified and the most similar feature vector in the face feature library. The similarity scores of the query results are normalized to obtain confidence indicators b1, b2, etc. in the interval [0,1]. The corresponding identity information is output and associated with the corresponding role information. When there are multiple face objects, a list containing multiple objects is output, each with a corresponding confidence level. A certain threshold is set to ensure the validity of the hit result.

[0109] (3) For scene event search, the pre-trained CLIP cross-modal large model is used for query. The query text is input into the text encoder to obtain its embedding vector, and the similarity between the image embedding vector and the text embedding vector is calculated. The images are sorted according to the similarity, and the similarity scores of the query results are normalized to obtain confidence indicators c1, c2, etc. in the interval [0, 1]. A certain threshold is set to ensure the validity of the hit result.

[0110] Perform final fusion calculation on relevant indicators.

[0111] (1) Initialization indicators: Each video contains multiple frames, and each frame has indicators in three dimensions: a, b, and c.

[0112] (2) Weighted average calculation: Calculate the weighted average index M of each frame according to the preset weight.

[0113] (3) Calculate the number of consecutive hits: Calculate the number of consecutive hit frames in each video and record it as N.

[0114] (4) Calculation of the overall index of the video: Calculate the weighted average index of all hit frames in each video, and use hit_weight and consecutive_weight to adjust the weights of the number of hits and the number of consecutive hits. The formula is as follows:

[0115] metric=(M×total_hit_count×hit_weight)+(∑(N×consecutive_weight))

[0116] In the formula, metric represents the calculation result of the final confidence of a video content;

[0117] total_hit_count indicates the total number of hits for the video in multiple dimensions;

[0118] hit_weight represents the weight distribution of individual frame hits;

[0119] consecutive_weight indicates the weight distribution of consecutive frame hits.

[0120] (5) Sorting videos: Sort videos based on their overall metrics. Videos that hit multiple frames in a row will be ranked first.

[0121] This method can effectively evaluate the hits of multiple frames in the video and make the video with consecutive hits have higher overall indicators. This not only considers the indicators of a single frame, but also the impact of the number of consecutive hits, thereby improving the rationality of the sorting.

[0122] Those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. As long as they are within the spirit of the present invention, any changes or modifications to the above embodiments will fall within the scope of the claims of the present invention.

Claims

1. A fusion film and television drama content retrieval method based on a large language model, face recognition, object detection and cross-modal vector, characterized by: Video preprocessing and storage; Based on the search text of the video content input by the user, the large language model is used to perform word segmentation and optimization processing; Then, the cast and crew of the video content are identified by OCR and the character relationships are matched; Perform target detection on objects; For the scenes and events involved in user retrieval, the cross-modal mapping established by the CLIP model is used to convert the video frames into vector representations, and then compared with the vector representations of the query text to calculate the similarity between the video frames and the query text. Finally, secondary sorting is performed based on the number of hits and the final weighted score.

2. The method for retrieval of integrated film and television drama content based on large language model, face recognition, target detection and cross-modal vector according to claim 1 is characterized in that: The video preprocessing and storage specifically includes the following steps: S1, video frame extraction and time recording; S2, object detection and classification; S3, credits information processing; S4, face recognition detection; S5, cross-modal vectorization processing.

3. The method for retrieval of integrated film and television drama content based on large language model, face recognition, target detection and cross-modal vector according to claim 2 is characterized in that: The step S1 specifically includes: Extract image frames from the video stream at set time intervals and record the timestamp corresponding to each frame. The timestamp information will be used for subsequent precise positioning and analysis; and / or The step S2 specifically includes: Perform object detection on each frame using an object detection algorithm. Once an object is detected in the image, a pre-trained deep learning model is further applied to perform fine-grained object classification. The classification results are annotated on the original image and stored for subsequent use; and / or The step S3 specifically includes: Using OCR to identify credit information displayed at the beginning or end of the video; and / or Combine named entity recognition in natural language processing to identify people's names, and correctly match actors and role names through regular expression matching; The credits information will be stored in a structured manner to facilitate quick subsequent lookup and reference; and / or The step S4 specifically includes: For each frame of the image, a pre-built face database and a trained face recognition model are used to identify the person appearing in the video; The facial recognition model extracts facial features and compares them with known person features in a database to determine the person's identity and associate it with the corresponding role; and / or The step S5 specifically includes: The pre-trained CLIP model is used to extract cross-modal features from video frames and convert images into high-dimensional feature vectors.

4. The method for retrieval of integrated film and television drama content based on large language model, face recognition, target detection and cross-modal vector according to claim 3 is characterized in that: Based on the search text of the video content entered by the user, the large language model is used to perform word segmentation and optimization processing, including: When a user makes a video retrieval request, the user's input text is first preprocessed using a large language model to extract key words; And then through multiple rounds of dialogue interaction with the user, we can more accurately understand the user's search intention.

5. The method for retrieval of integrated film and television drama content based on large language model, face recognition, target detection and cross-modal vector according to claim 3 is characterized by: The Elasticsearch search engine is used to match the pre-processed object labels according to the user's query keywords, return a list of objects related to the query, and give a confidence score for each object; Using face recognition technology, the user can find the closest match in the face database based on the description provided by the user or the photo uploaded directly, and also provide a confidence score for each character; The CLIP model is used to convert the text input by the user into a feature vector, which is then compared with the image feature vector generated in the preprocessing stage to find the most similar scenes or events, which are also accompanied by a similarity score.

6. The method for retrieval of integrated film and television drama content based on large language model, face recognition, target detection and cross-modal vector according to claim 3 is characterized in that: Secondary sorting is performed based on the number of hits and the final weighted score, including: For each video segment, a preliminary score is obtained by weighted average based on the matching degree of the three dimensions of objects, people, and scene events; Additional points are awarded based on the number of consecutive matching frames; All video segments are sorted from high to low according to their total scores, and the results that best meet the query requirements are presented to users.

7. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the fusion film and television drama content retrieval method based on a large language model, face recognition, target detection and cross-modal vector as described in one of claims 1-6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the processor to implement the integrated film and television content retrieval method based on a large language model, face recognition, target detection and cross-modal vector as described in one of claims 1-6 when executed.

9. A computer program product, characterized in that: The computer program product includes a computer program, which, when executed by a processor, implements the integrated film and television drama content retrieval method based on a large language model, face recognition, target detection and cross-modal vector according to one of claims 1 to 6.

Images