A method and system for constructing a large model knowledge base based on video data

Abstract

The present invention relates to the field of data processing technology, and specifically discloses a method and system for constructing a large model knowledge base based on video data, the method comprising: acquiring video data, extracting audio data according to the video data, and determining whether to set the text data of the audio data as a target text; when the text data of the audio data is not set as the target text, extracting the picture data of the video data, and setting the text data of the picture data as the target text; preprocessing the target text to obtain a preprocessed text vector, and storing the preprocessed text vector in a vector database. The present invention can effectively and systematically process multimodal data by extracting audio and picture data from video data, which is conducive to improving the large model knowledge base and improving the accuracy of answers.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and more specifically, to a method and system for constructing a large model knowledge base based on video data. Background Art

[0002] With the successful launch of OpenAI's large language model ChatGPT, the large language model (LLM) can not only understand many human questions and instructions and smoothly carry out multi-round conversations, but also interact with external systems through function calling. However, large models still face challenges in hallucination problems, poor knowledge timeliness, and insufficient vertical field capabilities. The emergence of RAG (retrieval-augmented generation) provides a new idea for solving these problems. A key advantage of RAG is that it can make up for the knowledge blind spots of generative models. The knowledge of generative models is usually limited to their training data, while RAG can dynamically utilize the latest and broader external knowledge sources to improve the accuracy and timeliness of answers.

[0003] However, the current mainstream RAG technology mainly targets text data, which is vectorized and input into a vector database. There is no effective and systematic way to process multimodal data (pictures, videos). Summary of the invention

[0004] The present invention provides a method and system for constructing a large model knowledge base based on video data, which is used to solve the problem that multimodal data cannot be applied to the large model knowledge base in the prior art, including:

[0005] Acquire video data, extract audio data according to the video data, and determine whether to set text data of the audio data as target text;

[0006] When the text data of the audio data is not set as the target text, the picture data of the video data is extracted, and the text data of the picture data is set as the target text;

[0007] The target text is preprocessed to obtain a preprocessed text vector, and the preprocessed text vector is stored in a vector database.

[0008] Further, extracting audio data according to video data and determining whether to set text data of the audio data as target text includes:

[0009] Extracting audio data from the video data, performing text detection on the audio data from the video data, and obtaining text data from the audio data;

[0010] Obtaining the number of characters in the text data, determining whether the number of characters in the text data is greater than a first preset threshold, and if the number of characters in the text data is greater than the first preset threshold, directly setting the text data as the target text;

[0011] If the number of characters in the text data is less than or equal to the first preset threshold, the text data is not set as the target text.

[0012] Further, extracting the picture data of the video data and setting the text data of the picture data as the target text includes:

[0013] Intercepting picture data of the video data according to the video data corresponding to the audio data, and extracting text data in the picture data;

[0014] Determine the number of characters of the text data according to the text data in the image data, determine whether the number of characters of the text data is greater than a second preset threshold, and if the number of characters of the text data is greater than the second preset threshold, directly set the text data as the target text;

[0015] If the number of characters in the text data is less than or equal to the second preset threshold, the image data is understood based on the image understanding service to obtain image understanding data, and the image understanding data is set as the target text.

[0016] Furthermore, the step of intercepting the picture data of the video data according to the video data corresponding to the audio data includes:

[0017] Acquire a time axis corresponding to audio data having a character number less than or equal to a first preset threshold, and determine video data corresponding to the audio data according to the time axis;

[0018] Extracting image frames of video data, performing grayscale processing on the image frames of video data, and obtaining a video grayscale image;

[0019] The video grayscale image is averagely segmented to obtain a plurality of grayscale image blocks after segmentation, a preset grayscale classification is obtained, and the grayscale entropy of the grayscale image block is calculated according to the preset grayscale classification;

[0020] Calculate the average value of the grayscale entropy of the grayscale image block to obtain the grayscale feature value of the image frame;

[0021] Image frames whose grayscale feature values ​​are greater than a second preset threshold are screened out to obtain key image frames, each key image frame is clustered, a central image frame is determined according to the clustering result, and the central image frame is used as picture data of the video data.

[0022] Furthermore, the grayscale entropy of the grayscale image block is calculated according to the preset grayscale grading, including:

[0023] The grayscale entropy of the grayscale image block is calculated according to the grayscale entropy calculation formula, and the grayscale entropy calculation formula is specifically:

[0024]

[0025] Wherein, H is the grayscale entropy, m is the preset grayscale classification, and p(i) is the frequency of the i-th grayscale classification in the grayscale image block.

[0026] Furthermore, clustering the key image frames includes:

[0027] Randomly select k key image frames as the initial clustering centers and set the maximum number of iterations;

[0028] Calculate the distance between the grayscale entropy of the remaining key image frames and the k initial cluster centers, and cluster each key image frame into the cluster partition closest to the initial cluster center;

[0029] Calculate the mean value of the grayscale entropy of the key image frames in each cluster partition, and set a new cluster center according to the mean value of the grayscale entropy of the key image frames in each cluster partition;

[0030] Repeat the iteration of the new cluster center until the cluster center no longer changes or the maximum number of iterations is reached, and the final k cluster partitions and corresponding cluster centers are obtained.

[0031] Furthermore, the performing image understanding on the image data based on the image understanding service includes:

[0032] Establishing an image database, obtaining a number of image data in the image database, and preprocessing the number of image data in the image database to obtain a training sample set;

[0033] Establishing an initial image understanding model according to the training sample set and training the initial image understanding model to obtain a trained image understanding model;

[0034] Inputting the image data to be understood into the trained image understanding model to obtain image understanding data of the image data to be understood;

[0035] Furthermore, the preprocessing of the target text includes:

[0036] Get the target text, perform word segmentation on the target text based on the Jieba word segmentation tool, remove stop words from the text after word segmentation, and obtain the text segmentation;

[0037] The text word segmentation is vectorized according to a preset vector algorithm to obtain a vectorized text vector.

[0038] Furthermore, after storing the preprocessed text vector into the vector database, the method further includes:

[0039] Calculating the cosine similarity of the text vectors of each target text, and selecting the target text whose cosine similarity is greater than a third preset threshold;

[0040] The target texts whose cosine similarity is greater than a third preset threshold are linked to obtain link relationships of the target texts, and the link relationships of the target texts are stored in a vector database.

[0041] In order to achieve the above object, the present invention also provides a large model knowledge base construction system based on video data, comprising:

[0042] The first module is used to obtain video data, extract audio data according to the video data, and determine whether to set the text data of the audio data as the target text;

[0043] The second module is used for extracting the picture data of the video data and setting the text data of the picture data as the target text when the text data of the audio data is not set as the target text;

[0044] The third module is used to preprocess the target text to obtain a preprocessed text vector, and store the preprocessed text vector into a vector database.

[0045] The beneficial effect of the present invention lies in that, by extracting audio and image data from video data and performing video-to-audio processing on the video data, the loss of image key frames caused by video data frame loss can be prevented. At the same time, through the understanding service of image data, the situation where the image extracted data is empty or garbled is effectively prevented, and multimodal data is effectively and systematically processed, which is conducive to improving the large model knowledge base and improving the accuracy of answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 The overall flow chart of a method for constructing a large model knowledge base based on video data proposed in an embodiment of the present invention is shown;

[0048] Figure 2 A structural schematic diagram of a large model knowledge base construction system based on video data proposed in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0051] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0052] The present application embodiment provides a method for constructing a large model knowledge base based on video data, such as Figure 1 As shown, including:

[0053] S101, acquiring video data, extracting audio data according to the video data, and determining whether to set text data of the audio data as target text;

[0054] In some embodiments of the present application, the extracting of audio data based on video data and determining whether to set the text data of the audio data as the target text includes: extracting the audio data in the video data, performing text detection on the audio data in the video data, and obtaining the text data in the audio data; obtaining the number of characters in the text data, and determining whether the number of characters in the text data is greater than a first preset threshold value, if the number of characters in the text data is greater than the first preset threshold value, directly setting the text data as the target text; if the number of characters in the text data is less than or equal to the first preset threshold value, not setting the text data as the target text.

[0055] In this embodiment, audio data in the video data is extracted, and text detection is performed on the audio data. A first preset threshold is set according to the length of the extracted audio. In this embodiment, the first preset threshold is set to 500 characters, and text data with a character count greater than 500 characters is set as target text.

[0056] S102, when the text data of the audio data is not set as the target text, extracting the picture data of the video data, and setting the text data of the picture data as the target text;

[0057] In some embodiments of the present application, the extracting of picture data of video data and setting the text data of the picture data as target text includes: intercepting the picture data of the video data according to the video data corresponding to the audio data, and extracting the text data in the picture data; determining the number of characters in the text data according to the text data in the picture data, and judging whether the number of characters in the text data is greater than a second preset threshold value; if the number of characters in the text data is greater than the second preset threshold value, directly using the text data as the target text; if the number of characters in the text data is less than or equal to the second preset threshold value, performing image understanding on the picture data based on the image understanding service to obtain image understanding data, and using the image understanding data as the target text.

[0058] In this embodiment, when the number of characters is less than or equal to 500 characters, in order to prevent the loss of video data, the picture data of the video data is intercepted according to the video data corresponding to the audio data, and the text data in the picture data is extracted based on the OCR technology. In this embodiment, the second preset threshold is set to 50 characters, and the text data with a character number greater than 50 characters is set as the target text. When the number of characters is less than or equal to 50 characters, in order to prevent the image extracted data from being empty or garbled, the image understanding data is obtained by using the image understanding service, and the image understanding data is set as the target text.

[0059] In some embodiments of the present application, the method of intercepting picture data of video data based on video data corresponding to audio data includes: obtaining a time axis corresponding to audio data having a character number less than or equal to a first preset threshold, and determining the video data corresponding to the audio data based on the time axis; extracting image frames of the video data, and performing grayscale processing on the image frames of the video data to obtain a video grayscale image; performing average segmentation on the video grayscale image to obtain a number of grayscale image blocks after segmentation, obtaining a preset grayscale grading, and calculating the grayscale entropy of the grayscale image blocks based on the preset grayscale grading; calculating the average grayscale entropy of the grayscale image blocks to obtain grayscale eigenvalues ​​of the image frames; screening out image frames having grayscale eigenvalues ​​greater than a second preset threshold to obtain key image frames, clustering the key image frames, determining the central image frame based on the clustering results, and using the central image frame as the picture data of the video data.

[0060] In some embodiments of the present application, the step of calculating the grayscale entropy of the grayscale image block according to the preset grayscale grading includes:

[0061] The grayscale entropy of the grayscale image block is calculated according to the grayscale entropy calculation formula, and the grayscale entropy calculation formula is specifically:

[0062]

[0063] Wherein, H is the grayscale entropy, m is the preset grayscale classification, and p(i) is the frequency of the i-th grayscale classification in the grayscale image block.

[0064] In this embodiment, grayscale entropy, as a statistical form of image features, can reflect the average amount of information in the image. The larger the grayscale entropy, the more complex the image information. By calculating the grayscale entropy of the grayscale image block, key image frames with complex image information are screened out, and clustering is performed based on the grayscale entropy of the key image frames to extract the central image frame of the video data, effectively removing redundant frames without losing key video data.

[0065] In some embodiments of the present application, the clustering of each key image frame includes: randomly selecting k key image frames as initial clustering centers and setting a maximum number of iterations; calculating the distance between the grayscale entropy of the remaining key image frames and the k initial clustering centers, and clustering each key image frame into a cluster partition that is closest to the initial cluster center; calculating the mean of the grayscale entropy of the key image frames in each cluster partition, and setting a new cluster center according to the mean of the grayscale entropy of the key image frames in each cluster partition; repeatedly iterating the new cluster center until the cluster center no longer changes or the maximum number of iterations is reached, thereby obtaining the final k cluster partitions and the corresponding cluster centers.

[0066] In this embodiment, the key image frames are clustered according to the grayscale entropy of each key image frame based on the k-means clustering algorithm. The k value is set according to the number of key image frames. In this embodiment, the k value is set to 10, and the maximum number of iterations is 250. By clustering the key image frames, the central image frame is extracted to prevent data redundancy.

[0067] In some embodiments of the present application, the image understanding service based on the image data includes: establishing an image database, obtaining a number of image data in the image database, preprocessing the number of image data in the image database, and obtaining a training sample set; establishing an initial image understanding model according to the training sample set and training the initial image understanding model to obtain a trained image understanding model; inputting the image data to be understood into the trained image understanding model to obtain image understanding data of the image data to be understood;

[0068] In this embodiment, a picture database is established by pre-collecting picture data associated with video data, an initial picture understanding model is constructed based on a multi-layer convolutional neural network, and the initial picture understanding model is trained by using a training sample set obtained by preprocessing the picture database, thereby obtaining a trained picture understanding model and outputting picture understanding data based on the picture data to be understood.

[0069] S103, preprocessing the target text to obtain a preprocessed text vector, and storing the preprocessed text vector in a vector database.

[0070] In some embodiments of the present application, the preprocessing of the target text includes:

[0071] Get the target text, perform word segmentation on the target text based on the jieba word segmentation tool, remove stop words from the text after word segmentation, and obtain text segmentation; perform vectorization on the text segmentation according to the preset vector algorithm to obtain the text vector after vectorization.

[0072] In this embodiment, stop words are specifically words that have no practical meaning. After the target text is segmented by the jieba word segmentation tool, the text segmentation is vectorized based on the word2vec algorithm to obtain a text vector, and the text vector is stored in a vector database.

[0073] In some embodiments of the present application, after storing the preprocessed text vector into the vector database, it also includes: calculating the cosine similarity of the text vectors of each target text, screening out the target texts whose cosine similarity is greater than the third preset threshold; linking the target texts whose cosine similarity is greater than the third preset threshold to obtain the link relationship of each target text, and storing the link relationship of each target text into the vector database.

[0074] In this embodiment, the similarity of each target text is obtained based on the cosine similarity between each text vector, the target texts with similarity greater than a third preset threshold are linked, and the link relationship of each target text is stored in a vector database to improve the vector database and enhance the accuracy of the large model answer.

[0075] Based on the same technical concept, such as Figure 2 As shown, the present invention also provides a large model knowledge base construction system based on video data, including:

[0076] The first module is used to obtain video data, extract audio data based on the video data, and determine whether to set the text data of the audio data as the target text; the second module is used to extract the image data of the video data when the text data of the audio data is not set as the target text, and set the text data of the image data as the target text; the third module is used to preprocess the target text to obtain the preprocessed text vector, and store the preprocessed text vector into the vector database.

[0077] By applying the above technical scheme, the present invention obtains video data, extracts audio data according to the video data, and determines whether to set the text data of the audio data as the target text; when the text data of the audio data is not set as the target text, the image data of the video data is extracted, and the text data of the image data is set as the target text; the target text is preprocessed to obtain a preprocessed text vector, and the preprocessed text vector is stored in a vector database. By extracting audio and image data from video data, the present invention can effectively and systematically process multimodal data, which is conducive to improving the large model knowledge base and improving the accuracy of answers.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present invention can be implemented by hardware, or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present invention.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for constructing a large model knowledge base based on video data, characterized in that: The method comprises: Acquire video data, extract audio data according to the video data, and determine whether to set text data of the audio data as target text; When the text data of the audio data is not set as the target text, the picture data of the video data is extracted, and the text data of the picture data is set as the target text; Preprocess the target text to obtain a preprocessed text vector, and store the preprocessed text vector into a vector database; The step of extracting audio data according to video data and determining whether to set text data of the audio data as target text comprises: Extracting audio data from the video data, performing text detection on the audio data from the video data, and obtaining text data from the audio data; Obtaining the number of characters in the text data, determining whether the number of characters in the text data is greater than a first preset threshold, and if the number of characters in the text data is greater than the first preset threshold, directly setting the text data as the target text; If the number of characters in the text data is less than or equal to the first preset threshold, the text data is not set as the target text; The step of extracting the image data of the video data and setting the text data of the image data as the target text includes: Intercepting picture data of the video data according to the video data corresponding to the audio data, and extracting text data in the picture data; Determine the number of characters of the text data according to the text data in the image data, determine whether the number of characters of the text data is greater than a second preset threshold, and if the number of characters of the text data is greater than the second preset threshold, directly set the text data as the target text; If the number of characters in the text data is less than or equal to a second preset threshold, performing image understanding on the image data based on the image understanding service to obtain image understanding data, and setting the image understanding data as the target text; The step of intercepting the image data of the video data according to the video data corresponding to the audio data comprises: Acquire a time axis corresponding to audio data having a character number less than or equal to a first preset threshold, and determine video data corresponding to the audio data according to the time axis; Extracting image frames of video data, performing grayscale processing on the image frames of video data, and obtaining a video grayscale image; The video grayscale image is averagely segmented to obtain a plurality of grayscale image blocks after segmentation, a preset grayscale classification is obtained, and the grayscale entropy of the grayscale image block is calculated according to the preset grayscale classification; Calculate the average value of the grayscale entropy of the grayscale image block to obtain the grayscale feature value of the image frame; Image frames whose grayscale feature values ​​are greater than a second preset threshold are screened out to obtain key image frames, each key image frame is clustered, a central image frame is determined according to the clustering result, and the central image frame is used as picture data of the video data.

2. The method for constructing a large model knowledge base based on video data according to claim 1, characterized in that: The step of calculating the grayscale entropy of the grayscale image block according to the preset grayscale grading comprises: The grayscale entropy of the grayscale image block is calculated according to the grayscale entropy calculation formula, and the grayscale entropy calculation formula is specifically: Wherein, H is the grayscale entropy, m is the preset grayscale classification, and p(i) is the frequency of the i-th grayscale classification in the grayscale image block.

3. The method for constructing a large model knowledge base based on video data according to claim 1, characterized in that: The clustering of each key image frame comprises: Randomly select k key image frames as the initial clustering centers and set the maximum number of iterations; Calculate the distance between the grayscale entropy of the remaining key image frames and the k initial cluster centers, and cluster each key image frame into the cluster partition closest to the initial cluster center; Calculate the mean value of the grayscale entropy of the key image frames in each cluster partition, and set a new cluster center according to the mean value of the grayscale entropy of the key image frames in each cluster partition; Repeat the iteration of the new cluster center until the cluster center no longer changes or the maximum number of iterations is reached, and the final k cluster partitions and corresponding cluster centers are obtained.

4. The method for constructing a large model knowledge base based on video data according to claim 1, characterized in that: The image understanding service based on the image data includes: Establishing an image database, obtaining a number of image data in the image database, and preprocessing the number of image data in the image database to obtain a training sample set; Establishing an initial image understanding model according to the training sample set and training the initial image understanding model to obtain a trained image understanding model; The image data to be understood is input into the trained image understanding model to obtain image understanding data of the image data to be understood.

5. The method for constructing a large model knowledge base based on video data according to claim 1, characterized in that: The preprocessing of the target text includes: Get the target text, perform word segmentation on the target text based on the Jieba word segmentation tool, remove stop words from the text after word segmentation, and obtain the text segmentation; The text segmentation is vectorized according to a preset vector algorithm to obtain a vectorized text vector.

6. The method for constructing a large model knowledge base based on video data according to claim 5, characterized in that: After storing the preprocessed text vector into the vector database, the method further includes: Calculating the cosine similarity of the text vectors of each target text, and selecting the target text whose cosine similarity is greater than a third preset threshold; The target texts whose cosine similarity is greater than a third preset threshold are linked to obtain link relationships of the target texts, and the link relationships of the target texts are stored in a vector database.

7. A large model knowledge base construction system based on video data, characterized in that: include: The first module is used to obtain video data, extract audio data according to the video data, and determine whether to set the text data of the audio data as the target text; The second module is used for extracting the picture data of the video data and setting the text data of the picture data as the target text when the text data of the audio data is not set as the target text; The third module is used to preprocess the target text to obtain a preprocessed text vector, and store the preprocessed text vector into a vector database; The first module extracts audio data according to the video data, and determines whether to set the text data of the audio data as the target text, including: Extracting audio data from the video data, performing text detection on the audio data from the video data, and obtaining text data from the audio data; Obtaining the number of characters in the text data, determining whether the number of characters in the text data is greater than a first preset threshold, and if the number of characters in the text data is greater than the first preset threshold, directly setting the text data as the target text; If the number of characters in the text data is less than or equal to the first preset threshold, the text data is not set as the target text; The second module extracts the picture data of the video data and sets the text data of the picture data as the target text, including: Intercepting picture data of the video data according to the video data corresponding to the audio data, and extracting text data in the picture data; Determine the number of characters of the text data according to the text data in the image data, determine whether the number of characters of the text data is greater than a second preset threshold, and if the number of characters of the text data is greater than the second preset threshold, directly set the text data as the target text; If the number of characters in the text data is less than or equal to a second preset threshold, performing image understanding on the image data based on the image understanding service to obtain image understanding data, and setting the image understanding data as the target text; The step of intercepting the image data of the video data according to the video data corresponding to the audio data comprises: Acquire a time axis corresponding to audio data having a character number less than or equal to a first preset threshold, and determine video data corresponding to the audio data according to the time axis; Extracting image frames of video data, performing grayscale processing on the image frames of video data, and obtaining a video grayscale image; The video grayscale image is averagely segmented to obtain a plurality of grayscale image blocks after segmentation, a preset grayscale classification is obtained, and the grayscale entropy of the grayscale image block is calculated according to the preset grayscale classification; Calculate the average value of the grayscale entropy of the grayscale image block to obtain the grayscale feature value of the image frame; Image frames whose grayscale feature values ​​are greater than a second preset threshold are screened out to obtain key image frames, each key image frame is clustered, a central image frame is determined according to the clustering result, and the central image frame is used as picture data of the video data.

Images