Recommended video feature storage and rapid retrieval method based on picture similarity

Abstract

The invention relates to the technical field of multimedia information processing, in particular to a recommended video feature storage and quick retrieval method based on picture similarity, which comprises the following steps: S1, establishing an alternative video service database; s2, original video feature extraction: after an original video is obtained, extracting picture feature value, character feature value and face feature value information of key frames of the original video by adopting a video key frame processing mode which is the same as that in the step S1; s3, retrieving characteristic values of alternative video picture frames; and S4, calculating the similarity of the alternative videos. According to the method, the accuracy and retrieval speed of video recommendation can be effectively improved, and more convenient and personalized video browsing experience is brought to users.

Description

Technical Field

[0001] The present invention relates to the technical field of multimedia information processing, and in particular to a method for storing and quickly retrieving recommended video features based on picture similarity. Background Art

[0002] With the progress of science and technology and information technology, especially the continuous development of computer technology, network technology and large-capacity storage technology, people have used various means to collect and produce a large number of various types of multimedia video data. These video data have great application and research value, but also bring challenges in video storage, management and retrieval.

[0003] Although existing video processing, feature extraction, similarity calculation, and storage and retrieval technologies have made some progress, there are still some shortcomings. For example, in terms of feature extraction, existing methods may not be able to accurately extract all the key information in the video; in terms of similarity calculation, existing algorithms may not be able to accurately measure the similarity between two videos; in terms of storage and retrieval, existing technologies may not be able to efficiently process large-scale video data.

[0004] Therefore, a method for storing and quickly retrieving recommended video features based on picture similarity is needed to overcome the shortcomings of the prior art. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method for storing and quickly retrieving recommended video features based on picture similarity, which can significantly improve the accuracy and efficiency of video recommendations.

[0006] The present invention is achieved through the following technical solutions:

[0007] A method for storing and quickly retrieving recommended video features based on picture similarity is provided, comprising the following steps:

[0008] S1. Establishing a database of candidate video services:

[0009] First, the video key frames are processed, and the image feature values, text feature values, and face feature value information in the video key frame images are extracted, and then normalized and stored in the corresponding feature vector library. The business data of each feature vector library forms a business database;

[0010] S2. Original video feature extraction:

[0011] After the original video is obtained, the same video key frame processing method as step S1 is used to extract the image feature value, text feature value and face feature value information of the original video key frame;

[0012] S3, alternative video frame feature value retrieval:

[0013] The feature vector library is searched at a ratio of 1:10, and the candidate image feature value vector library, text feature value vector library, and face feature value vector library are searched respectively. The feature search is sorted from small to large according to the cosine similarity; S4, similarity calculation of candidate videos:

[0014] According to the candidate feature values ​​extracted in the above steps, the video key frame data in the business database corresponding to the candidate video information is spliced ​​to obtain n candidate videos;

[0015] Generate corresponding feature value sequences from the image feature values, text feature values, and face feature values ​​in the original video according to the time when the key frames appear from front to back;

[0016] Generate corresponding feature value sequences for image feature values, text feature values, and face feature values ​​in the candidate video according to the time when the key frames appear from front to back;

[0017] Calculate image similarity, text similarity and face similarity respectively;

[0018] Similarity calculation of alternative videos: S v =K1I+K2T+K3F

[0019] Where: S v is the similarity of the candidate videos, K1 is the image similarity coefficient, I is the image similarity of the video, K2 is the text similarity coefficient, T is the text similarity of the video, K3 is the face similarity coefficient, and F is the face similarity of the video;

[0020] Similar videos are sorted by S v Sort from smallest to largest.

[0021] Furthermore, in step S4, image similarity matching calculation is performed: looping through all candidate videos, taking out each video for calculation, respectively calculating the key frame and cosine value of the minimum cosine similarity between the key frame image features of the original video and the key frame features in the candidate video; accumulating the cosine values ​​of the key frames in all matching candidate videos, dividing by the corresponding number of key frames in the original video, to obtain the average image similarity of the two videos;.

[0022] Furthermore, in step S4, the text similarity matching calculation is performed: all the alternative videos are looped, each video is taken out for calculation, and the key frames and cosine values ​​with the smallest cosine similarity between the key frame text features of the original video and the key frame text features in the alternative video are calculated respectively; the cosine values ​​of the key frames in all matching alternative videos are accumulated, and divided by the corresponding number of key frames in the original video to obtain the average text similarity of the two videos.

[0023] Furthermore, in step S4, face similarity matching calculation is performed: all candidate videos are looped, each video is taken out for calculation, and the key frame and cosine value with the smallest cosine similarity between the face features of the key frame of the original video and the face features of the key frame in the candidate video are calculated respectively; the cosine values ​​of the key frames in all matching candidate videos are accumulated, and divided by the corresponding number of key frames in the original video to obtain the average face similarity of the two videos.

[0024] Furthermore, when processing the video key frames, JavaCV is used to extract the key frames of the video. If the key frame width is greater than 640px, it is scaled to 640px.

[0025] Furthermore, the feature values ​​of key frame images in the video are extracted: the images are feature extracted through CLIP-ViT-B-32-IMAGE, normalized and stored in the image feature vector library, and the business data of the feature value unique identifier of the image feature vector library, the video information to which it belongs, and the location information in the video are stored in the business database.

[0026] Furthermore, the text feature values ​​in the key frame images in the video are extracted: the text in the picture is identified by OCR technology, the feature values ​​are extracted by segmenting the text according to position and punctuation marks, the text feature values ​​are stored in a text feature vector library, and the business data of the feature value unique identifier of the text feature vector library, the video information to which it belongs, and the position information in the video are stored in a business database.

[0027] Furthermore, facial feature values ​​of key frame images in the video are extracted: facial features in the picture are identified through face recognition, converted into facial feature values, the facial feature values ​​are stored in a facial feature vector library, and the business data of the feature value unique identifier of the facial feature vector library, the video information to which it belongs, and the location information in the video are stored in a business database.

[0028] Beneficial effects of the present invention:

[0029] The present invention uses advanced feature extraction technology to accurately obtain the key features of the video. In the storage link, an efficient data storage architecture is used to optimize space utilization and ensure the security of feature data. During retrieval, a video set that is most similar to the query video feature is quickly found based on a carefully designed similarity algorithm.

[0030] The recommended video feature storage and rapid retrieval method based on picture similarity provided by the present invention can realize efficient storage and rapid retrieval of video features. Compared with traditional video retrieval methods that rely on keyword matching or image feature matching, it can realize efficient and high-precision processing of large-scale video data, and can effectively improve the accuracy and retrieval speed of video recommendations, bringing users a more convenient and personalized video browsing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The present invention is a flowchart for establishing a candidate video service database.

[0032] Figure 2 This is a flow chart of the video retrieval method in the present invention. DETAILED DESCRIPTION

[0033] In order to clearly illustrate the technical features of this solution, this solution is described below through a specific implementation method.

[0034] A method for storing and quickly retrieving recommended video features based on picture similarity comprises the following steps:

[0035] S1. Establishing a database of candidate video services:

[0036] First, the video key frames are processed, and the image feature values, text feature values ​​and face feature value information in the video key frame images are extracted, and then normalized and stored in the corresponding feature vector library. The business data of each feature vector library forms a business database.

[0037] When processing video key frames, JavaCV is used to extract key frames from the video. If the key frame width is greater than 640px, it is scaled to 640px.

[0038] Extract the feature values ​​of key frame images in the video: The image is feature extracted through CLIP-ViT-B-32-IMAGE, normalized and stored in the image feature vector library, and the business data of the feature value unique identifier of the image feature vector library, the video information to which it belongs, and the location information in the video are stored in the business database.

[0039] Extract text feature values ​​from key frame images in the video: Use OCR technology to identify the text in the picture, extract feature values ​​by segmenting the text according to position and punctuation marks, store the text feature values ​​in a text feature vector library, and store the business data of the feature value unique identifier of the text feature vector library, the video information to which it belongs, and the location information in the video in the business database.

[0040] Extract facial feature values ​​of key frame images in the video: Use face recognition to identify facial features in the picture, convert them into facial feature values, store the facial feature values ​​in a facial feature vector library, and store the business data of the facial feature vector library's unique identifier, the video information to which it belongs, and the location information in the video into a business database.

[0041] S2. Original video feature extraction:

[0042] After the original video is obtained, the same video key frame processing method as step S1 is adopted to extract the image feature value, text feature value and face feature value information of the original video key frame.

[0043] S3, alternative video frame feature value retrieval:

[0044] The feature vector library is searched at a ratio of 1:10, and the candidate image feature value vector library, text feature value vector library, and face feature value vector library are searched respectively. The feature retrieval is sorted from small to large according to the cosine similarity.

[0045] S4. Calculation of similarity of candidate videos:

[0046] According to the candidate feature values ​​extracted in the above steps, the video key frame data in the business database corresponding to the candidate video information is spliced ​​to obtain n candidate videos;

[0047] Generate corresponding feature value sequences from the image feature values, text feature values, and face feature values ​​in the original video according to the time when the key frames appear from front to back;

[0048] Generate corresponding feature value sequences for image feature values, text feature values, and face feature values ​​in the candidate video according to the time when the key frames appear from front to back;

[0049] Calculate image similarity, text similarity and face similarity respectively;

[0050] Image similarity matching calculation: loop through all candidate videos, take out each video for calculation, and calculate the key frame and cosine value with the smallest cosine similarity between the key frame image features of the original video and the key frame features in the candidate video; add up the cosine values ​​of the key frames in all matching candidate videos, divide by the corresponding number of key frames in the original video, and get the average image similarity of the two videos;.

[0051] Text similarity matching calculation: loop through all candidate videos, take out each video for calculation, and calculate the key frame and cosine value with the smallest cosine similarity between the key frame text features of the original video and the key frame text features in the candidate video; add up the cosine values ​​of the key frames in all matching candidate videos, divide by the corresponding number of key frames in the original video, and get the average text similarity of the two videos.

[0052] Face similarity matching calculation: loop all candidate videos, take out each video for calculation, calculate the key frame and cosine value with the smallest cosine similarity between the face features of the key frame of the original video and the face features of the key frame in the candidate video; accumulate the cosine values ​​of the key frames in all matching candidate videos, divide by the corresponding number of key frames in the original video, and get the average face similarity of the two videos.

[0053] Similarity calculation of alternative videos: S v =K1I+K2T+K3F

[0054] Where: S vis the similarity of the candidate videos, K1 is the image similarity coefficient, I is the image similarity of the video, K2 is the text similarity coefficient, T is the text similarity of the video, K3 is the face similarity coefficient, and F is the face similarity of the video.

[0055] Similar videos are sorted by S v Sort from smallest to largest.

[0056] Of course, the above description is not limited to the above examples. Technical features not described in the present invention can be achieved by or by using existing technologies, which will not be described here. The above embodiments and drawings are only used to illustrate the technical scheme of the present invention and are not limitations of the present invention. The present invention is described in detail with reference to the preferred implementation methods. Ordinary technicians in this field should understand that changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention do not depart from the purpose of the present invention, and should also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for storing and quickly retrieving recommended video features based on picture similarity, characterized in that: The following steps are involved: S1. Establishing a database of candidate video services: First, the video key frames are processed, and the image feature values, text feature values, and face feature value information in the video key frame images are extracted, and then normalized and stored in the corresponding feature vector library. The business data of each feature vector library forms a business database; S2. Original video feature extraction: After the original video is obtained, the same video key frame processing method as step S1 is used to extract the image feature value, text feature value and face feature value information of the original video key frame; S3, alternative video frame feature value retrieval: Search from the feature vector library at a ratio of 1:10, and search the candidate image feature value vector library, text feature value vector library, and face feature value vector library respectively. The feature retrieval is sorted from small to large according to the cosine similarity; S4. Calculation of similarity of candidate videos: According to the candidate feature values ​​extracted in the above steps, the video key frame data in the business database corresponding to the candidate video information is spliced ​​to obtain n candidate videos; Generate corresponding feature value sequences from the image feature values, text feature values, and face feature values ​​in the original video according to the time when the key frames appear from front to back; Generate corresponding feature value sequences for image feature values, text feature values, and face feature values ​​in the candidate video according to the time when the key frames appear from front to back; Calculate image similarity, text similarity and face similarity respectively; Similarity calculation of alternative videos: S v =K1I+K2T+K3F Where: S v is the similarity of the candidate videos, K1 is the image similarity coefficient, I is the image similarity of the video, K2 is the text similarity coefficient, T is the text similarity of the video, K3 is the face similarity coefficient, and F is the face similarity of the video; Similar videos are sorted by S v Sort from smallest to largest.

2. The method for storing and quickly retrieving recommended video features based on picture similarity according to claim 1, characterized in that: In step S4, image similarity matching calculation: loop all candidate videos, take out each video for calculation, and calculate the key frame and cosine value of the minimum cosine similarity between the key frame image features of the original video and the key frame features in the candidate video; Accumulate the cosine values ​​of the key frames in all matching alternative videos, divide by the number of corresponding key frames in the original video, and get the average image similarity of the two videos;.

3. The method for storing and quickly retrieving recommended video features based on picture similarity according to claim 1, characterized in that: In step S4, text similarity matching calculation: loop all candidate videos, take out each video for calculation, and calculate the key frame and cosine value of the minimum cosine similarity between the key frame text feature of the original video and the key frame text feature of the candidate video; The cosine values ​​of the key frames in all matching candidate videos are accumulated and divided by the number of corresponding key frames in the original video to obtain the average text similarity of the two videos.

4. The method for storing and quickly retrieving recommended video features based on picture similarity according to claim 1, characterized in that: In step S4, face similarity matching calculation: loop all candidate videos, take out each video for calculation, and calculate the key frame and cosine value of the minimum cosine similarity between the face features of the key frame of the original video and the face features of the key frame in the candidate video; The cosine values ​​of the key frames in all matching candidate videos are accumulated and divided by the number of corresponding key frames in the original video to obtain the average face similarity of the two videos.

5. The method for storing and quickly retrieving recommended video features based on picture similarity according to claim 1, characterized in that: When processing video key frames, JavaCV is used to extract key frames from the video. If the key frame width is greater than 640px, it is scaled to 640px.

6. The method for storing and quickly retrieving recommended video features based on picture similarity according to claim 5, characterized in that: Extract the feature values ​​of key frame images in the video: The image is feature extracted through CLIP-ViT-B-32-IMAGE, normalized and stored in the image feature vector library, and the business data of the feature value unique identifier of the image feature vector library, the video information to which it belongs, and the location information in the video are stored in the business database.

7. The method for storing and quickly retrieving recommended video features based on picture similarity according to claim 1, characterized in that: Extract text feature values ​​from key frame images in the video: Use OCR technology to identify the text in the picture, extract feature values ​​by segmenting the text according to position and punctuation marks, store the text feature values ​​in a text feature vector library, and store the business data of the feature value unique identifier of the text feature vector library, the video information to which it belongs, and the location information in the video in the business database.

8. The method for storing and quickly retrieving recommended video features based on picture similarity according to claim 1, characterized in that: Extract facial feature values ​​of key frame images in the video: Use face recognition to identify facial features in the picture, convert them into facial feature values, store the facial feature values ​​in a facial feature vector library, and store the business data of the facial feature vector library's unique identifier, the video information to which it belongs, and the location information in the video into a business database.

Images