An intelligent database video retrieval method based on big data technology

Abstract

The present invention relates to the field of intelligent information retrieval technology, and specifically to an intelligent database video retrieval method based on big data technology, comprising the following steps: S1: performing spatiotemporal slicing processing on the original video stream; S2: extracting visual feature vectors, audio waveform vectors, and text description vectors; S3: constructing a cross-modal association matrix; S4: constructing a hierarchical hybrid index structure, including a real-time update layer and a static storage layer; S5: after receiving a user search request, screening a candidate video set based on the hierarchical hybrid index structure to generate a search result; S6: updating the cross-modal association matrix and synchronously updating the weight parameters of the layer. The present invention improves retrieval accuracy through cross-modal feature fusion, optimizes data storage and retrieval efficiency by adopting a hierarchical hybrid index structure, and dynamically adjusts index weights in combination with user behavior feedback, thereby achieving efficient, accurate, and intelligent video retrieval.

Description

Technical Field

[0001] The present invention relates to the field of intelligent information retrieval technology, and in particular to an intelligent database video retrieval method based on big data technology. Background Art

[0002] With the rapid growth of video data, intelligent database video retrieval technology has become an important research direction in the field of big data processing; existing video retrieval methods mainly rely on keyword matching or index structures based on unimodal features. This method is difficult to effectively capture the deep semantic information of video content, resulting in low relevance and accuracy of retrieval results; in addition, video data usually contains multimodal information such as vision, audio and text, and there are complex correlations between different modalities, and traditional retrieval methods find it difficult to fully utilize this multimodal information for effective matching; in addition, due to the huge amount of video data, how to improve the system response speed while ensuring retrieval accuracy has also become a problem that needs to be solved urgently in the field of intelligent database video retrieval.

[0003] Existing technologies commonly encounter the following difficulties in video retrieval: First, multimodal information fusion methods lack an effective dynamic weight adjustment mechanism, resulting in uneven contributions of different modalities to retrieval results, affecting matching accuracy. Second, the design of the index structure struggles to balance the rapid update of highly popular videos with the long-term storage of less popular videos, impacting retrieval efficiency. Third, user behavior data is not effectively fed back into the retrieval system, resulting in retrieval results that cannot be optimized based on user preferences, impacting the user experience. Therefore, an intelligent database video retrieval method based on big data technology is urgently needed to address these issues. Summary of the Invention

[0004] Based on the above objectives, the present invention provides an intelligent database video retrieval method based on big data technology.

[0005] An intelligent database video retrieval method based on big data technology includes the following steps:

[0006] S1: Perform spatiotemporal slicing on the original video stream to generate video clip units with timestamps;

[0007] S2: extract visual feature vectors, audio waveform vectors, and text description vectors from the video clip units generated in S1 to generate a multimodal feature vector group;

[0008] S3: Build a cross-modal association matrix based on user historical behavior data, and dynamically weight the multimodal feature vector group to generate a semantic feature vector;

[0009] S4: Build a hierarchical hybrid index structure based on the video heat value. The hierarchical hybrid index structure includes a real-time update layer and a static storage layer, and then write the semantic feature vector generated by S3 into the real-time update layer or the static storage layer;

[0010] S5: After receiving the user's search request, the candidate video set is screened based on the hierarchical hybrid index structure, and the matching degree between the semantic feature vectors of the candidate video set and the search request is calculated using the improved cosine similarity algorithm to generate the search results. At the same time, the user's operation behavior data on the search results is recorded;

[0011] S6: Update the cross-modal association matrix based on the recorded user operation behavior data, and synchronously trigger the weight parameters of the real-time update layer in the hierarchical hybrid index structure to optimize the accuracy of subsequent retrieval results.

[0012] Optionally, the S1 specifically includes:

[0013] S11: Detecting scene switching points based on the color histogram difference between video frames. Calculating a three-dimensional histogram of the HSV color space for consecutive video frames. When the Bhattacharyya coefficient difference between adjacent frames exceeds a set threshold, it is determined to be a scene switching point.

[0014] S12: calculating the sum of the gradient amplitudes of each frame within a range of N frames before and after the detected scene switching point, and selecting the frame with the largest gradient sum as the key frame, where N is a preset sliding window size;

[0015] S13: With the key frame as the center, the time window is expanded to both sides. The initial time window length is set to T seconds. The time window boundary is dynamically adjusted according to the distance between the scene switching points on both sides of the key frame. The specific adjustments include:

[0016] If the distance between the left adjacent scene switching point and the current key frame is less than T / 2, adjust the left boundary to the position of the left scene switching point;

[0017] If the distance between the right adjacent scene switching point and the current key frame is less than T / 2, adjust the right boundary to the position of the right scene switching point;

[0018] S14: intercepting the video stream within the adjusted time window to generate a video segment unit with a start timestamp and an end timestamp, controlling the timestamp accuracy to millisecond level, and marking the frame number index of the key frame to which it belongs.

[0019] Optionally, the S2 specifically includes:

[0020] S21: Process the video frame data in each video clip unit generated by S1 using the ResNet-50 network structure, and extract a fixed-dimensional visual feature vector V after normalizing, resizing, and data augmenting the video frame.

[0021] S22: For the audio data in the video clip unit, first use short-time Fourier transform to perform frame processing on the audio signal, and then use Mel-frequency cepstral coefficient extraction algorithm to calculate the 13-dimensional coefficients of each frame of audio, so as to obtain the audio waveform vector M;

[0022] S23: For the text information in the video clip unit, first use optical character recognition technology to identify and extract the text information in the image, and then perform semantic encoding on the identified text to obtain a fixed-dimensional text description vector W;

[0023] S24: first normalize the visual feature vectors, audio waveform vectors, and text description vectors obtained in S21, S22, and S23, respectively; then concatenate the vectors according to a predetermined dimensional ratio to generate a multimodal feature vector group.

[0024] Optionally, the S24 specifically includes:

[0025] S241: performing L2 normalization processing on the visual feature vector, audio waveform vector, and text description vector obtained in S21, S22, and S23, respectively, to obtain a normalized visual feature vector V′, an audio waveform vector M′, and a text description vector W′;

[0026] S242: Assuming the total dimension of the multimodal feature vector group finally generated is D, the dimension distribution of each modal feature vector is determined according to the ratio of 5:3:2;

[0027] S243: Dimensionally adjust the normalized visual feature vector V′, audio waveform vector M′, and text description vector W′ to match the target dimension;

[0028] S244: The adjusted visual feature vector V″, audio waveform vector M″ and text description vector W″ are sequentially concatenated to generate a final multimodal feature vector group F.

[0029] Optionally, the S3 specifically includes:

[0030] S31: Perform statistical processing on the user's historical behavior data. Let the user interaction score collected for visual, audio and text modalities in the kth record of the user's historical behavior data be x. V,k ,x M,k and x W,k , and calculate the average score of each mode and record it as and Then, based on the user interaction scores between each modality, the Pearson correlation coefficient is used to construct the cross-modal correlation matrix A;

[0031] S32: Calculate the weight parameters of each modality based on the correlation coefficients among vision, audio, and text in the cross-modal association matrix A; wherein the visual modality weight parameter α is calculated as follows: The audio modal weight parameter β is calculated as follows: The text modality weight parameter γ is calculated as follows: The sum of all weight parameters is 1;

[0032] S33: Using the dynamic weight coefficient, perform weighted fusion on the feature vectors of each modality to generate a semantic feature vector S. The formula is: S = αV″ + βM″ + γW″, where S is the final generated semantic feature vector.

[0033] Optionally, the S4 specifically includes:

[0034] S41: For each video clip unit, calculate the video heat value H according to a fixed weight;

[0035] S42: Set a preset threshold T and compare the calculated H with T to determine the index layer to which it belongs; define the calculation formula of the index layer identifier I as follows: Wherein, I=1 indicates that the video clip unit is classified into the real-time update layer, and I=0 indicates that it is classified into the static storage layer;

[0036] S43: Constructing a hierarchical hybrid index structure including a real-time update layer and a static storage layer; the real-time update layer supports incremental write operations, and the static storage layer is used to store video clip units with low popularity;

[0037] S44: Based on the value of I, the semantic feature vector is written into the corresponding index layer.

[0038] Optionally, the S5 specifically includes:

[0039] S51: receiving a search request input by a user and converting the search request into a text description vector;

[0040] S52: Based on the constructed hierarchical hybrid index structure, the real-time update layer and the static storage layer are searched separately, and candidate video sets that meet the conditions are selected from each layer according to the video popularity value, timestamp and preset index screening conditions;

[0041] S53: extracting the semantic feature vector stored in each video clip unit in the candidate video set, and calculating the matching degree with the semantic query vector corresponding to the search request using an improved cosine similarity algorithm. Then, sorting the vectors according to the similarity values, screening out the video clips with the highest matching degree, and generating the final search results.

[0042] S54: Record the user's click, play, like and comment behaviors and store them in a historical behavior dataset.

[0043] Optionally, the S52 specifically includes:

[0044] S521: Define a set of conditions for screening candidate video clip units, including the video heat value threshold T H , timestamp range [T min , T max ] and preset index screening condition C;

[0045] S522: For each video clip unit stored in the index structure, calculate its screening score R;

[0046] S523: Based on the screening score R calculated in S522, screen out the video clip units that meet the conditions in the real-time update layer and the static storage layer. If S≥T S , then the video clip unit is added to the candidate video set, where T S The preset screening threshold.

[0047] Optionally, the S53 specifically includes:

[0048] S531: extracting the semantic feature vector stored in each video segment unit from the candidate video set, and converting the search request input by the user into a semantic query vector;

[0049] S532: Using an improved cosine similarity algorithm, a matching degree is calculated between the semantic feature vector of each candidate video segment unit and the semantic query vector. The improved cosine similarity algorithm introduces dynamic weight adjustment to eliminate the influence differences between different modalities, thereby measuring the semantic relevance between each candidate video segment and the search request.

[0050] S533: sorting all candidate video clip units from high to low according to the matching degree;

[0051] S534: Select the video clip unit with the highest matching degree in the sorted results according to the predetermined return quantity as the final search result, and return it to the user.

[0052] Optionally, the S6 specifically includes:

[0053] S61: Statistically analyzing the recorded user click, play, like, and comment behavior data, categorizing the interaction scores for visual, audio, and text modalities within each video clip unit, and calculating the latest correlation coefficient between each modality using a weighted statistical method;

[0054] S62: using the latest correlation coefficients to update the cross-modal correlation matrix, replacing the parameters in the original matrix that reflect the correlation coefficients between the visual, audio, and text modalities with determined values ​​calculated through statistical analysis, thereby forming a new cross-modal correlation matrix;

[0055] S63: Based on the correlation coefficients of each modality in the updated cross-modal association matrix, the weight parameters corresponding to each modality in the layer are updated in real time according to the predetermined mapping relationship, and the corresponding weight parameters are synchronously transmitted to the real-time update layer in the hierarchical hybrid index structure, triggering immediate adjustment of the weight parameters in the real-time update layer to ensure that the weight parameters in subsequent writing and retrieval processes are consistent with the latest user behavior data.

[0056] Beneficial effects of the present invention:

[0057] The present invention, by constructing a cross-modal association matrix and adopting a dynamic weight fusion method, can effectively integrate the visual, audio and text features of a video, thereby improving the matching accuracy of multimodal information. By using an improved cosine similarity algorithm combined with a dynamic weight adjustment mechanism, the contribution ratio of each modal feature in the retrieval process is made more reasonable, thereby improving the accuracy of video retrieval. At the same time, the present invention updates the cross-modal association matrix based on user historical behavior data, enabling the system to continuously optimize the feature fusion method, making the retrieval results more in line with user needs.

[0058] The present invention constructs a hierarchical hybrid index structure to classify and store video data according to popularity values, thereby achieving rapid retrieval of high-popularity videos and efficient storage of low-popularity videos, and adopts an incremental update mechanism in the real-time update layer to improve the response speed of retrieval; in addition, through the feedback of user behavior data, the weight parameters in the index structure are dynamically adjusted, enabling the system to optimize subsequent retrieval strategies based on the user's actual interactive behavior, thereby improving the intelligence level of video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 A schematic diagram of an intelligent database video retrieval method according to an embodiment of the present invention;

[0061] Figure 2 Schematic diagram of the process of generating search results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It is also noted that, to provide a more detailed description, the following embodiments are best and preferred embodiments, and those skilled in the art may employ alternative methods for implementing certain known technologies. Furthermore, the accompanying drawings are intended only to provide a more detailed description of the embodiments and are not intended to limit the present invention.

[0063] It should be noted that references in the specification to "one embodiment," "an embodiment," "an exemplary embodiment," "some embodiments," etc. indicate that the described embodiments may include specific features, structures, or characteristics, but not every embodiment necessarily includes such specific features, structures, or characteristics. In addition, when specific features, structures, or characteristics are described in conjunction with an embodiment, it is within the knowledge of persons skilled in the relevant art to implement such features, structures, or characteristics in conjunction with other embodiments (whether or not explicitly described).

[0064] In general, terms can be understood, at least in part, from their use in context. For example, depending at least in part on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending at least in part on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0065] like Figure 1-Figure 2 As shown, an intelligent database video retrieval method based on big data technology includes the following steps:

[0066] S1: Perform spatiotemporal slicing on the original video stream to generate video clip units with timestamps;

[0067] S2: extract visual feature vectors, audio waveform vectors, and text description vectors from the video clip units generated in S1 to generate a multimodal feature vector group;

[0068] S3: Build a cross-modal association matrix based on user historical behavior data, and dynamically weight the multimodal feature vector group to generate a semantic feature vector;

[0069] S4: Builds a hierarchical hybrid index structure based on the video popularity value. The hierarchical hybrid index structure includes a real-time update layer and a static storage layer. The semantic feature vectors generated by S3 are then written to the real-time update layer or the static storage layer. The real-time update layer supports incremental write operations.

[0070] S5: After receiving the user's search request, the candidate video set is screened based on the hierarchical hybrid index structure, and the matching degree between the semantic feature vectors of the candidate video set and the search request is calculated using the improved cosine similarity algorithm to generate the search results. At the same time, the user's operation behavior data on the search results is recorded;

[0071] S6: Update the cross-modal association matrix based on the recorded user operation behavior data, and synchronously trigger the weight parameters of the real-time update layer in the hierarchical hybrid index structure to optimize the accuracy of subsequent retrieval results.

[0072] S1 specifically includes:

[0073] S11: Detecting scene switching points based on the color histogram difference between video frames. Calculating a three-dimensional histogram of the HSV color space for consecutive video frames. When the Bhattacharyya coefficient difference between adjacent frames exceeds a set threshold, it is determined to be a scene switching point.

[0074] The calculation formula of the Bhattacharyya coefficient difference is as follows: Among them, D Bhattacharyya is the difference of Bhattacharyya coefficient; H1(i) and H2(i) represent the probability density of adjacent frames in the i-th dimension histogram distribution; n is the histogram dimension. In this scheme, the HSV color space includes hue H, saturation S, and lightness V, each of which is divided into 16 intervals, that is, n = 16 × 16 × 16; when D Bhattacharrya When >0.3, the frame is determined to be a scene switching point;

[0075] S12: Calculate the sum of the gradient amplitudes of each frame within the N frames before and after the detected scene switching point, and select the frame with the largest gradient sum as the key frame, where N is the preset sliding window size. The formula for calculating the sum of the gradient amplitudes is as follows: Among them, G sum is the sum of the gradient amplitudes of the current frame; I is the grayscale value of the frame image; x, y are the pixel coordinates; and are the horizontal and vertical gradients calculated by the Sobel operator respectively; the sliding window size N is set according to the video frame rate FPS, and the formula is: in, Indicates rounding down;

[0076] S13: With the key frame as the center, the time window is expanded to both sides. The initial time window length is set to T seconds. The time window boundary is dynamically adjusted according to the distance between the scene switching points on both sides of the key frame. The specific adjustments include:

[0077] If the distance between the left adjacent scene switching point and the current key frame is less than T / 2, adjust the left boundary to the position of the left scene switching point;

[0078] If the distance between the right adjacent scene switching point and the current key frame is less than T / 2, adjust the right boundary to the position of the right scene switching point;

[0079] S14: The video stream within the adjusted time window is intercepted to generate a video segment unit with a start timestamp and an end timestamp, the timestamp accuracy is controlled to the millisecond level, and the frame number index of the key frame to which it belongs is marked; the timestamp adopts the ISO 8601 extended format and is recorded in the form of YYYY-MM-DDThh:mm:ss.sssZ; the index is a globally unique identifier associated with the key frame of each segment unit, in the format of <video ID>_<key frame sequence number>.

[0080] S2 specifically includes:

[0081] S21: Process the video frame data in each video clip unit generated by S1 using the ResNet-50 network structure, and extract a fixed-dimensional visual feature vector V after normalizing, resizing, and data augmenting the video frame.

[0082] The specific steps are as follows:

[0083] Normalize the video frame so that the pixel value range is mapped to 0,1. The normalization formula is as follows: Among them, P′ represents the normalized video frame pixel value, P is the original pixel value, and P min and P max Represent the minimum and maximum pixel values ​​respectively;

[0084] The normalized frames are resized to make all input frames of the same size. Assume that the input frame size is W×H, and after normalization, it is uniformly adjusted to W′×H′. The adjustment formula is as follows: W′=W×S and H′=H×S, where S represents the scaling factor, which is calculated as: Among them, W t and H t Standard size width and height respectively;

[0085] The ResNet-50 network is used for feature extraction. The standardized frame is input into the convolution layer of ResNet-50, and the visual feature vector V of fixed dimension is output. The expression is: V = F ResNet (I), where I is the input frame image matrix, F ResNet (·) represents the ResNet-50 network processing function, and V is the final extracted visual feature vector.

[0086] S22: For the audio data in the video clip unit, first use the short-time Fourier transform (STFT) to frame the audio signal, and then use the Mel-frequency cepstral coefficient (MFCC) extraction algorithm to calculate the 13-dimensional coefficients of each frame of audio, thereby obtaining the audio waveform vector M;

[0087] The specific steps include:

[0088] First, perform STFT transformation on the input audio signal to obtain spectrum data S(f, t), which is calculated as follows: Where S(f, t) is the time-frequency matrix, x(n) represents the discrete audio signal, w(n) is the window function, N is the window length, f is the frequency index, and t is the time frame index;

[0089] Then, the Mel-frequency cepstral coefficients (MFCC) are calculated using the following formula:

[0090] Among them, M k is the k-th dimension MFCC feature, H k (f) is the weight function of the Mel filter bank, and F is the number of filters.

[0091] S23: For the text information in the video clip unit, first use optical character recognition (OCR) technology to identify and extract the text information in the image, and then perform semantic encoding on the identified text to obtain a fixed-dimensional text description vector W;

[0092] S24: First, the visual feature vectors, audio waveform vectors and text description vectors obtained in S21, S22 and S23 respectively are normalized; then the vectors are spliced ​​according to a predetermined dimensional ratio to generate a multimodal feature vector group, and the timestamp index of the source video clip unit is attached; through the above steps, the feature extraction, normalization and fixed-ratio splicing of each modal data realize the precise quantitative expression of multimodal features, ensure the efficient fusion of data at the same scale, thereby improving the accuracy and response speed of the video retrieval system.

[0093] S24 specifically includes:

[0094] S241: Perform L2 normalization on the visual feature vector, audio waveform vector, and text description vector obtained in S21, S22, and S23, respectively, to obtain a normalized visual feature vector V′, audio waveform vector M′, and text description vector W′. The normalization calculation formula is as follows: Among them, ||·|| represents the Euclidean norm calculation;

[0095] S242: Assume that the total dimension of the multimodal feature vector group finally generated is D, and determine the dimension distribution of each modal feature vector according to the ratio of 5:3:2. The calculation formula is: and D W =D-(D V +D M ), where D V ,D M and D W Represent the target dimensions of visual features, audio waveform features and text description features respectively, represents the rounding down operation to ensure that the sum of the dimensions of each vector is consistent with D;

[0096] S243: Adjust the dimensions of the normalized visual feature vector V′, audio waveform vector M′, and text description vector W′ to match the target dimension. Specifically, a linear projection transformation matrix is ​​used for dimensionality reduction. The calculation formulas are: V″=W V V′, M″=W M M′ and W″=W W W′, where and are the dimension reduction projection matrices of visual, audio and text features respectively, V″, M″ and W″ are the adjusted target dimension feature vectors;

[0097] S244: The adjusted visual feature vector V″, audio waveform vector M″ and text description vector W″ are spliced ​​in sequence to generate the final multimodal feature vector group F, which is expressed as: F = [V″; M″; W″], where [·; ·; ·] represents the concatenation operation of the vectors, so that the dimension of the final feature vector group F is D; through the above steps, the unified normalization processing, proportional distribution, dimensionality adjustment and splicing of the visual, audio and text feature vectors are achieved, ensuring the structural consistency of the multimodal feature vector group, and optimizing the subsequent video retrieval process in combination with the timestamp index.

[0098] S3 specifically includes:

[0099] S31: Perform statistical processing on the user's historical behavior data. Let the user interaction score collected for visual, audio and text modalities in the kth record of the user's historical behavior data be x. V,k ,x M,k and x W,k (where k = 1, 2, ..., L, L is the total number of records), and the average score of each modality is calculated and recorded as and Then, based on the user interaction scores between each modality, the Pearson correlation coefficient is used to construct the cross-modal correlation matrix A, where the correlation coefficient r between vision and audio is VM The calculation formula is:

[0100] Correlation coefficient r between vision and text WW and the correlation coefficient r between audio and text MW The same form is used for calculation respectively; the expression of the cross-modal association matrix A is: Among them, the symmetric relationship holds, that is, r VM =e MV 、r VW =r WV and r MW =r WM ; Each element in the matrix is ​​a fixed value;

[0101] S32: Calculate the weight parameters of each modality based on the correlation coefficients among vision, audio, and text in the cross-modal association matrix A; wherein the visual modality weight parameter α is calculated as follows: The audio modal weight parameter β is calculated as follows: The text modality weight parameter γ is calculated as follows: The sum of all weight parameters is 1;

[0102] S33: Based on the visual feature vector V″, audio waveform vector M″ and text description vector W″ obtained in S24 after L2 normalization and dimension adjustment, the dynamic weight coefficient is used to perform weighted fusion on the feature vectors of each modality to generate a semantic feature vector S. The formula is: S = αV″+βM″+γW″, where S is the final semantic feature vector, and each of its components is a fixed value; through the above steps, the constructed cross-modal association matrix can accurately reflect the intrinsic correlation between the various modalities in the user's historical behavior data, and the dynamic weight fusion mechanism ensures the organic integration of multimodal features, so that the generated semantic feature vector improves the accuracy and response speed of video retrieval.

[0103] S4 specifically includes:

[0104] S41: For each video clip unit, calculate the video heat value H according to the fixed weight; the calculation formula is: H = aV1 + bL + cC, where V1 represents the number of views of the video, L represents the number of likes of the video, and C represents the number of comments on the video; a, b, and c are statistically determined positive coefficients and are all fixed values;

[0105] S42: Set a preset threshold T and compare the calculated H with T to determine the index layer to which it belongs; define the calculation formula of the index layer identifier I as follows: Wherein, I=1 indicates that the video clip unit is classified into the real-time update layer, and I=0 indicates that it is classified into the static storage layer;

[0106] S43: Constructing a hierarchical hybrid index structure including a real-time update layer and a static storage layer; the real-time update layer supports incremental write operations, and the static storage layer is used to store video clip units with low popularity;

[0107] S44: Based on the value of I, the semantic feature vector is written into the corresponding index layer; through the above steps, a hierarchical hybrid index structure based on the video heat value is constructed, and the semantic feature vector generated by S3 is classified according to the heat value and written into the real-time update layer or static storage layer, thereby improving the real-time response capability and data management efficiency of the video retrieval system.

[0108] S5 specifically includes:

[0109] S51: receiving a search request input by a user and converting the search request into a text description vector;

[0110] S52: Based on the constructed hierarchical hybrid index structure, the real-time update layer and the static storage layer are searched separately, and candidate video sets that meet the conditions are selected from each layer according to the video popularity value, timestamp and preset index screening conditions;

[0111] S53: Extracting the semantic feature vectors stored in each video clip unit in the candidate video set, and calculating the matching degree with the semantic query vector corresponding to the search request using an improved cosine similarity algorithm. Then, sorting the video clips based on the similarity values, screening out the video clips with the highest matching degree, and generating the final search results. The improved cosine similarity algorithm explicitly introduces a dynamic weight adjustment parameter based on the traditional cosine similarity to eliminate the influence differences between the features of each modality and achieve accurate comparison of the semantic information of each modality.

[0112] S54: Record the user's click, play, like and comment behaviors and store them in the historical behavior dataset to provide a basis for updating the cross-modal association matrix in the subsequent step S6; the above steps vectorize the text of the user's retrieval request, screen the candidate video set based on the hierarchical hybrid index structure, and match and sort using the improved cosine similarity algorithm, effectively improving the accuracy and response speed of video retrieval and ensuring the relevance and timeliness of the retrieval results.

[0113] S52 specifically includes:

[0114] S521: Define a set of conditions for screening candidate video clip units, including the video heat value threshold T H , timestamp range [T min , T max ] and preset index screening condition C; wherein, the video heat value threshold T H Used to filter high-heat videos; timestamp range [T min , T max] The video release or update time must be within this range; the preset index screening conditions C include video categories, user preference tags and other specific search restrictions;

[0115] S522: For each video clip unit stored in the index structure, calculate its screening score R, which is calculated as follows: R = w H H+w T ·R T +w C ·R C , where H is the heat value of the video clip, which is calculated according to step S41; R T Score the timestamp relevance, calculated as follows:

[0116] Among them, T video Represents the timestamp of the video clip, ensuring that its temporal relevance score is in the range of 0,1; R C The index screening condition matching degree is calculated as follows: Among them, C video is the label set of the video clip, |C| represents the number of labels in the filter condition, |C∩C video | indicates the number of tags that meet the filtering conditions; w H ,w T and w C are the weight coefficients of video popularity, timestamp relevance, and index screening conditions, respectively, and their sum satisfies: w H +w T +w C =1;

[0117] S523: Based on the screening score R calculated in S522, screen out the video clip units that meet the conditions in the real-time update layer and the static storage layer. If S≥T S , then the video clip unit is added to the candidate video set, where T S The preset screening threshold ensures that only the most relevant video clips are selected as the candidate video set; the above steps achieve accurate screening of the candidate video set through quantitative calculation of video heat value, timestamp and preset index screening conditions, effectively improving the accuracy of the retrieval candidate range and the response speed of the system.

[0118] S53 specifically includes:

[0119] S531: extracting the semantic feature vector stored in each video segment unit from the candidate video set, and converting the search request input by the user into a semantic query vector;

[0120] S532: Using an improved cosine similarity algorithm, a matching degree is calculated between the semantic feature vector of each candidate video segment unit and the semantic query vector. The improved cosine similarity algorithm introduces dynamic weight adjustment to eliminate the influence differences between different modalities, thereby measuring the semantic relevance between each candidate video segment and the search request.

[0121] S533: sorting all candidate video clip units from high to low according to the matching degree, so that the video clips with higher matching degrees are ranked at the top;

[0122] S534: Select the video clip unit with the highest matching degree in the sorted results according to the predetermined return quantity as the final search result, and return it to the user.

[0123] The calculation process to generate the final search results is as follows:

[0124] S531: For each video clip unit in the candidate video set, extract the stored semantic feature vector and record it as S i ; At the same time, the user search request is converted into a fixed-dimensional semantic query vector denoted as Q through a predetermined semantic encoding module, where i is the index of the candidate video clip unit;

[0125] S532: Using an improved cosine similarity algorithm, calculate the matching degree between each candidate video segment unit and the semantic query vector. The formula is: Among them, C i represents the improved cosine similarity between the candidate video segment unit i and the semantic query vector; S ij and Q j are vectors S i and the jth component in Q; λ j is the weight coefficient corresponding to the jth dimension, which is determined by the dynamic weight fusion process of multimodal features and is a fixed constant; D is the total dimension of the semantic feature vector;

[0126] S533: The similarity C of all candidate video clip units is calculated. i Arrange in descending order, and based on the sorting results and a predetermined return quantity K, select the top K candidate video clip units with the highest similarity as the final retrieval results and return them to the user.

[0127] S6 specifically includes:

[0128] S61: Statistically analyzing the recorded user click, play, like, and comment behavior data, categorizing the interaction scores for visual, audio, and text modalities within each video clip unit, and calculating the latest correlation coefficient between each modality using a weighted statistical method;

[0129] S62: using the latest correlation coefficients to update the cross-modal correlation matrix, replacing the parameters in the original matrix that reflect the correlation coefficients between the visual, audio, and text modalities with determined values ​​calculated through statistical analysis, thereby forming a new cross-modal correlation matrix;

[0130] S63: Based on the correlation coefficients of each modality in the updated cross-modal association matrix, the weight parameters corresponding to each modality in the layer are updated in real time according to the predetermined mapping relationship, and the corresponding weight parameters are synchronously transmitted to the real-time update layer in the hierarchical hybrid index structure, triggering immediate adjustment of the weight parameters in the real-time update layer to ensure that the weight parameters in subsequent writing and retrieval processes are consistent with the latest user behavior data; the above steps realize the real-time update of the cross-modal association matrix through precise statistics and weighted calculations of user operation behavior data, and synchronously adjust the weight parameters in the real-time update layer, thereby ensuring that the retrieval results accurately respond to user needs and continuously optimize system performance.

[0131] The present invention encompasses any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention. To provide a thorough understanding of the present invention, specific details are described in detail below in connection with the preferred embodiments of the present invention, but those skilled in the art will be able to fully understand the present invention without these detailed descriptions. Furthermore, to avoid unnecessary confusion regarding the essence of the present invention, well-known methods, processes, procedures, components, and circuits have not been described in detail.

[0132] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. An intelligent database video retrieval method based on big data technology, characterized in that: The following steps are involved: S1: Perform spatiotemporal slicing on the original video stream to generate video clip units with timestamps; S2: extract visual feature vectors, audio waveform vectors, and text description vectors from the video clip units generated in S1 to generate a multimodal feature vector group; S3: Build a cross-modal association matrix based on user historical behavior data, and dynamically weight the multimodal feature vector group to generate a semantic feature vector; The S3 specifically includes: S31: Perform statistical processing on the user's historical behavior data, assuming that the user's historical behavior data In the records, the user interaction scores collected for visual, audio and text modalities are recorded as and , and calculate the average score of each mode and record it as and ; Then, based on the user interaction scores between each modality, the Pearson correlation coefficient is used to construct a cross-modal correlation matrix ; S32: According to the cross-modal correlation matrix The correlation coefficients between vision, audio and text are used to calculate the weight parameters of each modality; among them, the visual modality weight parameter Calculate as follows: ; Audio modal weight parameters Calculate as follows: ; Text modality weight parameter Calculate as follows: ;The sum of all weight parameters is 1; S33: Use dynamic weight coefficients to perform weighted fusion on the feature vectors of each modality to generate a semantic feature vector , the formula is: ,in, This is the final generated semantic feature vector. is the visual feature vector, is the audio waveform vector, is the text description vector; S4: Build a hierarchical hybrid index structure based on the video heat value. The hierarchical hybrid index structure includes a real-time update layer and a static storage layer, and then write the semantic feature vector generated by S3 into the real-time update layer or the static storage layer; The S4 specifically includes: S41: For each video clip unit, calculate the video heat value according to the fixed weight ; S42: Set a preset threshold , and the calculated and Compare to determine the index layer to which it belongs; define the index layer identifier The calculation formula is: ,in, Indicates that the video segment unit belongs to the real-time update layer. Indicates that it is classified into the static storage layer; S43: Constructing a hierarchical hybrid index structure including a real-time update layer and a static storage layer; the real-time update layer supports incremental write operations, and the static storage layer is used to store video clip units with low popularity; S44: Based on The value of , writes the semantic feature vector into the corresponding index layer; S5: After receiving the user's search request, the candidate video set is screened based on the hierarchical hybrid index structure, and the matching degree between the semantic feature vectors of the candidate video set and the search request is calculated using the improved cosine similarity algorithm to generate the search results. At the same time, the user's operation behavior data on the search results is recorded; S6: Update the cross-modal association matrix based on the recorded user operation behavior data, and synchronously trigger the weight parameters of the real-time update layer in the hierarchical hybrid index structure to optimize the accuracy of subsequent retrieval results.

2. The intelligent database video retrieval method based on big data technology according to claim 1 is characterized in that: Said S1 specifically includes: S11: Detecting scene switching points based on the color histogram difference between video frames. Calculating a three-dimensional histogram of the HSV color space for consecutive video frames. When the Bhattacharyya coefficient difference between adjacent frames exceeds a set threshold, it is determined to be a scene switching point. S12: calculating the sum of the gradient amplitudes of each frame within a range of N frames before and after the detected scene switching point, and selecting the frame with the largest gradient sum as the key frame, where N is a preset sliding window size; S13: With the key frame as the center, the time window is expanded to both sides. The initial time window length is set to T seconds. The time window boundary is dynamically adjusted according to the distance between the scene switching points on both sides of the key frame. The specific adjustments include: If the distance between the left adjacent scene switching point and the current key frame is less than T / 2, adjust the left boundary to the position of the left scene switching point; If the distance between the right adjacent scene switching point and the current key frame is less than T / 2, adjust the right boundary to the position of the right scene switching point; S14: intercepting the video stream within the adjusted time window to generate a video segment unit with a start timestamp and an end timestamp, controlling the timestamp accuracy to millisecond level, and marking the frame number index of the key frame to which it belongs.

3. The intelligent database video retrieval method based on big data technology according to claim 1 is characterized in that: The S2 specifically includes: S21: The video frame data in each video clip unit generated by S1 is processed using the ResNet-50 network structure, and after normalizing, resizing and data enhancement of the video frame, a fixed-dimensional visual feature vector is extracted. ; S22: For the audio data in the video clip unit, first use the short-time Fourier transform to frame the audio signal, and then use the Mel frequency cepstral coefficient extraction algorithm to calculate the 13-dimensional coefficients of each frame of audio to obtain the audio waveform vector ; S23: For the text information in the video clip unit, first use optical character recognition technology to identify and extract the text information in the image, and then perform semantic encoding on the recognized text to obtain a fixed-dimensional text description vector ; S24: first normalize the visual feature vectors, audio waveform vectors, and text description vectors obtained in S21, S22, and S23, respectively; then concatenate the vectors according to a predetermined dimensional ratio to generate a multimodal feature vector group.

4. The intelligent database video retrieval method based on big data technology according to claim 3 is characterized in that: The S24 specifically includes: S241: Perform L2 normalization on the visual feature vector, audio waveform vector, and text description vector obtained in S21, S22, and S23, respectively, to obtain the normalized visual feature vector , audio waveform vector and text description vector ; S242: Assume that the total dimension of the multimodal feature vector group finally generated is ,according to The ratio of determines the dimension allocation of each modal eigenvector; S243: Normalized visual feature vector , audio waveform vector and text description vector Adjust the dimension to match the target dimension; S244: The adjusted visual feature vector , audio waveform vector and text description vector Splice in sequence to generate the final multimodal feature vector group .

5. The intelligent database video retrieval method based on big data technology according to claim 1 is characterized in that: The S5 specifically includes: S51: receiving a search request input by a user and converting the search request into a text description vector; S52: Based on the constructed hierarchical hybrid index structure, the real-time update layer and the static storage layer are searched separately, and candidate video sets that meet the conditions are selected from each layer according to the video popularity value, timestamp and preset index screening conditions; S53: extracting the semantic feature vector stored in each video clip unit in the candidate video set, and calculating the matching degree with the semantic query vector corresponding to the search request using an improved cosine similarity algorithm. Then, sorting the vectors according to the similarity values, screening out the video clips with the highest matching degree, and generating the final search results. S54: Record the user's click, play, like and comment behaviors and store them in a historical behavior dataset.

6. The intelligent database video retrieval method based on big data technology according to claim 5 is characterized in that: The S52 specifically includes: S521: Define a set of conditions for screening candidate video clip units, including a video heat value threshold , timestamp range and preset index filtering conditions ; S522: For each video segment unit stored in the index structure, calculate its screening score ; S523: Screening score calculated based on S522 , filter out the video clip units that meet the conditions in the real-time update layer and the static storage layer. If , then the video clip unit is added to the candidate video set, where The preset screening threshold.

7. The intelligent database video retrieval method based on big data technology according to claim 6 is characterized in that: The S53 specifically includes: S531: extracting the semantic feature vector stored in each video segment unit from the candidate video set, and converting the search request input by the user into a semantic query vector; S532: Using an improved cosine similarity algorithm, a matching degree is calculated between the semantic feature vector of each candidate video segment unit and the semantic query vector. The improved cosine similarity algorithm introduces dynamic weight adjustment to eliminate the influence differences between different modalities, thereby measuring the semantic relevance between each candidate video segment and the search request. S533: sorting all candidate video clip units from high to low according to the matching degree; S534: Select the video clip unit with the highest matching degree in the sorted results according to the predetermined return quantity as the final search result, and return it to the user.

8. The intelligent database video retrieval method based on big data technology according to claim 1 is characterized in that: The S6 specifically includes: S61: Statistically analyzing the recorded user click, play, like, and comment behavior data, categorizing the interaction scores for visual, audio, and text modalities within each video clip unit, and calculating the latest correlation coefficient between each modality using a weighted statistical method; S62: using the latest correlation coefficients to update the cross-modal correlation matrix, replacing the parameters in the original matrix that reflect the correlation coefficients between the visual, audio, and text modalities with determined values ​​calculated through statistical analysis, thereby forming a new cross-modal correlation matrix; S63: Based on the correlation coefficients of each modality in the updated cross-modal association matrix, the weight parameters corresponding to each modality in the hierarchical hybrid index structure are updated in real time according to the predetermined mapping relationship, and the corresponding weight parameters are synchronously transmitted to the real-time update layer in the hierarchical hybrid index structure, triggering immediate adjustment of the weight parameters in the real-time update layer to ensure that the weight parameters in subsequent writing and retrieval processes are consistent with the latest user behavior data.

Images