Video cloud intelligent AI retrieval method and device and medium

Abstract

The invention discloses a video cloud intelligent AI retrieval method and device and a medium, belongs to the crossing field of video processing, artificial intelligence and cloud computing, and aims to solve the technical problems of slow retrieval and poor matching of massive videos, reduce cloud resource occupation, improve cloud stability and user experience and improve the video retrieval efficiency. According to the technical scheme, an edge node layer preprocesses original video data, an improved MobileNetV3 model is adopted to extract key frames according to a preset frequency, after invalid frames are filtered, low-dimensional visual features and low-dimensional audio features are extracted, and meanwhile edge tags associated with video scenes are generated; transmitting the low-dimensional visual features, the low-dimensional audio features and the edge tags to a video cloud layer; the video cloud layer receives and processes the low-dimensional visual features, the low-dimensional audio features and the edge tags; the video cloud layer constructs a hierarchical index of'edge label-low-dimensional feature-high-dimensional feature '; performing hierarchical retrieval according to the retrieval request; and hierarchical retrieval optimization.

Description

TECHNICAL FIELD

[0001] The application relates to the field of video processing, artificial intelligence and cloud computing, in particular to a video cloud intelligent AI retrieval method, device and medium. BACKGROUND

[0002] With the exponential increase of existing video data, for the retrieval demand of PB-level massive video data, the existing retrieval technology has the following pain points:

[0003] ①Low architecture efficiency: pure cloud full-volume processing, bandwidth occupation is more than 70%, single video preprocessing is 4-6 hours, retrieval delay is more than 15 seconds, and real-time demand cannot be met;

[0004] ②Single feature dimension: only static visual or single audio features are extracted, the spatio-temporal correlation of "action-scene-audio" cannot be captured, and the retrieval accuracy is less than 55% (such as mis-matching "rainy day pedestrian standing" as "rainy day pedestrian running");

[0005] ③Rough fusion method: fixed weight splicing of multi-modal features, without adaptation according to video types (action type / speech type), low feature discrimination, and poor complex semantic matching.

[0006] Therefore, how to overcome the problems of slow massive video retrieval and matching bias, reduce cloud resource occupation, and improve cloud stability and user experience is a technical problem to be solved at present. SUMMARY

[0007] The technical task of the application is to provide a video cloud intelligent AI retrieval method, device and medium to solve the problem of how to overcome the problems of slow massive video retrieval and matching bias, reduce cloud resource occupation, and improve cloud stability and user experience.

[0008] The technical task of the application is realized in the following way, a video cloud intelligent AI retrieval method, which specifically comprises the following steps:

[0009] The edge node layer pre-processes the original video data: an improved MobileNetV3 model is used to extract key frames at a preset frequency, low-dimensional visual features and low-dimensional audio features are extracted after filtering invalid frames, edge labels associated with video scenes are generated, and the low-dimensional visual features, low-dimensional audio features and edge labels are transmitted to the video cloud layer;

[0010] The video cloud layer receives low-dimensional visual features, low-dimensional audio features and edge labels and processes them: the improved 3D-ResNet model is used to extract features of the video segments corresponding to the edge labels to obtain dynamic video features and dynamic audio features, and the weights of the dynamic video features and the dynamic audio features are dynamically allocated based on the video type through the attention mechanism, and the dynamic video features and the dynamic audio features are fused to generate a multi-modal dynamic feature matrix;

[0011] The video cloud layer constructs a hierarchical index of "edge label-low-dimensional feature-high-dimensional feature": the bottom layer of the hierarchical index is an edge label index, the middle layer is a low-dimensional feature index, and the top layer is a high-dimensional feature index corresponding to the multi-modal dynamic feature matrix;

[0012] According to the retrieval request, the hierarchical retrieval is performed: the user terminal layer initiates a retrieval request, the edge node layer matches the edge label according to the retrieval request to filter a candidate video set and transmits it to the video cloud layer, the video cloud layer sequentially completes the middle layer low-dimensional feature matching and the top layer high-dimensional feature similarity calculation through the hierarchical index, sorts to generate a retrieval result and returns it to the user terminal layer;

[0013] Hierarchical retrieval optimization: based on the feedback information of the user on the retrieval result, the weights of the dynamic visual features and the dynamic audio features and the threshold of the high-dimensional feature similarity calculation are adjusted by using a reinforcement learning algorithm to optimize the hierarchical retrieval.

[0014] As preferred, the improvement of the improved MobileNetV3 model is as follows:

[0015] On the input mode, an audio input branch is added to the original MobileNetV3 model to receive a mel spectrum diagram, supporting dual-modal synchronous input of video and audio:

[0016] On the feature extraction, invalid frame filtering and edge label generation (scene-target mapping) are added to the original MobileNetV3 model;

[0017] On the output form, a structured label (such as "security scene with pedestrian") and a low-dimensional feature matrix are used;

[0018] On the computing power adaptation, optimization is made for edge devices (NVIDIA Jetson Nano / ARM chip): single-frame computing power ≤0.8 GFLOPs (20% lower than the original Small version).

[0019] As preferred, the preset frequency is 2 seconds / frame;

[0020] Invalid frames include pure black frames and empty scene frames without dynamic targets, and the filtering proportion of invalid frames is ≥75%;

[0021] The dimension of the low-dimensional visual feature is 256, and the dimension of the low-dimensional audio feature is 128.

[0022] The edge label includes a scene type label and a target type label. The scene type label includes "security scene", "film and television scene", and "short video scene". The target type label includes "containing pedestrians", "containing vehicles", and "containing specific objects".

[0023] As preferred, the improvement of the improved 3D-ResNet model is as follows:

[0024] The input layer adopts "16 frames / group + frame alignment preprocessing", realizes frame sequence grouping and alignment preprocessing, and specifically: frame sequence screening: the key frames (invalid frames have been filtered) transmitted by the input edge node layer are grouped into 16 frames / group (covering 2 seconds of dynamic segments, matching the frame extraction frequency of the edge node layer) in chronological order; frame alignment: a new "optical flow frame alignment layer" is added, which calculates the optical flow vector (such as Farneback optical flow algorithm) of adjacent frames, corrects the inter-frame jitter (such as target displacement caused by slight camera shaking), ensures the continuity of the motion trajectory, and reduces the noise interference of dynamic feature extraction, laying a foundation for subsequent accurate capture of the motion trajectory of the key node.

[0025] Feature fusion: a new "dynamic feature screening layer" is added, which retains 512 channels strongly related to "key node motion and scene dynamics" by using a trained binary mask (based on 50,000+ dynamic scene samples), and clips the remaining redundant channels.

[0026] Output layer: standardize the multi-dimensional features to ensure the uniformity of the length of the feature vector, which is convenient for subsequent cosine similarity calculation; the dimension is reduced by 75% compared with the original, which not only retains the core dynamic information, but also reduces the algorithmic cost of cloud feature fusion and index construction.

[0027] As preferred, the dynamic allocation of the weights of dynamic video features and dynamic audio features based on video types through the attention mechanism is as follows:

[0028] When the video type is an action video, the weight proportion of dynamic visual features is 0.7, and the weight proportion of dynamic audio features is 0.3.

[0029] When the video type is a speech video, the weight proportion of dynamic audio features is 0.6, and the weight proportion of dynamic visual features is 0.4.

[0030] The dimension of the multi-modal dynamic feature matrix is 768, and in the process of generating the multi-modal dynamic feature matrix, privacy encryption processing is also included, specifically: local differential privacy technology is used to add perturbation noise to the sensitive information contained in the dynamic visual features, and a homomorphic encryption algorithm is used to encrypt and store the multi-modal dynamic feature matrix.

[0031] As preferred, the retrieval request comprises a text request, an image request or an audio request;

[0032] The middle-layer low-dimensional feature matching narrows down the candidate video set to within 1% of the original candidate set scale;

[0033] The high-dimensional feature similarity calculation adopts a cosine similarity algorithm;

[0034] The retrieval result is a ranked Top20 video clip, and the retrieval response delay is ≤2 seconds;

[0035] The high-dimensional feature similarity calculation is completed in an encrypted domain, and the sensitive information leakage risk is <0.1%.

[0036] As preferred, the feedback information comprises user-labeled "relevant results", "irrelevant results" and manual top operation;

[0037] The reinforcement learning algorithm takes user satisfaction as a reward function, and the user satisfaction is the proportion of relevant results in the retrieval results; each iteration of 1000 retrieval requests completes once optimization, if the relevant accuracy of any feature is more than 90%, the weight of the corresponding feature is increased by 5%-10%; if the number of retrieval results is <5, the threshold of similarity calculation is reduced by 5%-8%.

[0038] More preferably, the edge node layer is deployed in a video acquisition device, and the video acquisition device comprises a security camera and a short video shooting terminal; the hardware chip carried by the video acquisition device is selected from an NVIDIA Jetson Nano chip and an embedded ARM chip;

[0039] The video cloud layer adopts a distributed computing cluster, and the distributed computing cluster supports FP16 semi-precision quantization processing to optimize the computing efficiency.

[0040] An electronic device comprising a memory and at least one processor;

[0041] The memory has stored thereon a computer program;

[0042] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the video cloud intelligent AI retrieval method as described above.

[0043] A computer-readable storage medium having stored therein a computer program, the computer program being executable by a processor to implement the video cloud intelligent AI retrieval method as described above.

[0044] The video cloud intelligent AI retrieval method, device and medium of the present application have the following advantages:

[0045] (I) This invention adopts a three-layer collaborative architecture of "edge node layer - video cloud layer - user terminal layer". The edge node uses MobileNetV3 to extract frames to filter invalid data, extract low-dimensional features and generate labels. The video cloud layer uses improved 3D-ResNet and CNN-LSTM to improve dimensional dynamic features, dynamically allocates weights according to video type to fuse and generate feature matrices and build a three-layer index. The user terminal layer supports multiple retrieval methods and collects feedback. Combined with reinforcement learning to optimize the model, the retrieval process is completed in four steps. Taking the security scenario as an example, the edge uses NVIDIA Jetson Nano and the cloud uses Alibaba Cloud ECS cluster. Results can be returned in 1.5 seconds and accuracy can be improved through feedback. This solution can reduce the preprocessing cycle to 25 minutes, the retrieval latency to ≤2 seconds, and the accuracy to 92%. It is adaptable to multiple scenarios and has strong compatibility.

[0046] (II) This invention meets the retrieval needs of PB-level massive video data, and solves the pain points of existing technical architectures such as low efficiency (high bandwidth consumption and latency of more than 15 seconds for pure cloud processing), single feature dimension (only static visual or single audio features, accuracy of less than 55%), and coarse fusion method (poor adaptability of fixed weights). It is widely applicable to scenarios such as security monitoring (real-time target positioning) and film and television content management (accurate matching of plot segments), solving the industry problem of "slow retrieval and biased matching", while reducing cloud resource consumption and improving stability and user experience;

[0047] (III) This invention improves data retrieval efficiency: the preprocessing cycle is reduced to 25 minutes (90% reduction), the retrieval delay is ≤2 seconds (87% reduction), and the cloud load is reduced by 50%;

[0048] (iv) This invention improves data retrieval accuracy: the retrieval accuracy rate reaches 92% (an increase of 37 percentage points), and the secondary retrieval rate is reduced to 12%;

[0049] (v) This invention has good adaptability: user satisfaction in multiple scenarios exceeds 90%, and there is no need to refactor the system;

[0050] (vi) This invention has good compatibility: it is compatible with mainstream devices at the edge and connects to Alibaba Cloud / Tencent Cloud in the cloud. Attached Figure Description

[0051] The invention will be further described below with reference to the accompanying drawings.

[0052] Appendix Figure 1 This is a schematic diagram of the video cloud intelligent AI retrieval method. Detailed Implementation

[0053] The video cloud intelligent AI retrieval method, device and medium of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] Example 1:

[0055] As shown in the accompanying drawings Figure 1 The embodiment provides a video cloud intelligent AI retrieval method, and the method specifically comprises the following steps:

[0056] S1, the edge node layer pre-processes original video data: adopting an improved MobileNetV3 model to extract key frames at a preset frequency, extracts low-dimensional visual features and low-dimensional audio features after filtering invalid frames, simultaneously generates edge labels associated with video scenes, and transmits the low-dimensional visual features, the low-dimensional audio features and the edge labels to the video cloud layer;

[0057] S2, the video cloud layer receives the low-dimensional visual features, the low-dimensional audio features and the edge labels and processes them: the improved 3D-ResNet model is used to extract features of video segments corresponding to the edge labels, to obtain dynamic video features and dynamic audio features, and based on the video type, the attention mechanism is used to dynamically allocate the weights of the dynamic video features and the dynamic audio features, and the dynamic video features and the dynamic audio features are fused to generate a multi-modal dynamic feature matrix;

[0058] S3, the video cloud layer constructs a hierarchical index of "edge label-low-dimensional feature-high-dimensional feature": the bottom layer of the hierarchical index is an edge label index, the middle layer is a low-dimensional feature index, and the top layer is a high-dimensional feature index corresponding to the multi-modal dynamic feature matrix;

[0059] S4, hierarchical retrieval according to a retrieval request: the user terminal layer initiates a retrieval request, the edge node layer matches the edge labels according to the retrieval request to filter a candidate video set and transmits the candidate video set to the video cloud layer, the video cloud layer sequentially completes low-dimensional feature matching in the middle layer and high-dimensional feature similarity calculation in the top layer through the hierarchical index, sorts to generate a retrieval result and returns the retrieval result to the user terminal layer;

[0060] S5, hierarchical retrieval optimization: based on feedback information of the user on the retrieval result, a reinforcement learning algorithm is used to adjust the weights of the dynamic visual features and the dynamic audio features and the threshold value of the high-dimensional feature similarity calculation, so as to optimize the hierarchical retrieval.

[0061] The improvement of the improved MobileNetV3 model in step S1 of the embodiment is as follows:

[0062] ①On the input mode, an audio input branch is added to the original MobileNetV3 model to receive a mel spectrum diagram, so as to support the synchronous input of video and audio in a double mode:

[0063] ②On the feature extraction: invalid frame filtering and edge label generation (scene-target mapping) are added to the original MobileNetV3 model;

[0064] ③In output form, structured labels (such as "security scene - contains pedestrians") and low-dimensional feature matrices are used;

[0065] ④In terms of computing power adaptation, optimization is made for edge devices (NVIDIA Jetson Nano / ARM chips): single-frame computing power ≤0.8 GFLOPs (20% lower than the original Small version).

[0066] The original MobileNetV3 cannot directly meet the needs of the "edge multi-modal processing, low-dimensional feature output, and label generation" of the present embodiment, so it is customized and modified as shown in the following table:

[0067]

[0068] The preset frequency in step S1 of the present embodiment is 2 seconds / frame;

[0069] Invalid frames include pure black frames and empty scene frames without dynamic targets, and the filtering ratio of invalid frames is ≥75%;

[0070] The dimension of the low-dimensional visual feature is 256, and the dimension of the low-dimensional audio feature is 128.

[0071] The edge label includes scene type label and target type label, the scene type label includes "security scene", "film scene", and "short video scene", and the target type label includes "contains pedestrians", "contains vehicles", and "contains specific objects".

[0072] The improvement of the improved 3D-ResNet model in step S2 of the present embodiment is as follows:

[0073] The input layer uses "16 frames / group + frame alignment preprocessing" to realize frame sequence grouping and alignment preprocessing, specifically: frame sequence screening: the key frames filtered from invalid frames are transmitted by the edge node layer, and are grouped into 16 frames / group in chronological order (covering 2 seconds of dynamic segments, matching the frame extraction frequency of the edge node layer); frame alignment: a new "optical flow frame alignment layer" is added, which calculates the optical flow vector (such as the Farneback optical flow algorithm) of adjacent frames to correct the inter-frame jitter (such as the target displacement caused by slight camera shaking), ensure the continuity of the motion trajectory, and reduce the noise interference of dynamic feature extraction, laying a foundation for subsequent accurate capture of key node motion trajectories;

[0074] Feature fusion: a new "dynamic feature screening layer" is added, which retains 512 channels strongly related to "key node motion and scene dynamics" by using a trained binary mask (based on 50,000+ dynamic scene samples), and clips the remaining redundant channels;

[0075] Output layer: Standardize multi-dimensional features to ensure the length of feature vectors is uniform, facilitating subsequent cosine similarity calculation; reduce dimension by 75% compared to the original, retaining core dynamic information while reducing the computational overhead of cloud feature fusion and index construction.

[0076] The improvements of the improved 3D-ResNet model are as follows:

[0077]

[0078] The step S2 in the embodiment based on the video type dynamically allocates the weights of the dynamic video features and the dynamic audio features through the attention mechanism, and the specific process is as follows:

[0079] When the video type is an action video, the weight proportion of the dynamic visual features is 0.7, and the weight proportion of the dynamic audio features is 0.3.

[0080] When the video type is a speech video, the weight proportion of the dynamic audio features is 0.6, and the weight proportion of the dynamic visual features is 0.4.

[0081] The dimension of the multi-modal dynamic feature matrix is 768, and in the process of generating the multi-modal dynamic feature matrix, privacy encryption processing is also included, which is specifically: adding perturbation noise to sensitive information contained in the dynamic visual features using local differential privacy technology, and encrypting and storing the multi-modal dynamic feature matrix using a homomorphic encryption algorithm.

[0082] The retrieval request in step S4 in the embodiment includes a text request, an image request, or an audio request.

[0083] The middle-layer low-dimensional feature matching narrows down the candidate video set to within 1% of the original candidate set size.

[0084] The high-dimensional feature similarity calculation uses a cosine similarity algorithm.

[0085] The retrieval result is a sorted Top20 video segment, and the retrieval response delay is ≤2 seconds.

[0086] The high-dimensional feature similarity calculation is completed in the encrypted domain, and the sensitive information leakage risk is <0.1%.

[0087] The feedback information in step S5 in the embodiment includes the user's marked "relevant results" and "irrelevant results" and manual top operation.

[0088] The reinforcement learning algorithm takes user satisfaction as the reward function, and the user satisfaction is the proportion of relevant results in the retrieval results; the optimization is completed once for every 1000 iterations of retrieval requests, if the relevant accuracy of any feature is more than 90%, the weight of the corresponding feature is increased by 5%-10%; if the number of retrieval results is <5, the threshold value of similarity calculation is reduced by 5%-8%.

[0089] The edge node layer in the embodiment is deployed on a video acquisition device, which includes a security camera and a short video shooting terminal. The hardware chip carried by the video acquisition device is selected from an NVIDIA Jetson Nano chip and an embedded ARM chip.

[0090] The video cloud layer adopts a distributed computing cluster, which supports FP16 semi-precision quantization processing to optimize the computing efficiency.

[0091] The edge node layer in the embodiment deploys an improved MobileNetV3 model, extracts 256-dimensional low-dimensional visual features (target outline / color) and 128-dimensional low-dimensional audio features (basic frequency) from key frames extracted at 2 seconds / frame and 75% invalid frames (such as empty scenes), generates edge labels (such as “security-including pedestrians”), and only transmits “low-dimensional features + labels” to the cloud to reduce the bandwidth.

[0092] The working process of the video cloud layer in the embodiment is as follows:

[0093] ① High-dimensional feature extraction: an improved 3D-ResNet model is used to extract 512-dimensional dynamic visual features (joint trajectory / scene dynamics);

[0094] ② Dynamic fusion: weights are assigned according to video types (action class: visual 0.7 / audio 0.3; speech class: audio 0.6 / visual 0.4), and an attention mechanism is used to generate a 768-dimensional multi-modal feature matrix;

[0095] ③ Hierarchical indexing: three-layer indexing of “label→low-dimensional→high-dimensional”, second-level filtering of categories→reduction to 1% candidates→precise matching.

[0096] The user terminal layer in the embodiment supports text / image / audio retrieval and collects “relevant / irrelevant” feedback for retrieval optimization.

[0097] The four-step retrieval process in the embodiment is as follows:

[0098] ① The user terminal sends a request (such as “blue-clothed worker factory area handling”);

[0099] ② The edge node matches the label and filters the candidate set to transmit to the cloud;

[0100] ③ The cloud middle layer performs low-dimensional matching (such as blue cloth), reduces to 100, and the top layer calculates the similarity to rank the top 20.

[0101] The reinforcement learning algorithm in the embodiment takes “user satisfaction” as the reward, increases the feature weight by 5%-10% when the feature accuracy in the relevant results is more than 90% every 1000 iterations, reduces the similarity threshold by 5%-8% when the results are less than 5, and preserves the coverage.

[0102] Embodiment 2:

[0103] The embodiment also provides an electronic device, comprising a memory and a processor.

[0104] The memory stores computer-executable instructions.

[0105] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the video cloud intelligent AI retrieval method in any of the embodiments of the application.

[0106] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0107] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, application programs required by at least one function, etc.; and the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0108] Embodiment 3:

[0109] The embodiment also provides a computer-readable storage medium, wherein a plurality of instructions are stored, the instructions are loaded by a processor to make the processor execute the video cloud intelligent AI retrieval method in any of the embodiments of the application. Specifically, a system or device equipped with a storage medium can be provided, and software program code for realizing the functions of any of the above embodiments is stored on the storage medium, and the computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage medium.

[0110] In this case, the program code read from the storage medium can realize the functions of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute a part of the application.

[0111] The storage medium for providing the program code includes floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, nonvolatile memory cards and ROM. Alternatively, the program code can be downloaded from a server computer through a communication network.

[0112] In addition, it should be clear that not only the program code read by the computer can be executed, but also the operating system and the like operating on the computer can be caused to perform part or all of the actual operations based on the instructions of the program code, thereby realizing the functions of any one of the above embodiments.

[0113] In addition, it should be understood that the program code read by the storage medium is written into the memory provided in the expansion board inserted into the computer or the memory provided in the expansion unit connected to the computer, and then the CPU and the like installed on the expansion board or the expansion unit are caused to perform part or all of the actual operations based on the instructions of the program code, thereby realizing the functions of any one of the above embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can still be modified, or part or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A video cloud intelligent AI retrieval method, characterized in that, The method is as follows: The edge node layer preprocesses the raw video data: it uses an improved MobileNetV3 model to extract key frames at a preset frequency, filters out invalid frames, extracts low-dimensional visual features and low-dimensional audio features, generates edge labels associated with the video scene, and transmits the low-dimensional visual features, low-dimensional audio features and edge labels to the video cloud layer. The video cloud layer receives and processes low-dimensional visual features, low-dimensional audio features, and edge labels: the improved 3D-ResNet model extracts features from the video segments corresponding to the edge labels to obtain dynamic video features and dynamic audio features. Based on the video type, the weights of the dynamic video features and dynamic audio features are dynamically assigned through an attention mechanism. The dynamic video features and dynamic audio features are then fused to generate a multimodal dynamic feature matrix. The video cloud layer constructs a hierarchical index of "edge label - low-dimensional feature - high-dimensional feature": the bottom layer of the hierarchical index is the edge label index, the middle layer is the low-dimensional feature index, and the top layer is the high-dimensional feature index corresponding to the multimodal dynamic feature matrix. Layered retrieval based on retrieval request: The user terminal layer initiates a retrieval request, the edge node layer matches edge tags according to the retrieval request to filter candidate video sets and transmits them to the video cloud layer, the video cloud layer completes the matching of low-dimensional features in the middle layer and the calculation of high-dimensional feature similarity in the top layer in sequence through the layered index, sorts and generates retrieval results and returns them to the user terminal layer; Hierarchical retrieval optimization: Based on user feedback on retrieval results, reinforcement learning algorithms are used to adjust the weights of dynamic visual features and dynamic audio features, as well as the threshold for calculating high-dimensional feature similarity, thereby optimizing hierarchical retrieval.

2. The video cloud intelligent AI retrieval method according to claim 1, characterized in that, The improvements to the MobileNetV3 model are as follows: In terms of input modality, an audio input branch is added to the native MobileNetV3 model to receive Mel spectrograms, supporting simultaneous dual-modal input of video and audio: In terms of feature extraction: invalid frame filtering and edge label generation are added to the original MobileNetV3 model; In terms of output format, structured labels and low-dimensional feature matrices are used; In terms of computing power adaptation, it is optimized for edge devices: single-frame computing power ≤ 0.8 GFLOPs.

3. The video cloud intelligent AI retrieval method according to claim 1, characterized in that, The preset frequency is 2 seconds per frame; Invalid frames include pure black frames and empty scene frames without dynamic targets. The filtering ratio of invalid frames is ≥75%. The low-dimensional visual features have 256 dimensions, and the low-dimensional audio features have 128 dimensions. Edge labels include scene type labels and target type labels. Scene type labels include "security scene", "film and television scene" and "short video scene"; target type labels include "containing pedestrians", "containing vehicles" and "containing specific objects".

4. The video cloud intelligent AI retrieval method according to claim 1, characterized in that, The improvements to the 3D-ResNet model are as follows: The input layer uses "16 frames / group + frame alignment preprocessing" to achieve frame sequence grouping and alignment preprocessing. Specifically, frame sequence filtering: key frames transmitted by the input edge node layer are grouped into 16 frames / groups according to time order. Frame alignment: A new "optical flow frame alignment layer" is added, which corrects inter-frame jitter by calculating the optical flow vectors of adjacent frames, ensuring the continuity of motion trajectory; Feature fusion: A new "dynamic feature filtering layer" is added, which retains 512 channels that are strongly correlated with "joint motion and scene dynamics" through a trained binary mask, and prunes the remaining redundant channels; Output layer: Standardizes multidimensional features to ensure uniform feature vector magnitude.

5. The video cloud intelligent AI retrieval method according to claim 1, characterized in that, Based on video type, the weights of dynamic video features and dynamic audio features are dynamically assigned using an attention mechanism as follows: When the video type is action video, the weight of dynamic visual features is 0.7 and the weight of dynamic audio features is 0.

3. When the video type is audio-based, the weight of dynamic audio features is 0.6, and the weight of dynamic visual features is 0.

4. The multimodal dynamic feature matrix has a dimension of 768. The generation process of the multimodal dynamic feature matrix also includes privacy encryption processing, specifically: local differential privacy technology is used to add perturbation noise to sensitive information contained in the dynamic visual features, and homomorphic encryption algorithm is used to encrypt and store the multimodal dynamic feature matrix.

6. The video cloud intelligent AI retrieval method according to claim 1, characterized in that, Search requests can include text requests, image requests, or audio requests; Mid-level low-dimensional feature matching reduces the candidate video set to less than 1% of the original candidate set size; High-dimensional feature similarity calculation uses the cosine similarity algorithm; The search results are the top 20 video clips sorted by time, and the search response time is ≤2 seconds; High-dimensional feature similarity calculation is performed within the encrypted domain, with a risk of sensitive information leakage of less than 0.1%.

7. The video cloud intelligent AI retrieval method according to claim 1, characterized in that, Feedback information includes user-marked "relevant results" and "irrelevant results," as well as information on manually pinning results. The reinforcement learning algorithm uses user satisfaction as the reward function, which is the proportion of relevant results to the total search results. It completes one optimization every 1000 search requests. If the relevance accuracy of any feature exceeds 90%, the weight of the corresponding feature is increased by 5%-10%. If the number of search results is less than 5, the similarity calculation threshold is reduced by 5%-8%.

8. The video cloud intelligent AI retrieval method according to any one of claims 1 to 7, characterized in that, The edge node layer is deployed on video acquisition devices, including security cameras and short video shooting terminals; the hardware chips used in the video acquisition devices are selected from NVIDIA Jetson Nano chips and embedded ARM chips. The video cloud layer uses a distributed computing cluster, which supports FP16 half-precision quantization to optimize computing efficiency.

9. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the video cloud intelligent AI retrieval method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the video cloud intelligent AI retrieval method as described in any one of claims 1 to 8.

Images