Adaptive evolutionary video data processing method, device and system
By adaptively adjusting the parameters of the video data processing model and combining 3D convolution and dynamic attention mechanisms, the inefficiency problem caused by static models is solved, achieving efficient data processing and resource optimization in long-tail scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM ARTIFICIAL INTELLIGENCE TECHNOLOGY (BEIJING) CO LTD
- Filing Date
- 2025-08-28
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies suffer from low data processing efficiency due to the use of static video data processing models, especially when there are differences between the data and the training dataset, making it impossible to meet real-time requirements.
By determining the data distribution feature vector of the target video data, the target model parameter set is retrieved in the action space according to the preset model architecture, and model parameter description information is generated. The video data processing model in the terminal device is adjusted, and cross-frame motion feature capture is achieved by combining three-dimensional convolutional blocks and dynamic attention mechanisms. Federated meta-learning and meta-learning methods are used for model training and resource allocation optimization.
It achieves matching between video data processing models and data, improves data processing efficiency, solves the inefficiency problem caused by static models, and enhances algorithm performance and resource utilization in long-tail scenarios.
Smart Images
Figure CN120726543B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of electronic digital data processing, and more specifically, to an adaptive evolutionary video data processing method, apparatus, and system. Background Technology
[0002] In related technologies, when processing video data, a static video data processing model is typically defined and then used to process the video data. The problem with this approach is that when there are differences between the data to be processed and the training dataset, it leads to low data processing efficiency.
[0003] There is currently no effective solution to the above problems. Summary of the Invention
[0004] This application provides an adaptive evolutionary video data processing method, apparatus, and system to at least solve the technical problem of low model data processing efficiency caused by the use of static data processing models in related technologies.
[0005] According to one aspect of the embodiments of this application, an adaptive evolutionary video data processing method is provided, comprising: determining a data distribution feature vector of target video data; retrieving a target model parameter set in an action space based on the data distribution feature vector and a preset model architecture, wherein the action space includes multiple optional model parameter sets; generating model parameter description information based on the preset model architecture and the target model parameter set; and sending the model parameter description information to a terminal device, wherein the model parameter description information is used to adjust the video data processing model deployed in the terminal device to obtain a target video data processing model.
[0006] Optionally, the data distribution feature vector is used to represent the first data feature of the target video data. The first data feature includes at least one of the following: category distribution entropy, data processing difficulty evaluation result, and spatial resolution. The data processing difficulty evaluation result is determined by: determining the second data feature of the target video data, wherein the second data feature includes at least one of the following: image entropy and gradient magnitude; processing the second data feature through a difficulty classifier to obtain the data processing difficulty evaluation result of the target video data, wherein the data processing difficulty evaluation result includes difficult-to-process data and easy-to-process data.
[0007] Optionally, the method further includes: determining a metadata processing model and training the metadata processing model using data feature vectors from a preset knowledge base; after training the metadata processing model, sending a first set of model parameters of the metadata processing model to each terminal device, wherein the terminal device adjusts the parameters of the target video data processing model deployed locally according to the first set of model parameters, and then trains the adjusted target video data processing model using local data to obtain a second set of model parameters, the second set of model parameters including the model parameters of the trained target video data processing model; receiving the second set of model parameters sent by each terminal device, and data processing difficulty information of the local data of the terminal device; determining the weights corresponding to each set of second model parameters according to the data processing difficulty information, and performing weighted aggregation processing on each set of second model parameters according to the weights to obtain a third set of model parameters; sending the third set of model parameters to each terminal device, wherein the third set of model parameters is used to update the target video data processing model deployed locally on the terminal device.
[0008] Optionally, a preset knowledge base is used to determine the similarity between data feature vectors stored in the preset knowledge base, and to merge data feature vectors with a similarity greater than a preset similarity threshold; the method further includes: receiving video data samples collected by a terminal device, and determining the data feature vectors of the video data samples; storing the data feature vectors of the video data samples in the preset knowledge base.
[0009] Optionally, the method further includes: receiving video data samples collected by the terminal device and storing the video data samples in a dynamic negative sample pool; after the number of video data samples stored in the dynamic negative sample pool meets the preset requirements, adjusting the preset model architecture according to the video data samples in the dynamic negative sample pool.
[0010] Optionally, the method further includes: determining the data processing difficulty evaluation result of each frame of the target video data; determining the computing power resource allocation information corresponding to the image based on the data processing difficulty evaluation result, wherein the computing power resource allocation information includes the computing power resources allocated to the terminal device when calling the target video data processing model to process the image; and calling the target video processing model to process the target video data according to the computing power resource allocation information.
[0011] Optionally, the target video data processing model includes a three-dimensional convolutional block, which is used to capture cross-frame motion features of the target video data from both spatial and temporal dimensions.
[0012] According to another aspect of the embodiments of this application, an adaptive evolutionary video data processing apparatus is also provided, comprising: a first processing module, configured to determine a data distribution feature vector of target video data; a second processing module, configured to retrieve a target model parameter set in an action space based on the data distribution feature vector and a preset model architecture, wherein the action space includes multiple optional model parameter sets; a third processing module, configured to generate model parameter description information based on the preset model architecture and the target model parameter set; and a fourth processing module, configured to send the model parameter description information to a terminal device, wherein the model parameter description information is used to adjust the video data processing model deployed in the terminal device to obtain a target video data processing model.
[0013] According to another aspect of the embodiments of this application, an adaptive evolutionary video data processing system is also provided, including a server and multiple terminal devices. The server is configured to: determine the data distribution feature vector of the target video data; retrieve a target model parameter set in the action space based on the data distribution feature vector and a preset model architecture, wherein the action space includes multiple optional model parameter sets; generate model parameter description information based on the preset model architecture and the target model parameter set; and send the model parameter description information to the terminal devices. The terminal devices are configured to adjust the video data processing model deployed in the terminal devices based on the model parameter description information to obtain a target video data processing model, which is used to process the target video data.
[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, wherein the program controls the device where the non-volatile storage medium is located to perform an adaptive evolutionary video data processing method when it runs.
[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes an adaptive evolutionary video data processing method during runtime.
[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that implements an adaptively evolving video data processing method when executed by a processor.
[0017] In this embodiment, a method is adopted to determine the data distribution feature vector of the target video data; based on the data distribution feature vector and the preset model architecture, a target model parameter set is retrieved in the action space, wherein the action space includes multiple optional model parameter sets; model parameter description information is generated according to the preset model architecture and the target model parameter set; the model parameter description information is sent to the terminal device, wherein the model parameter description information is used to adjust the video data processing model deployed in the terminal device to obtain the target video data processing model. By adjusting the model parameters according to the characteristics of the data to be processed, the goal of matching the target video data processing model with the data to be processed is achieved, thereby realizing the technical effect of improving data processing efficiency, and thus solving the technical problem of low model data processing efficiency caused by the use of static data processing models in related technologies. Attached Figure Description
[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0019] Figure 1 This is a schematic diagram of the structure of a computer terminal (or mobile device) according to an embodiment of this application;
[0020] Figure 2 This is a flowchart illustrating an adaptive evolutionary video data processing method according to an embodiment of this application;
[0021] Figure 3 This is a schematic diagram of the structure of an adaptive evolution video data processing system according to an embodiment of this application;
[0022] Figure 4 This is a schematic diagram of the structure of another adaptive evolution video data processing system provided according to an embodiment of this application;
[0023] Figure 5 This is a schematic diagram of the structure of an adaptive evolution video data processing device provided according to an embodiment of this application. Detailed Implementation
[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0026] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:
[0027] Dynamic Adaptive Algorithm Architecture: Design a reinforcement learning-driven neural architecture search (RL-NAS) to achieve dynamic optimization of the algorithm model structure, and automatically adjust network layer connections, feature fusion methods and computational resource allocation for dynamic changes in data distribution in long-tail scenarios.
[0028] Cross-domain knowledge transfer mechanism: Design a few-shot knowledge transfer framework based on meta-learning, which improves the rapid generalization ability of rare class samples by extracting common feature representations of cross-domain visual tasks.
[0029] Three-dimensional data circulation system: Construct a closed-loop feedback system of "real-time data perception - semantic feature distillation - architecture parameter evolution", and combine spatiotemporal dimension data augmentation technology to achieve synergistic optimization of algorithm performance and system resource utilization.
[0030] Currently, in long-tail scenarios in fields such as intelligent medical image diagnosis, autonomous driving long-tail scene recognition, and industrial complex defect detection, it is often necessary to process dynamically changing small sample, cross-modal, and multi-domain distributed data. However, related technologies have the following problems in long-tail scenarios:
[0031] Cross-domain migration is inefficient: In cross-device (edge-cloud) and cross-modal (RGB-infrared) scenarios, the models in the relevant technologies need to be retrained, which takes up to several weeks and cannot meet the real-time requirements.
[0032] Resource allocation imbalance: Related technologies allocate fixed computing resources to complex samples (such as fuzzy targets and minor defects), resulting in overcomputation of simple samples and insufficient feature extraction of difficult samples.
[0033] The architecture design lacks dynamism: related technologies rely on human experience to pre-set model structures, and cannot automatically adjust key parameters such as convolution kernel size and attention module weights according to real-time data characteristics.
[0034] Insufficient learning capacity with few samples: When the number of new category samples is small, the accuracy of transfer learning methods in related technologies drops significantly, requiring a large amount of manually labeled data.
[0035] Imbalance between resource utilization and algorithm performance: When processing multiple tasks, edge devices in related technologies often suffer from high latency (e.g., >500ms) or low energy efficiency (e.g., power consumption >10W) due to the fixed allocation of computing resources.
[0036] To address this issue, relevant solutions are provided in the embodiments of this application, which are described in detail below.
[0037] According to an embodiment of this application, an embodiment of an adaptive evolution video data processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0038] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device 10) for implementing an adaptive evolutionary video data processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0039] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element within the computer terminal 10 (or mobile device 10). As involved in the embodiments of this application, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).
[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the adaptive evolution video data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned adaptive evolution video data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0041] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0042] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device 10).
[0043] Under the aforementioned operating environment, embodiments of this application provide an adaptive evolutionary video data processing method, such as... Figure 2 As shown, the method includes the following steps:
[0044] Step S202: Determine the data distribution feature vector of the target video data;
[0045] In the technical solution provided in step S202, the data distribution feature vector is used to represent the first data feature of the target video data. The first data feature includes at least one of the following: category distribution entropy, data processing difficulty evaluation result, and spatial resolution. The data processing difficulty evaluation result is determined by: determining the second data feature of the target video data, wherein the second data feature includes at least one of the following: image entropy and gradient magnitude; processing the second data feature through a difficulty classifier to obtain the data processing difficulty evaluation result of the target video data, wherein the data processing difficulty evaluation result includes difficult-to-process data and easy-to-process data.
[0046] Step S204: Based on the data distribution feature vector and the preset model architecture, retrieve the target model parameter set in the action space, wherein the action space includes multiple optional model parameter sets;
[0047] In some embodiments of this application, the aforementioned preset model architecture can be stored in a dynamic model library. The dynamic model library can be divided into a base layer, a task layer, and an evolutionary layer, wherein the base layer includes a general visual feature extractor, the task layer includes a domain-specific detector, and the evolutionary layer includes a dynamic architecture generated by an RL engine.
[0048] Step S206: Generate model parameter description information based on the preset model architecture and the target model parameter set;
[0049] In some embodiments of this application, a reinforcement learning neural architecture search engine (RL-NASEngine) can be used to determine the target model parameter set and further obtain model parameter description information. The model parameter description information can be in the form of a model architecture description file, which may contain the model's network layer connection diagram, parameter values, etc. It may also include a preset computing resource allocation table. This computing resource allocation table includes recommended computing resource allocation methods for processing each image in the video. This computing resource allocation method can be a suggested type and specific value of computing resources, or a relative computing resource allocation scheme between different images. For example, when processing images with a difficulty evaluation result of "difficult," the required computing resources should be a certain multiple of those required for images with an evaluation result of "easy."
[0050] In some embodiments of this application, the model's architectural parameters can be encoded into an action space, the dimension of which can be set independently, such as 256. The model's architectural parameters may include convolutional kernel size, number of attention heads, feature pyramid levels, etc.
[0051] The reward function can be designed as: Multi-objective reward R = 0.6 × mAP + 0.3 × QPS + 0.1 × (1 - power consumption / W), thereby achieving Pareto optimization of accuracy, speed, and energy efficiency. Here, mAP is the average accuracy value, and QPS is the number of queries per second. The weight coefficients in the above reward function can also be set to other values, as long as the sum of all weight coefficients is 1 and all are greater than zero.
[0052] In some embodiments of this application, the policy network may employ a Transformer architecture. The input to the policy network can be the current architecture state and data distribution feature vector of a preset model architecture, and the output can be the parameter probability distribution of the model architecture. The parameter probability distribution includes the probability that each parameter's specific value is the optimal parameter value in the space. Then, based on the parameter probability distribution, the parameter with the highest probability of being the optimal parameter value among each type of parameter can be determined as the parameter value in the target model parameter set.
[0053] Step S208: Send model parameter description information to the terminal device. The model parameter description information is used to adjust the video data processing model deployed in the terminal device to obtain the target video data processing model.
[0054] In the technical solution provided in step S208, the target video data processing model includes a three-dimensional convolutional block, wherein the three-dimensional convolutional block is used to capture cross-frame motion features of the target video data from the spatial and temporal dimensions.
[0055] In some embodiments of this application, the target video data processing model can use dilated 3D convolution (dilation=2 or other user-defined values) to capture cross-frame motion features. The size of the convolution kernel can be set user-defined, for example, 5×5×3. The first two parameters of the convolution kernel size represent the spatial dimension, and the third parameter represents the temporal dimension.
[0056] Furthermore, the target video data processing model incorporates a dynamic attention mechanism, which adaptively adjusts the allocation of attention heads in the spatiotemporal dimensions based on the target model's parameter set. For example, it can allocate 8 attention heads for complex scenes and 2 for simple scenes. The final output of the target video data processing model is the spatiotemporal feature map of the target video data. Assuming the target video data is represented as T×H×W×C in each dimension, with a convolution kernel size of 5×5×3 and dilation=2, the spatiotemporal feature map can be represented as H / 4×W / 4×256×T in each dimension. Here, T represents the time dimension (the number of image frames in the video), H represents the height, W represents the width, and C represents the number of channels.
[0057] In some embodiments of this application, the target video data processing model can be trained by combining federated learning model training methods and meta-learning model training methods, including: determining a metadata processing model and training the metadata processing model using data feature vectors from a preset knowledge base; after training the metadata processing model, sending a first set of model parameters of the metadata processing model to each terminal device, wherein the terminal device adjusts the parameters of the target video data processing model deployed locally according to the first set of model parameters, and then trains the adjusted target video data processing model using local data to obtain a second set of model parameters, the second set of model parameters including the model parameters of the trained target video data processing model; receiving the second set of model parameters sent by each terminal device, and data processing difficulty information of the local data of the terminal device; determining the weights corresponding to each set of second model parameters according to the data processing difficulty information, and performing weighted aggregation processing on each set of second model parameters according to the weights to obtain a third set of model parameters; sending the third set of model parameters to each terminal device, wherein the third set of model parameters is used to update the target video data processing model deployed locally on the terminal device.
[0058] In some embodiments of this application, the aforementioned metadata processing model can also be referred to as a cross-domain feature extractor. Compared to federated learning model training methods in related technologies, combining the meta-learning model training method can effectively improve the convergence speed, making the convergence speed three times faster than before.
[0059] Furthermore, when aggregating model parameters provided by various terminal devices, the weights of the model parameters provided by each terminal device can be determined based on the difficulty of processing the data used by these devices during model training. Difficult-to-process data can have higher weights (e.g., 1.5), while easily-processed data will have relatively lower weights (e.g., 0.5).
[0060] In some embodiments of this application, since the training data used by the terminal device when training the local target video data processing model may contain both difficult-to-process and easy-to-process data, the final weights can be determined based on the ratio of difficult-to-process and easy-to-process data in the training data. For example, if the training data contains 60% difficult-to-process data and 40% easy-to-process data, and the preset weight of difficult-to-process data is 1.5 and the weight of easy-to-process data is 0.5, the weight corresponding to the second model parameter set uploaded by the terminal device can be 0.6×1.5+0.4×0.5=1.1.
[0061] In some embodiments of this application, a preset knowledge base is used to determine the similarity between data feature vectors stored in the preset knowledge base, and to merge data feature vectors with similarity greater than a preset similarity threshold; the method further includes: receiving video data samples collected by a terminal device, and determining the data feature vectors of the video data samples; storing the data feature vectors of the video data samples in the preset knowledge base.
[0062] In some embodiments of this application, the preset knowledge base (also known as a cross-domain knowledge base) also has a spatiotemporal feature indexing function based on a vector database (FAISS) (the dimension can be set to 768 or other values), and supports a hybrid retrieval function of "semantic + spatiotemporal similarity". Furthermore, the preset knowledge base can periodically merge similar feature vectors in the database. The similarity between feature vectors can be represented by cosine similarity, and a pair of feature vectors with a cosine similarity greater than 0.95 can be considered similar feature vectors.
[0063] In some embodiments of this application, video data samples collected by a terminal device can also be received and stored in a dynamic negative sample pool; after the number of video data samples stored in the dynamic negative sample pool meets the preset requirements, the preset model architecture is adjusted according to the video data samples in the dynamic negative sample pool.
[0064] Optionally, this application embodiment also provides a three-dimensional data circulation system, including a real-time perception layer, a semantic distillation layer, and an architecture evolution layer. The real-time perception layer can be used to collect missed samples, new category samples, etc., through online data probes set in terminal devices to construct a dynamic negative sample pool. New category samples refer to samples whose features are inconsistent with previously processed data, or whose corresponding task type is a new task type. The semantic distillation layer can utilize a Teacher-Student architecture to distill complex scene features into lightweight feature vectors and store the lightweight feature vectors in a cross-domain knowledge base. The dimension of the lightweight feature vectors can be set independently, for example, dimension = 128. The architecture evolution layer can input the data from the negative sample pool into the RL-NAS engine for iterative optimization of the preset model architecture. The update frequency of the negative sample pool can be set independently, for example, once every 10 minutes.
[0065] As an optional implementation, the optimization process can be triggered when the number of new samples reaches a preset number (e.g., 200). Here, new samples refer to samples added to the negative sample pool after the last model architecture update.
[0066] As an optional implementation, the data processing difficulty evaluation results of each frame in the target video data can also be determined; the computing power resource allocation information corresponding to the image can be determined based on the data processing difficulty evaluation results, wherein the computing power resource allocation information includes the computing power resources allocated to the terminal device when calling the target video data processing model to process the image; and the target video processing model is called to process the target video data according to the computing power resource allocation information.
[0067] In some embodiments of this application, a dynamic resource allocation module can be used to detect the computing power resource data of the terminal device in real time (such as CPU / GPU utilization, memory usage, etc.), and a difficulty classifier can be used to determine the data processing difficulty evaluation results of each frame of the video data to be processed. Based on the data processing difficulty evaluation results and real-time computing power resources, the available computing power resources for processing different images can be determined, thereby obtaining computing power resource allocation information.
[0068] The input to the difficulty sample classifier can be image entropy, gradient magnitude, etc., of video data, and the output data can be the data processing difficulty evaluation results of each frame of image, divided into easy-to-process data (also known as simple samples) and difficult-to-process data (also known as hard samples or complex samples). For difficult-to-process data, more computing resources can be allocated, for example, 80% of the computing power can be allocated to difficult image data or all cores of the GPU can be set to participate in the processing work, while 20% of the computing power can be allocated to easy-to-process image data or only one core of the GPU can be set to participate in the data processing work.
[0069] In some embodiments of this application, the aforementioned computing resource allocation information can be carried in real-time computing resource allocation instructions. The dynamic resource allocation module can be set in the terminal device, or it can be a separate module that schedules and allocates the computing resources of multiple terminal devices.
[0070] In some embodiments of this application, assuming that the adaptive evolutionary video data processing method provided in the embodiments of this application is used to process video data in sudden highway scenarios (such as cone recognition in construction areas, sample size = 15 images / class), and the computing power of the terminal device is limited, the specific execution process includes the following steps:
[0071] The first step, dynamic architecture generation: Based on the input characteristics of "low resolution (640×480) + small target (area < 32 pixels)," the RL-NAS engine generates a lightweight architecture: MobileNetV3 + attention gating module. Furthermore, this lightweight architecture reduces computational cost by 40% compared to related technologies like YOLOv5.
[0072] The second step is spatiotemporal feature distillation: 3D convolution is applied to 5 consecutive frames of images to capture the trajectory of the cone movement, and the feature map resolution is improved to 160×120 (compared to 2D convolution, the resolution of the obtained feature map can be improved by 2.25 times).
[0073] The third step, federated meta-learning transfer: using 5 labeled images of urban road cones and 10 unlabeled images of highways, the transfer is completed within a preset number of iterations using a meta-learning algorithm, which can improve the model's mAP@0.5 from 58% to 89%. mAP@0.5 represents the average accuracy value with a confidence threshold of 5.
[0074] The fourth step is resource allocation: allocating more computing resources to small cone samples at a distance and less computing resources to large samples at a distance, effectively reducing the average inference latency.
[0075] It should be noted that there is no strict execution order among steps two, three, and four. That is, steps two, three, and four can be executed in parallel, or step two can be executed first, followed by step three or four, and then step two can be re-executed based on the results of steps three and four. Furthermore, the specific values involved in the above steps are for illustrative purposes only and do not constitute a limitation on the solution provided in this application.
[0076] By employing a method that determines the data distribution feature vector of the target video data; retrieving the target model parameter set in the action space based on the data distribution feature vector and a preset model architecture, wherein the action space includes multiple optional model parameter sets; generating model parameter description information based on the preset model architecture and the target model parameter set; and sending the model parameter description information to the terminal device, wherein the model parameter description information is used to adjust the video data processing model deployed in the terminal device to obtain the target video data processing model, by adjusting the model parameters according to the characteristics of the data to be processed, the goal of matching the target video data processing model with the data to be processed is achieved, thereby realizing the technical effect of improving data processing efficiency, and thus solving the technical problem of low model data processing efficiency caused by the use of static data processing models in related technologies.
[0077] Furthermore, the method provided in this application is the first to apply an RL-NAS architecture generation engine to long-tail visual scenes (characterized by data sparsity, dynamic distribution, and limited computing power), achieving online adaptive optimization of the algorithm architecture and overcoming the generalization limitations of traditional static architectures. It also proposes a dynamic feature extraction method based on 3D convolution, combined with federated meta-learning to achieve few-shot cross-domain transfer, solving the data scarcity problem in long-tail scenes. A closed-loop feedback system of "data-feature-architecture" is constructed to achieve dynamic allocation of computing resources, balancing model performance and terminal device energy efficiency. Moreover, the video data processing model trained using the method provided in this application significantly improves upon related technologies in terms of few-shot accuracy and transfer efficiency.
[0078] This application also provides an adaptive evolutionary video data processing system. Figure 3 This is a schematic diagram of the system structure. From Figure 3 As can be seen from the diagram, the system includes a dynamic requirement analysis module 30, a basic model library 32, an RL-NAS architecture generation engine 34, a spatiotemporal feature distillation module 36, a cross-domain knowledge base 38, a federated meta-learning transfer module 310, a dynamic resource allocation module 312, a three-dimensional data circulation module 314, and a multimodal evaluation module 316.
[0079] In some embodiments of this application, the dynamic demand analysis module 30 is used to determine the video data to be processed and the data distribution feature vector of the video data. The preset model architecture can be stored in the basic model library 32. The RL-NAS architecture generation engine 34 can generate a model architecture description file for the target video processing model based on the data distribution feature vector and the preset model architecture in the basic model library 32. Then, the spatiotemporal feature distillation module adjusts the parameters of the video data processing model deployed in the terminal device according to the model architecture description file, and then uses the adjusted video data processing model to process the video data to be processed.
[0080] As an optional implementation, before the spatiotemporal feature distillation module 36 processes the video data to be processed, the video data processing model can be further trained and optimized using federated learning algorithms and meta-learning algorithms through the federated meta-learning transfer module 310. The training data for the federated meta-learning transfer module 310 can be obtained from the cross-domain knowledge base 38.
[0081] In some embodiments of this application, the dynamic resource allocation module 312 can also determine how the spatiotemporal feature distillation module allocates computing resources when processing video data, thereby avoiding waste of computing resources and improving processing efficiency.
[0082] In some embodiments of this application, a negative sample pool can be constructed by periodically acquiring new sample data through the three-dimensional data circulation module 314, and the negative sample pool can be used to update the cross-domain knowledge base 38 and the basic model library 32. After the cross-domain knowledge base 38 is updated, the video data processing model can be trained and optimized again using federated learning algorithms and meta-learning algorithms through the federated meta-learning transfer module 310. After the basic model library 32 is updated, the preset model architecture can be optimized by the RL-NAS architecture generation engine 34.
[0083] In addition, in this embodiment of the application, the video data processing process and results of the spatiotemporal feature distillation module can be evaluated by the multimodal evaluation module 316 to determine whether the workflow of the system needs to be optimized.
[0084] This application provides another adaptive evolutionary video data processing system. Figure 4 This is a schematic diagram of the system structure. From Figure 4 As can be seen, the system includes a server 40 and multiple terminal devices 42. The server 40 is used to determine the data distribution feature vector of the target video data; based on the data distribution feature vector and the preset model architecture, it retrieves the target model parameter set in the action space, where the action space includes multiple optional model parameter sets; it generates model parameter description information according to the preset model architecture and the target model parameter set; and it sends the model parameter description information to the terminal devices 42. The terminal devices 42 are used to adjust the video data processing model deployed in the terminal devices 42 according to the model parameter description information to obtain the target video data processing model, which is used to process the target video data.
[0085] In some embodiments of this application, the terminal device described above may be an edge computing device.
[0086] This application provides an adaptive evolutionary video data processing device. Figure 5 This is a schematic diagram of the device. Figure 5 As can be seen from the diagram, the device includes: a first processing module 50, used to determine the data distribution feature vector of the target video data; a second processing module 52, used to retrieve the target model parameter set in the action space based on the data distribution feature vector and a preset model architecture, wherein the action space includes multiple optional model parameter sets; a third processing module 54, used to generate model parameter description information based on the preset model architecture and the target model parameter set; and a fourth processing module 56, used to send the model parameter description information to the terminal device, wherein the model parameter description information is used to adjust the video data processing model deployed in the terminal device to obtain the target video data processing model.
[0087] In some embodiments of this application, a data distribution feature vector is used to represent a first data feature of the target video data. The first data feature includes at least one of the following: category distribution entropy, data processing difficulty evaluation result, and spatial resolution. The data processing difficulty evaluation result is determined by the first processing module 50 in the following manner: determining a second data feature of the target video data, wherein the second data feature includes at least one of the following: image entropy, gradient magnitude; processing the second data feature through a difficulty classifier to obtain a data processing difficulty evaluation result of the target video data, wherein the data processing difficulty evaluation result includes difficult-to-process data and easy-to-process data.
[0088] In some embodiments of this application, the target video data processing model includes a three-dimensional convolutional block, wherein the three-dimensional convolutional block is used to capture cross-frame motion features of the target video data from spatial and temporal dimensions.
[0089] In some embodiments of this application, the fourth processing module 56 is further configured to: determine a metadata processing model and train the metadata processing model using data feature vectors in a preset knowledge base; after training the metadata processing model, send a first set of model parameters of the metadata processing model to each terminal device, wherein the terminal device adjusts the parameters of the target video data processing model deployed locally according to the first set of model parameters, and trains the adjusted target video data processing model using local data after adjustment to obtain a second set of model parameters, the second set of model parameters including the model parameters of the trained target video data processing model; receive the second set of model parameters sent by each terminal device, and data processing difficulty information of the local data of the terminal device; determine the weights corresponding to each set of second model parameters according to the data processing difficulty information, and perform weighted aggregation processing on each set of second model parameters according to the weights to obtain a third set of model parameters; send the third set of model parameters to each terminal device, wherein the third set of model parameters is used to update the target video data processing model deployed locally on the terminal device.
[0090] In some embodiments of this application, a preset knowledge base is used to determine the similarity between data feature vectors stored in the preset knowledge base, and to merge data feature vectors with similarity greater than a preset similarity threshold; the fourth processing module 56 is also used to: receive video data samples collected by the terminal device, and determine the data feature vectors of the video data samples; and store the data feature vectors of the video data samples into the preset knowledge base.
[0091] In some embodiments of this application, the fourth processing module 56 is further configured to: receive video data samples collected by the terminal device and store the video data samples in a dynamic negative sample pool; after the number of video data samples stored in the dynamic negative sample pool meets the preset requirements, adjust the preset model architecture according to the video data samples in the dynamic negative sample pool.
[0092] In some embodiments of this application, the terminal device is further configured to: determine the data processing difficulty evaluation result of each frame of the target video data; determine the computing power resource allocation information corresponding to the image based on the data processing difficulty evaluation result, wherein the computing power resource allocation information includes the computing power resources allocated to the terminal device when calling the target video data processing model to process the image; and call the target video processing model to process the target video data according to the computing power resource allocation information.
[0093] It should be noted that each module in the aforementioned adaptive evolution video data processing device can be a program module (e.g., a set of program instructions to implement a specific function) or a hardware module. For the latter, it can take the following forms, but is not limited to them: each of the above modules is represented by a processor, or the functions of each of the above modules are implemented by a processor.
[0094] According to an embodiment of this application, a non-volatile storage medium is also provided. The non-volatile storage medium stores a program that, when running, controls the device where the non-volatile storage medium is located to execute the following adaptive evolutionary video data processing method: determining the data distribution feature vector of the target video data; retrieving the target model parameter set in the action space based on the data distribution feature vector and a preset model architecture, wherein the action space includes multiple optional model parameter sets; generating model parameter description information based on the preset model architecture and the target model parameter set; and sending the model parameter description information to the terminal device, wherein the model parameter description information is used to adjust the video data processing model deployed in the terminal device to obtain the target video data processing model.
[0095] According to an embodiment of this application, an electronic device is also provided, including a memory and a processor. The processor is used to run a program stored in the memory, wherein the program executes the following adaptive evolutionary video data processing method: determining the data distribution feature vector of the target video data; retrieving the target model parameter set in the action space based on the data distribution feature vector and a preset model architecture, wherein the action space includes multiple optional model parameter sets; generating model parameter description information based on the preset model architecture and the target model parameter set; and sending the model parameter description information to a terminal device, wherein the model parameter description information is used to adjust the video data processing model deployed in the terminal device to obtain the target video data processing model.
[0096] According to an embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the following adaptive evolutionary video data processing method: determining the data distribution feature vector of the target video data; retrieving a target model parameter set in the action space based on the data distribution feature vector and a preset model architecture, wherein the action space includes multiple optional model parameter sets; generating model parameter description information based on the preset model architecture and the target model parameter set; and sending the model parameter description information to a terminal device, wherein the model parameter description information is used to adjust the video data processing model deployed in the terminal device to obtain the target video data processing model.
[0097] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0098] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0100] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server 40, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0102] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. An adaptive evolutionary video data processing method, characterized in that, include: A data distribution feature vector of the target video data is determined, wherein the data distribution feature vector is used to represent a first data feature of the target video data, the first data feature including at least one of the following: category distribution entropy, data processing difficulty evaluation result, and spatial resolution, wherein the data processing difficulty evaluation result is determined by the following method: Determine a second data feature of the target video data, wherein the second data feature includes at least one of the following: image entropy, gradient magnitude; The second data feature is processed by a difficulty classifier to obtain the data processing difficulty evaluation result of the target video data, wherein the data processing difficulty evaluation result includes difficult data and easy data. Based on the data distribution feature vector and the preset model architecture, the target model parameter set is retrieved in the action space, wherein the action space includes multiple optional model parameter sets; Generate model parameter description information based on the preset model architecture and the target model parameter set; The model parameter description information is sent to the terminal device, wherein the model parameter description information is used to adjust the video data processing model deployed in the terminal device to obtain the target video data processing model.
2. The adaptive evolution video data processing method according to claim 1, characterized in that, The method further includes: A metadata processing model is determined, and the metadata processing model is trained using data feature vectors from a preset knowledge base; After training the metadata processing model, the first model parameter set of the metadata processing model is sent to each of the terminal devices. The terminal devices adjust the parameters of the target video data processing model deployed locally according to the first model parameter set, and then train the adjusted target video data processing model using local data to obtain a second model parameter set. The second model parameter set includes the model parameters of the trained target video data processing model. Receive the second model parameter set sent by each of the terminal devices, as well as the data processing difficulty information of the local data of the terminal devices; Based on the data processing difficulty information, the weights corresponding to each group of the second model parameter sets are determined, and the weights are used to perform weighted aggregation on each group of the second model parameter sets to obtain the third model parameter set. The third model parameter set is sent to each of the terminal devices, wherein the third model parameter set is used to update the target video data processing model deployed locally on the terminal device.
3. The adaptive evolution video data processing method according to claim 2, characterized in that, The preset knowledge base is used to determine the similarity between the data feature vectors stored in the preset knowledge base, and to merge the data feature vectors whose similarity is greater than a preset similarity threshold; the method further includes: Receive video data samples collected by the terminal device and determine the data feature vector of the video data samples; The data feature vectors of the video data samples are stored in the preset knowledge base.
4. The adaptive evolution video data processing method according to claim 1, characterized in that, The method further includes: Receive video data samples collected by the terminal device and store the video data samples in a dynamic negative sample pool; Once the number of video data samples stored in the dynamic negative sample pool meets the preset requirements, the preset model architecture is adjusted based on the video data samples in the dynamic negative sample pool.
5. The adaptive evolution video data processing method according to claim 1, characterized in that, The method further includes: Determine the data processing difficulty evaluation results for each frame of the target video data; Based on the data processing difficulty evaluation results, the computing power resource allocation information corresponding to the image is determined, wherein the computing power resource allocation information includes the computing power resources allocated to the terminal device when it calls the target video data processing model to process the image; The target video data is processed by calling the target video processing model according to the computing power resource allocation information.
6. The video data processing method according to claim 1, characterized in that, The target video data processing model includes a three-dimensional convolutional block, which is used to capture cross-frame motion features of the target video data from both spatial and temporal dimensions.
7. An adaptive evolutionary video data processing device, characterized in that, include: A first processing module is configured to determine a data distribution feature vector of the target video data, wherein the data distribution feature vector represents a first data feature of the target video data, and the first data feature includes at least one of the following: category distribution entropy, data processing difficulty evaluation result, and spatial resolution, wherein the data processing difficulty evaluation result is determined by the following method: Determine a second data feature of the target video data, wherein the second data feature includes at least one of the following: image entropy, gradient magnitude; The second data feature is processed by a difficulty classifier to obtain the data processing difficulty evaluation result of the target video data, wherein the data processing difficulty evaluation result includes difficult data and easy data. The second processing module is used to retrieve the target model parameter set in the action space based on the data distribution feature vector and the preset model architecture, wherein the action space includes multiple optional model parameter sets. The third processing module is used to generate model parameter description information based on the preset model architecture and the target model parameter set; The fourth processing module is used to send the model parameter description information to the terminal device, wherein the model parameter description information is used to adjust the video data processing model deployed in the terminal device to obtain the target video data processing model.
8. An adaptive evolutionary video data processing system, characterized in that, Includes servers and multiple terminal devices, among which, The server is configured to determine a data distribution feature vector of the target video data, wherein the data distribution feature vector represents a first data feature of the target video data, the first data feature including at least one of the following: category distribution entropy, data processing difficulty evaluation result, and spatial resolution. The data processing difficulty evaluation result is determined by: determining a second data feature of the target video data, wherein the second data feature includes at least one of the following: image entropy and gradient magnitude; processing the second data feature through a difficulty classifier to obtain the data processing difficulty evaluation result of the target video data, wherein the data processing difficulty evaluation result includes difficult data and easy data; retrieving a target model parameter set in the action space based on the data distribution feature vector and a preset model architecture, wherein the action space includes multiple selectable model parameter sets; generating model parameter description information based on the preset model architecture and the target model parameter set; and sending the model parameter description information to the terminal device. The terminal device is used to adjust the video data processing model deployed in the terminal device according to the model parameter description information to obtain a target video data processing model, which is used to process the target video data.
9. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device containing the non-volatile storage medium to perform the adaptive evolutionary video data processing method according to any one of claims 1 to 6.
10. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the adaptive evolutionary video data processing method according to any one of claims 1 to 6.
11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the adaptive evolutionary video data processing method according to any one of claims 1 to 6.