Image recognition method and apparatus based on inter-frame features

Abstract

The application discloses an image recognition method and device based on image frame features, and relates to the technical field of image recognition. The method comprises the following steps: constructing a global image recognition model and a global dynamic frame extraction model on a cloud server, and deploying the global image recognition model and the global dynamic frame extraction model to an edge computing gateway; performing migration fine-tuning on the edge computing gateway to obtain a scene image recognition model and a scene dynamic frame extraction model; performing frame processing on a video stream; performing feature extraction and fusion by using the scene image recognition model to obtain fused features; performing dynamic frame extraction by using the scene dynamic frame extraction model according to the fused features to obtain to-be-recognized image data of continuous frames after frame extraction; and performing image recognition on the to-be-recognized image data of continuous frames after frame extraction by using the scene image recognition model to obtain an image recognition result. The method solves the problems of data islands, insufficient model generalization ability, high computing resource requirement and difficult model deployment and maintenance in the prior art.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an image recognition method and apparatus based on inter-frame features of images. Background Technology

[0002] With the advancement of smart city construction, traffic monitoring systems, serving as the "eyes" of urban management, are becoming increasingly important. Traditional traffic monitoring systems mainly rely on manual methods or simple image processing algorithms for event detection, resulting in low efficiency, poor accuracy, and an inability to handle complex scenarios. In recent years, image recognition technology has made breakthroughs in tasks such as target detection and behavior recognition, providing strong technical support for intelligent traffic monitoring.

[0003] Traditional image recognition technologies typically face the following challenges:

[0004] 1) Data silos: Due to privacy and management permissions, camera data collected from different scenarios (especially data containing sensitive information such as accidents and violations) is difficult to aggregate into a central server for unified training. This results in data silos, causing models trained on data from a single region to perform poorly in other regions.

[0005] 2) Insufficient model generalization ability: Different scenarios are highly diverse and dynamic, with significant differences in traffic flow at different intersections, lighting conditions (daytime, nighttime, sunny / cloudy), weather conditions (sunny, rainy, snowy, foggy), and camera angles. A single global model trained on a general dataset in the cloud struggles to maintain high recognition accuracy in all edge scenarios, exhibiting insufficient generalization ability.

[0006] 3) High computational resource requirements: Video streams captured by cameras typically have high frame rates and large data volumes. While edge computing gateways can alleviate the computational burden on the cloud, their computing power remains limited. Performing high-precision image recognition on every frame of the video stream would result in a huge computational load, making it difficult to meet real-time requirements. Traditional fixed-interval frame extraction methods are indiscriminate and may lose transient frames containing crucial information (such as rapidly passing defects), leading to missed detections.

[0007] 4) Difficult model deployment and maintenance: The use cases of models are not static, requiring models to be updated quickly to adapt to new situations. The traditional model of manually collecting data, centralized training, and then batch deployment is time-consuming, costly, and requires regular maintenance. Summary of the Invention

[0008] This invention provides an image recognition method and apparatus based on inter-frame features, which solves the problems of data silos, insufficient model generalization ability, high computational resource requirements, and difficulties in model deployment and maintenance in the prior art.

[0009] In a first aspect, embodiments of the present invention provide an image recognition method based on inter-frame features, the method comprising:

[0010] On the cloud server, a global image recognition model and a global dynamic frame extraction model are built based on a dynamic weighted federated learning mechanism and deployed to all edge computing gateways of the traffic monitoring system.

[0011] At the edge computing gateway, the global image recognition model and the global dynamic frame extraction model are transferred and fine-tuned to obtain the scene's image recognition model and the scene's dynamic frame extraction model.

[0012] The video stream from the camera is captured, and the video stream is processed frame by frame to obtain continuous frames of image data to be identified.

[0013] The image recognition model for the application scenario extracts and fuses features from each frame of the image data to be recognized to obtain fused features;

[0014] Based on the fusion features, the dynamic frame extraction model of the application scenario dynamically extracts frames from the image data to be identified in consecutive frames to obtain the image data to be identified in consecutive frames after frame extraction.

[0015] The image recognition model for the application scenario performs image recognition on the image data to be recognized in consecutive frames after frame extraction, and obtains the image recognition result.

[0016] The technical solution provided in this application has at least the following beneficial effects:

[0017] Through dynamic weighted federated learning, a high-quality global model can be collaboratively built without sharing the original data. Each edge computing gateway only uploads the updated model parameters after local model training (rather than the original, sensitive image data) to the cloud for aggregation, fundamentally solving the "data silo" problem. This allows all participants to collaboratively train a higher-performance global model that integrates knowledge from all scenarios without leaking their core data assets. The dynamic weight mechanism is superior to traditional average aggregation, effectively improving the generalization ability and robustness of the global model. The edge computing gateway uses a small amount of local scene image data to quickly fine-tune the global model, obtaining a scene model specifically adapted to the current scene. This process requires only a small amount of labeled data and can even be combined with unsupervised or self-supervised learning, greatly reducing the accuracy degradation caused by "domain offset" and avoiding the high cost of full data labeling and retraining for each scene. This achieves high-precision, low-cost local deployment of the model. The reinforcement learning-based dynamic frame extraction model works in collaboration with the image recognition model. While ensuring recognition accuracy, it significantly reduces the number of frames requiring high-precision processing, thereby significantly reducing the computational load and power consumption of the edge computing gateway. This enables it to easily handle high-speed video streams and meet the stringent real-time requirements of traffic monitoring scenarios, effectively avoiding the missed detection problem caused by the loss of key frames in traditional fixed frame extraction methods. The cloud-edge-device collaborative architecture allows the model to be continuously iterated and updated in the cloud and seamlessly deployed to the edge. The online learning capability (experience replay pool) at the edge also enables the model to continuously adapt to minor changes in the environment. The entire framework has high flexibility and scalability. When the scene environment changes, the edge computing gateway can automatically adapt to the new situation and continuously optimize the model performance without manual intervention. This self-learning and self-optimization capability forms an efficient model iteration closed loop, greatly shortening the model update cycle, improving the framework's adaptability and maintenance efficiency, and perfectly meeting the needs of modern agile recognition.

[0018] In one alternative implementation, a global image recognition model and a global dynamic frame extraction model are constructed on a cloud server based on a dynamic weighted federated learning mechanism, and deployed to all edge computing gateways of the traffic monitoring system, including:

[0019] On a cloud server, an original image recognition model is built using an image recognition algorithm, and an original dynamic frame extraction model is built using a reinforcement learning algorithm. The original image recognition model and the original dynamic frame extraction model are then distributed to several training servers in different scenarios.

[0020] On the training server, a local image dataset is collected and input into the original image recognition model and the original dynamic frame extraction model for local training, resulting in a local image recognition model and a local dynamic frame extraction model.

[0021] Collect the model parameter update amount, training data amount, and performance evaluation indicators of the local image recognition model and the local dynamic frame extraction model, and upload them to the cloud server.

[0022] On the cloud server, dynamic federated weights are generated based on the amount of training data and performance evaluation metrics of each training server, using a dynamic federated weight mechanism.

[0023] Based on the model parameter update amounts and corresponding dynamic federated weights of all training servers, the initial model parameters of the original image recognition model and the original dynamic frame extraction model are securely aggregated to obtain the global image recognition model and the global dynamic frame extraction model.

[0024] Deploy the global image recognition model and the global dynamic frame extraction model to all edge computing gateways in the traffic monitoring system that communicate with the cloud server.

[0025] In one alternative implementation, the image recognition model includes a single-frame static feature extraction module based on the ResNet-50 algorithm, an inter-frame dynamic feature extraction module based on the PWC-Net-CNN algorithm, a dynamic gating module based on the GRU algorithm, an image recognition module based on the MLP algorithm, and a temporal integration module based on the LSTM algorithm.

[0026] The dynamic frame-skipping model includes a meta-policy learning module based on the MPO algorithm and a dynamic frame-skipping policy generation module based on the DQN algorithm. The dynamic frame-skipping policy generation module includes an agent and an experience replay pool.

[0027] In one optional implementation, at the edge computing gateway, the global image recognition model and the global dynamic frame extraction model are fine-tuned to obtain the scene's image recognition model and the scene's dynamic frame extraction model, including:

[0028] At the edge computing gateway, several scene image data are collected, and prediction labels are added to each scene image data to obtain a scene image dataset;

[0029] Freeze the global model parameters of the global image recognition model and the global dynamic frame extraction model, set the initial LoRA adapter, and perform migration fine-tuning on the initial LoRA adapter based on the scene image dataset.

[0030] During each iteration of migration fine-tuning, an improved optimization algorithm is used to search for the adapter parameters of the initial LoRA adapter to obtain optimized adapter parameters.

[0031] Based on the optimized adapter parameters, the initial LoRA adapters for the global image recognition model and the global dynamic frame extraction model are optimized to obtain the scene's image recognition model and the scene's dynamic frame extraction model.

[0032] In one alternative implementation, during each iteration of the migration fine-tuning process, an improved optimization algorithm is used to search for the adapter parameters of the initial LoRA adapter to obtain optimized adapter parameters, including:

[0033] Based on the LoRA adapter optimization problem, a multi-objective optimization function is set, and the multi-objective optimization function is used as the fitness function of the IPRGO algorithm;

[0034] The adapter parameters of the LoRA adapter are encoded into individual vectors of the IPRGO algorithm, and the IPRGO population parameters and maximum number of iterations are set.

[0035] Based on the IPRGO population parameters, the initial IPRGO population is obtained by initialization using the Tent chaotic mapping sequence; each IPRGO individual in the IPRGO population corresponds to a candidate adapter parameter.

[0036] Based on the scene image dataset, the fitness function is used to obtain the fitness value of each initial IPRGO individual, and the initial IPRGO individual with the best fitness value is taken as the optimal solution.

[0037] Based on the fiber root growth, lateral root growth and taproot growth of the IPRGO algorithm, the initial IPRGO population is iteratively updated to obtain the updated IPRGO population.

[0038] Based on the scene image dataset, the fitness function is used to obtain the fitness value of each updated IPRGO individual, and the updated IPRGO individual with the best fitness value is updated as the optimal solution;

[0039] When the number of iterations reaches the maximum number of iterations or the fitness value of the optimal solution meets the requirements, the iterative update of the IPRGO population is terminated, and the optimal solution of the current iteration is output.

[0040] The individual vectors of the IPRGO individuals corresponding to the optimal solution are decoded to obtain the optimized adapter parameters.

[0041] In one optional implementation, a video stream from a camera is acquired, and the video stream is processed frame by frame to obtain continuous frames of image data to be identified, including:

[0042] At the edge computing gateway, video cameras within the communication range are connected, and video streams from the cameras are captured.

[0043] According to the preset time window, the video stream is frame-by-frame to obtain the raw image data of consecutive frames;

[0044] Image preprocessing is performed on the raw image data of each frame to obtain image data to be recognized for consecutive frames, and the image data to be recognized for consecutive frames is then input into the image recognition model of the scene in sequence.

[0045] In one optional implementation, an image recognition model for the scene is used to extract and fuse features from each frame of the image data to be recognized, resulting in fused features, including:

[0046] The single-frame static feature extraction module of the image recognition model for the application scenario extracts the single-frame static features of each frame of the image data to be recognized.

[0047] The inter-frame dynamic feature extraction module of the image recognition model for the application scenario extracts the optical flow field of the image data to be recognized in the current frame and the previous frame, and performs motion feature encoding based on the optical flow field to obtain the inter-frame dynamic features.

[0048] The dynamic gating module of the image recognition model for the scene generates a gating vector. Based on the gating vector, it fuses the static features of a single frame and the dynamic features between frames, and then inputs the fused features into the dynamic frame extraction model of the scene.

[0049] In one optional implementation, based on the fusion features and a dynamic frame-sampling model for the application scenario, dynamic frame-sampling is performed on the continuous frames of the image data to be identified, resulting in the extracted continuous frames of the image data to be identified, including:

[0050] Based on the fusion characteristics, the meta-policy learning module of the dynamic frame extraction model for the application scenario adjusts the agent of the dynamic frame extraction policy generation module to obtain an updated agent.

[0051] The fused features of the image data to be identified in consecutive frames are written into the updated state space of the agent to obtain the updated state space.

[0052] Based on the updated state space, the updated agent is generated using the dynamic frame-skipping strategy module. It makes dynamic frame-skipping decisions in the action space to obtain the dynamic frame-skipping strategy and collects the corresponding real-time experience.

[0053] According to the dynamic frame extraction strategy, the image data to be identified in consecutive frames is dynamically extracted to obtain the image data to be identified in consecutive frames after frame extraction.

[0054] Several historical experiences are randomly selected from the experience replay pool, and the dynamic frame-sampling model of the scene is continuously trained based on the real-time experience and several historical experiences to obtain an updated dynamic frame-sampling model.

[0055] In one optional implementation, an image recognition model for the scene is used to perform image recognition on the image data to be recognized in consecutive frames after frame extraction, and the image recognition result is obtained, including:

[0056] The single-frame static feature extraction module of the image recognition model for the application scenario extracts the single-frame static features of the image data to be recognized in each frame after frame extraction.

[0057] The inter-frame dynamic feature extraction module of the image recognition model for the application scenario extracts the optical flow field of the image data to be recognized in the current frame and the previous frame after frame extraction, and performs motion feature encoding based on the optical flow field after frame extraction to obtain the inter-frame dynamic features after frame extraction.

[0058] The dynamic gating module of the image recognition model for the application scenario generates a gating vector after frame extraction. Based on the gating vector after frame extraction, the static features of a single frame after frame extraction and the dynamic features between frames after frame extraction are fused to obtain the fused features after frame extraction.

[0059] Based on the fused features after frame extraction, the image recognition module of the scene's image recognition model calls the Softmax function to perform image recognition and obtain a single-frame image recognition probability vector.

[0060] Iterate through all frames of the image data to be recognized after frame extraction, and input the single-frame image recognition probability vector of all frames after frame extraction into the timing integration module.

[0061] Based on the single-frame image recognition probability vectors of all frames after frame extraction, the temporal integration module of the scene's image recognition model is used to obtain the image recognition result.

[0062] Secondly, embodiments of the present invention provide an image recognition apparatus based on inter-frame features of images, used to implement an image recognition method, the apparatus comprising:

[0063] The global model deployment unit is used to build a global image recognition model and a global dynamic frame extraction model on a cloud server based on a dynamic weight federated learning mechanism, and deploy them to all edge computing gateways of the traffic monitoring system.

[0064] The migration fine-tuning unit is used at the edge computing gateway to perform migration fine-tuning on the global image recognition model and the global dynamic frame extraction model to obtain the scene's image recognition model and the scene's dynamic frame extraction model.

[0065] The frame processing unit is used to acquire the video stream from the camera, process the video stream into frames, and obtain continuous frames of image data to be recognized.

[0066] The feature extraction and fusion unit is used in the image recognition model of the application scenario to extract and fuse features from each frame of the image data to be recognized to obtain fused features.

[0067] The dynamic frame extraction unit is used to dynamically extract consecutive frames of the image data to be identified based on the fusion features and the dynamic frame extraction model of the application scenario, so as to obtain the image data to be identified in consecutive frames after frame extraction.

[0068] The image recognition unit is used for image recognition models in the application scenario. It performs image recognition on the image data to be recognized in consecutive frames after frame extraction to obtain the image recognition result.

[0069] A third aspect of this invention provides an electronic device, which includes:

[0070] At least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0071] The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0072] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention;

[0074] Figure 2 This is a flowchart illustrating the steps of an image recognition method based on inter-frame features provided in an embodiment of the present invention.

[0075] Figure 3 This is a schematic diagram of the functional units of an image recognition device based on inter-frame features provided in an embodiment of the present invention. Detailed Implementation

[0076] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0077] The present invention will be further described below with reference to the accompanying drawings.

[0078] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0079] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0080] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0081] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating device, a data storage module, a network communication module, a user interface module, and electronic programs.

[0082] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the image recognition device based on image inter-frame features stored in the memory 1005 through the processor 1001 and executes the image recognition method based on image inter-frame features provided in the embodiment of the present invention.

[0083] Reference Figure 2 The present invention provides an image recognition method based on inter-frame features, the method comprising:

[0084] S201: On the cloud server, based on the dynamic weight federated learning mechanism, a global image recognition model and a global dynamic frame extraction model are built and deployed to all edge computing gateways of the traffic monitoring system.

[0085] S202: At the edge computing gateway, the global image recognition model and the global dynamic frame extraction model are transferred and fine-tuned to obtain the scene's image recognition model and the scene's dynamic frame extraction model.

[0086] S203: Acquire the video stream from the camera, perform frame processing on the video stream, and obtain continuous frames of image data to be identified;

[0087] S204: The image recognition model for the application scenario extracts and fuses features from each frame of the image data to be recognized to obtain fused features;

[0088] S205: Based on the fusion features, the dynamic frame extraction model of the application scenario is used to dynamically extract the image data to be identified from the continuous frames, so as to obtain the image data to be identified from the continuous frames after frame extraction.

[0089] S206: Image recognition model for use cases, which performs image recognition on the image data to be recognized in consecutive frames after frame extraction, and obtains the image recognition result.

[0090] The technical solution provided in this application has at least the following beneficial effects:

[0091] Through dynamic weighted federated learning, a high-quality global model can be collaboratively built without sharing the original data. Each edge computing gateway only uploads the updated model parameters after local model training (rather than the original, sensitive image data) to the cloud for aggregation, fundamentally solving the "data silo" problem. This allows all participants to collaboratively train a higher-performance global model that integrates knowledge from all scenarios without leaking their core data assets. The dynamic weight mechanism is superior to traditional average aggregation, effectively improving the generalization ability and robustness of the global model. The edge computing gateway uses a small amount of local scene image data to quickly fine-tune the global model, obtaining a scene model specifically adapted to the current scene. This process requires only a small amount of labeled data and can even be combined with unsupervised or self-supervised learning, greatly reducing the accuracy degradation caused by "domain offset" and avoiding the high cost of full data labeling and retraining for each scene. This achieves high-precision, low-cost local deployment of the model. The reinforcement learning-based dynamic frame extraction model works in collaboration with the image recognition model. While ensuring recognition accuracy, it significantly reduces the number of frames requiring high-precision processing, thereby significantly reducing the computational load and power consumption of the edge computing gateway. This enables it to easily handle high-speed video streams and meet the stringent real-time requirements of traffic monitoring scenarios, effectively avoiding the missed detection problem caused by the loss of key frames in traditional fixed frame extraction methods. The cloud-edge-device collaborative architecture allows the model to be continuously iterated and updated in the cloud and seamlessly deployed to the edge. The online learning capability (experience replay pool) at the edge also enables the model to continuously adapt to minor changes in the environment. The entire framework has high flexibility and scalability. When the scene environment changes, the edge computing gateway can automatically adapt to the new situation and continuously optimize the model performance without manual intervention. This self-learning and self-optimization capability forms an efficient model iteration closed loop, greatly shortening the model update cycle, improving the framework's adaptability and maintenance efficiency, and perfectly meeting the needs of modern agile recognition.

[0092] In one alternative implementation, a global image recognition model and a global dynamic frame extraction model are constructed on a cloud server based on a dynamic weighted federated learning mechanism, and deployed to all edge computing gateways of the traffic monitoring system, including:

[0093] S2011: On a cloud server, use an image recognition algorithm to build an original image recognition model, use a reinforcement learning algorithm to build an original dynamic frame extraction model, and distribute the original image recognition model and the original dynamic frame extraction model to several training servers in different scenarios.

[0094] S2012: On the training server, collect local image datasets and input the local image datasets into the original image recognition model and the original dynamic frame extraction model for local training to obtain the local image recognition model and the local dynamic frame extraction model.

[0095] S2013: Collect the model parameter update amount, training data amount, and performance evaluation indicators of the local image recognition model and the local dynamic frame extraction model, and upload them to the cloud server.

[0096] S2014: On the cloud server, dynamic federated weights are generated based on the amount of training data and performance evaluation metrics of each training server, using a dynamic federated weight mechanism.

[0097] The formula is:

[0098]

[0099] In the formula, For the iteration number t, the first... Dynamic federated weights for each training server; The training data weights for training servers k and j are generated based on the amount of training data. Here are the performance evaluation weights for training servers k and j generated based on performance evaluation metrics; t is the current iteration number; k and j are the training server indicators; and K is the number of training servers. This is a balancing factor used to balance the impact of training data volume and performance evaluation metrics on dynamic federated weights.

[0100]

[0101] In the formula, Let z be the coefficient of the z-th performance evaluation index; The performance evaluation value of the z-th performance evaluation metric for training servers k,j is given. The performance evaluation metrics for the image recognition model include the F1-Score obtained by evaluating the performance of the local image recognition model on the validation set that is not involved in training, the area under the ROC curve (AUC), and the reciprocal of the confidence entropy. The performance evaluation metrics for the dynamic frame extraction model include the average cumulative reward and policy stability obtained by evaluating the performance of the local dynamic frame extraction model on the validation set that is not involved in training. z is the performance evaluation metric indicator.

[0102]

[0103] In the formula, The amount of data for training servers k and j;

[0104] S2015: Based on the model parameter update amount and corresponding dynamic federated weights of all training servers, the initial model parameters of the original image recognition model and the original dynamic frame extraction model are securely aggregated to obtain the global image recognition model and the global dynamic frame extraction model.

[0105] The formula is:

[0106]

[0107] In the formula, The global model parameters for iteration number t+1, t; For the first The number of model parameter updates per training server; t is the current iteration number; For the iteration number t, the first... Dynamic federated weights for each training server;

[0108] S2016: Deploy the global image recognition model and the global dynamic frame extraction model to all edge computing gateways in the traffic monitoring system that communicate with the cloud server.

[0109] In one alternative implementation, the image recognition model includes a single-frame static feature extraction module based on the ResNet-50 algorithm, an inter-frame dynamic feature extraction module based on the Pyramid Warping and Cost Volume (PWC-Net)-Convolutional Neural Network (CNN) algorithm, a dynamic gating module based on the Gated Recurrent Unit (GRU) algorithm, an image recognition module based on the Multi-Layer Perceptron (MLP) algorithm, and a temporal integration module based on the Long Short-Term Memory (LSTM) algorithm.

[0110] It's worth noting that ResNet-50 was chosen for the single-frame static feature extraction module due to its powerful feature extraction capabilities and mature pre-trained model. This effectively avoids the overfitting risk associated with training from scratch and accelerates convergence. In practice, the original global average pooling layer and fully connected classification layer of ResNet-50 are removed, retaining only its convolutional layers as a fixed or finely tuned single-frame static feature extractor. Given an input image, this module outputs the corresponding static feature vector. The inter-frame dynamic feature extraction module is based on PWC-Net-CNN (Pyramid, Warping, and Cost Volume). The algorithm is constructed using a CNN (PWC-Net) framework. The PWC-Net network is an efficient and accurate optical flow estimation network that takes two consecutive frames as input and outputs a dense optical flow field. To extract high-level dynamic features, the optical flow field output by PWC-Net is input into a lightweight CNN (e.g., several convolutional layers) for encoding, ultimately outputting a dynamic feature vector that encodes the scene change information between the two frames. The GRU (Governing Root Unit) of the dynamic gating module, with its update and reset gate mechanisms, naturally possesses the ability to filter and fuse information. Static and dynamic features are concatenated and used as input to the GRU unit at the current time step. The GRU outputs a fused feature; this fusion process is dynamic, and the GRU's gating mechanism automatically learns the fusion strategy based on the input features, achieving adaptive weighting of static and dynamic information. The image recognition module consists of one or more fully connected layers, whose input is the fused feature output by the dynamic gating module. After the nonlinear transformation of the MLP, the output is a vector of dimension N (N is the total number of preset traffic event categories, such as: normal, running a red light, speeding, congestion, accident, etc.). The Softmax function is called to convert this vector into a probability distribution, that is, a single-frame image recognition probability vector. The temporal integration module integrates the single-frame recognition results within a time window (e.g., 5 consecutive frames after dynamic frame extraction). By utilizing the continuity and consistency of the temporal sequence, it makes a more robust and accurate final judgment. For example, a single frame may be misjudged due to occlusion, but if several consecutive frames point to the same event, it can be confirmed as true. LSTM is particularly good at processing and predicting important events with long intervals and delays in time series. Its input is a sequence of single-frame image recognition probability vectors. The LSTM network learns the temporal dependencies in this sequence and outputs a final image recognition result that has been temporally smoothed and verified. It is usually also a probability vector or a specific category label.

[0111] The dynamic frame extraction model includes a meta-policy learning module based on the meta-policy optimization (MPO) algorithm and a dynamic frame extraction policy generation module based on the deep Q-network (DQN) algorithm. The dynamic frame extraction policy generation module includes an agent and an experience replay pool.

[0112] It's worth noting that the MPO of the meta-policy learning module is a gradient-based meta-learning algorithm. Its core idea is "learning how to optimize." Instead of directly learning the optimal policy for a task, it learns an optimizer (i.e., a meta-policy). This optimizer can quickly optimize an initial policy into a high-performance policy for a new task using a small number of gradient descent steps. A "task" corresponds to a traffic scenario. The meta-policy learning module itself can be regarded as a more advanced neural network or an optimization algorithm. It receives a small amount of empirical data collected by a DQN agent in a new scenario and then outputs the update amount for the parameters of the DQN agent. During cloud training, the meta-policy learning module will face the data distribution sampled from multiple different historical scenarios (tasks). Its optimization goal is: for any new task, the update amount output by the MPO module can maximize the expected cumulative reward of the DQN agent on that task after one or several update steps.

[0113] The dynamic frame extraction strategy generation module is the "executor" of the model, directly responsible for making real-time frame extraction decisions in specific scenarios. It is a classic reinforcement learning structure, comprising: Agent: Composed of a deep neural network, namely a deep Q-network, which receives the current state of the environment as input and outputs a Q-value vector. Each value in the vector corresponds to the expected cumulative reward of a possible action ("delete" or "skip"). The agent makes decisions by selecting the action with the highest Q-value; Experience replay pool: A storage unit used to store experience data generated by the agent's interaction with the environment. Each experience is usually a quadruple, representing the state at the corresponding time step, the action performed, the reward obtained, and the state at the next time step. The experience replay mechanism breaks the correlation between data, improving the stability and efficiency of training; State: All the information on which the agent makes decisions at the corresponding time step. In order to make wise decisions, the state space must contain rich contextual information, including fused features, historical action information, and historical reward information; Action space: Very simple, a discrete set of binary values, including "delete" and "skip";

[0114] The reward function of the dynamic frame-skipping strategy generation module is the core of guiding the agent's learning. Its design must be closely linked to the final goal, and the formula is as follows:

[0115]

[0116] In the formula, is the comprehensive reward value at time l, which is the immediate feedback obtained from the environment after the agent performs an action at time step l of the video stream. This value is the core signal that guides the DQN agent to update its policy network parameters. The higher the value of l, the better the action is in the current state. Let be the action at time l, which is the decision made by the agent at time l. In this embodiment, the action space is a discrete binary set: =1: Indicates skipping the l-th frame. , =0: indicates that the l-th frame image is deleted. ;l is a time indicator, and also indicates the number of image frames; This is an indicator function, a logical judgment function whose output value is only 0 or 1. When the condition inside the parentheses is true, it outputs 1; otherwise, it outputs 0. This is the image of the l-th frame; A set of keyframes; The computational cost reward is a negative value, representing the computational resource consumption caused by skipping a frame of an image. This item exists to punish unnecessary processing and incentivize the agent to skip as few frames as possible to save resources. Its absolute value can be set according to the actual computing power of the edge computing gateway. The information value reward is a positive value representing the information value that can be obtained by skipping a frame. This value is usually related to the "richness" or "novelty" of the frame; for example, frames with large inter-frame differences, complex textures, or containing potential target features. The value should be higher, which incentivizes the agent to skip frames with high information content. This value can be evaluated by a pre-trained lightweight network or estimated by intermediate features of the image recognition model, such as the norm of the feature vector. The penalty for missing a keyframe is a large negative value, which means that the agent made a deletion decision, but the deleted frame happened to be a keyframe. This is the most important penalty in this reward function. Its purpose is to force the agent to avoid missing key information at all costs and ensure the accuracy of downstream image recognition tasks. Setting a sufficiently large penalty value can ensure that the agent will place "avoiding missed detections" as a higher priority than "saving computation" during the learning process.

[0117] In one optional implementation, at the edge computing gateway, the global image recognition model and the global dynamic frame extraction model are fine-tuned to obtain the scene's image recognition model and the scene's dynamic frame extraction model, including:

[0118] S2021: At the edge computing gateway, collect several scene image data and add prediction labels to each scene image data to obtain a scene image dataset;

[0119] S2022: Freeze the global model parameters of the global image recognition model and the global dynamic frame extraction model, set the initial low-rank adaptation (LoRA) adapter, and perform transfer fine-tuning on the initial LoRA adapter based on the scene image dataset.

[0120] S2023: In each iteration of the migration fine-tuning process, an improved optimization algorithm is used to search for the adapter parameters of the initial LoRA adapter to obtain the optimized adapter parameters.

[0121] It is worth noting that the adapter parameters include the learning rate, the first low-rank matrix, the second low-rank matrix, and the scaling factor.

[0122] The learning rate is used to update the gradient step size of the LoRA adapter parameters. The dimension r of the low-rank matrix directly affects the model capacity and computational cost. When fine-tuning the model, it is usually necessary to update the pre-trained weight matrix. Traditional fine-tuning methods update directly, resulting in a huge number of parameters. The core idea of ​​LoRA is to decompose the weight update into the product of two low-rank matrices. The first low-rank matrix reduces the dimension from d to r, and the second low-rank matrix increases the dimension from r to d. The scaling factor alpha is used to control the intensity of LoRA updates. Usually, alpha / r is a key ratio.

[0123] S2024: Based on the optimized adapter parameters, optimize the initial LoRA adapters of the global image recognition model and the global dynamic frame extraction model to obtain the scene's image recognition model and the scene's dynamic frame extraction model.

[0124] It is worth noting that by freezing the scene model and configuring the LoRA adapter, the model can be transferred and fine-tuned, reducing the number of parameters involved in scene training, lowering the investment in training resources and computing power, and shortening the deployment and training cycle.

[0125] In one alternative implementation, during each iteration of migration fine-tuning, the improved Plant Root Growth Optimization (IPRGO) algorithm is used to search for the adapter parameters of the initial LoRA adapter to obtain optimized adapter parameters, including:

[0126] S20231: Based on the LoRA adapter optimization problem, a multi-objective optimization function is set, and the multi-objective optimization function is used as the fitness function of the IPRGO algorithm;

[0127] The formula is:

[0128]

[0129] In the formula, The fitness value of individual X in IPRGO; The cross-entropy loss objective function value of the model in the validation set for the scenario constructed for the adapter parameters corresponding to IPRGO individual X; The efficiency objective function value of the model in the validation set for the scenario constructed for the adapter parameters corresponding to IPRGO individual X is defined as the sum of the number of parameters and the computational cost. The objective function for the stability of the model in the validation set for the scenario constructed for the adapter parameters corresponding to IPRGO individual X is defined as the variance of performance. The fitness weighting coefficient can be adjusted according to actual needs; X is the IPRGO individual reference parameter.

[0130] S20232: Encode the adapter parameters of the LoRA adapter into individual vectors of the IPRGO algorithm, and set the IPRGO population parameters and the maximum number of iterations;

[0131] S20233: Based on the IPRGO population parameters, the initial IPRGO population is obtained by initializing using the Tent chaotic mapping sequence; each IPRGO individual in the IPRGO population corresponds to a candidate adapter parameter;

[0132] The formula is:

[0133]

[0134] In the formula, The i-th initial IPRGO individual in the initial IPRGO population; Let i be the i-th chaotic variable; represents the upper and lower bounds of the search space; i represents the IPRGO individual indicator;

[0135]

[0136] In the formula, Let i be the (i-1)th chaotic variable; compared with random initialization, chaotic initialization can ensure that the population is evenly distributed in the solution space, thus enhancing diversity.

[0137] S20234: Based on the scene image dataset, use the fitness function to obtain the fitness value of each initial IPRGO individual, and take the initial IPRGO individual with the best fitness value as the optimal solution;

[0138] In this embodiment, several scene image data are divided into a training sample set and a test sample set in a 7:3 ratio. The candidate adapter parameters corresponding to the initial IPRGO individuals are substituted into the LoRA adapter. The model with the LoRA adapter is trained using the training sample set and evaluated on the test sample set. The corresponding fitness value is obtained according to the fitness function.

[0139] S20235: Based on the fiber root growth, lateral root growth and taproot growth of the IPRGO algorithm, the initial IPRGO population is iteratively updated to obtain the updated IPRGO population.

[0140] The formula is:

[0141]

[0142] In the formula, The i-th updated IPRGO individual obtained by fiber root growth at iteration number t+1; The convergence factor; The optimal solution for iteration number t; The individual is a randomly selected IPRGO instance; t is the current iteration number;

[0143]

[0144] In the formula, These are the maximum and minimum values ​​of the convergence factor; t represents the maximum number of iterations; t represents the current number of iterations. , To adjust the parameters; It is the hyperbolic tangent function;

[0145]

[0146] In the formula, The i-th updated IPRGO individual is obtained by lateral root growth at iteration number t+1; is a Levy distribution random number; b is the Levy step size, and b∈[1,2]; The mean of the fiber roots over iteration number t;

[0147]

[0148] In the formula, The fiber root growth obtained at iteration number t is the first iteration. One updated IPRGO individual; For IPRGO individual indicators; The number of IPRGO individuals;

[0149]

[0150] In the formula, The i-th updated IPRGO individual obtained by the primary root growth at iteration number t+1; The reverse solution for the i-th updated IPRGO individual at iteration number t; The fitness function;

[0151]

[0152] In the formula, The reverse center point, determined by the dynamic boundary of the current search space, for iteration number t; It is a dynamic inverse factor that increases with iteration (intensified inverse exploration in later stages).

[0153] All , as well as Merge into the updated IPRGO population;

[0154] S20236: Based on the scene image dataset, use the fitness function to obtain the fitness value of each updated IPRGO individual, and update the updated IPRGO individual with the best fitness value as the optimal solution;

[0155] S20237: When the number of iterations reaches the maximum number of iterations or the fitness value of the optimal solution meets the requirements, terminate the iterative update of the IPRGO population and output the optimal solution of the current iteration;

[0156] S20238: Decode the individual vectors of the IPRGO individuals corresponding to the optimal solution to obtain the optimized adapter parameters.

[0157] In one optional implementation, a video stream from a camera is acquired, and the video stream is processed frame by frame to obtain continuous frames of image data to be identified, including:

[0158] S2031: At the edge computing gateway, connect to the video cameras within the communication range and capture the video streams from the cameras;

[0159] S2032: According to the preset time window, the video stream is frame-by-frame to obtain the original image data of consecutive frames;

[0160] S2033: Perform image preprocessing on the original image data of each frame to obtain the image data to be recognized for consecutive frames, and input the image data to be recognized for consecutive frames into the image recognition model of the scene in sequence.

[0161] In one optional implementation, an image recognition model for the scene is used to extract and fuse features from each frame of the image data to be recognized, resulting in fused features, including:

[0162] S2041: Single-frame static feature extraction module for image recognition model in application scenarios, extracting single-frame static features of each frame of image data to be recognized;

[0163] S2042: The inter-frame dynamic feature extraction module of the image recognition model for the application scenario extracts the optical flow field of the image data to be recognized in the current frame and the previous frame, and performs motion feature encoding based on the optical flow field to obtain inter-frame dynamic features;

[0164] S2043: The dynamic gating module of the image recognition model for the scene generates a gating vector, performs feature fusion on the static features of a single frame and the dynamic features between frames based on the gating vector, and inputs the fused features into the dynamic frame extraction model of the scene.

[0165] In one optional implementation, based on the fusion features and a dynamic frame-sampling model for the application scenario, dynamic frame-sampling is performed on the continuous frames of the image data to be identified, resulting in the extracted continuous frames of the image data to be identified, including:

[0166] S2051: Based on the fusion features, the meta-policy learning module of the dynamic frame extraction model for the use scenario adjusts the agent of the dynamic frame extraction policy generation module to obtain an updated agent.

[0167] S2052: Write the fused features of the image data to be identified in consecutive frames into the updated state space of the agent to obtain the updated state space;

[0168] S2053: Based on the updated state space, the updated agent of the dynamic frame-skipping strategy generation module is generated, and dynamic frame-skipping decisions are made in the action space to obtain the dynamic frame-skipping strategy and collect the corresponding real-time experience.

[0169] According to the dynamic frame extraction strategy, the image data to be identified in consecutive frames is dynamically extracted to obtain the image data to be identified in consecutive frames after frame extraction.

[0170] S2054: Randomly extract several historical experiences from the experience replay pool, and continuously train the dynamic frame-sampling model of the scene based on the real-time experience and several historical experiences to obtain an updated dynamic frame-sampling model.

[0171] In one optional implementation, an image recognition model for the scene is used to perform image recognition on the image data to be recognized in consecutive frames after frame extraction, and the image recognition result is obtained, including:

[0172] S2061: Single-frame static feature extraction module for image recognition model in application scenarios, extracting the single-frame static features of the image data to be recognized in each frame after frame extraction.

[0173] S2062: The inter-frame dynamic feature extraction module of the image recognition model for the application scenario extracts the optical flow field of the image data to be recognized in the current frame and the previous frame after frame extraction, and performs motion feature encoding based on the optical flow field after frame extraction to obtain the inter-frame dynamic features after frame extraction.

[0174] S2063: The dynamic gating module of the image recognition model for the application scenario generates a gating vector after frame extraction. Based on the gating vector after frame extraction, the static features of a single frame after frame extraction and the dynamic features between frames after frame extraction are fused to obtain the fused features after frame extraction.

[0175] S2064: Based on the fused features after frame extraction, the image recognition module of the scene's image recognition model calls the Softmax function to perform image recognition and obtain a single-frame image recognition probability vector;

[0176] S2065: Traverse all frames of the image data to be recognized after frame extraction, and input the single-frame image recognition probability vector of all frames after frame extraction into the timing integration module.

[0177] S2066: Based on the single-frame image recognition probability vectors of all frames after frame extraction, the temporal integration module of the scene's image recognition model is used to obtain the image recognition result.

[0178] Specific implementation examples are provided:

[0179] In the cloud server of the traffic control center, an original image recognition model (used to identify targets such as vehicles, pedestrians, non-motorized vehicles, and traffic signs) and an original dynamic frame extraction model are initialized. The cloud server distributes these two original models to 10 training servers (responsible for aggregating the analysis tasks of some cameras in the area).

[0180] Training server A collects anonymized data on illegally parked vehicles in area A (central business district) during off-peak hours for one week and trains the model locally. Due to the complex lighting conditions in area A, the model learns robust features under various lighting conditions such as strong light, backlight, and nighttime. Training server B collects anonymized pedestrian data in area B (residential area) for one week and uses this data to teach the model how to distinguish between normal pedestrians and illegally parked electric scooters on non-motorized vehicle lanes.

[0181] In each federated learning process, the training servers in areas A and B, etc., do not upload the original video data. Instead, they encrypt and upload the updated model parameters, training data volume, and performance evaluation metrics to the cloud service area. After receiving the model parameters from all 10 areas, the cloud server generates dynamically allocated aggregation weights based on the training data volume and model performance evaluation metrics of each area. For example, areas A and B, due to their data diversity and significant model improvements, receive higher aggregation weights. The cloud server obtains a global image recognition model and a global dynamic frame extraction model through weighted averaging, integrating knowledge from different areas, time periods, and scenarios across the city, resulting in extremely strong generalization capabilities. This step successfully breaks down data silos and, while ensuring the data privacy and security of each area, collaboratively trains a general model whose performance far exceeds the capabilities of any single area, effectively improving the model's generalization ability.

[0182] The cloud-based system deploys the final trained global model to all edge computing gateways. Taking Zone C (a complex intersection containing overpasses and tunnels) as an example, after receiving the global model, edge computing gateway C uses a small amount of representative real-time video stream data from the intersection over the past 24 hours to perform transfer fine-tuning on the model. This fine-tuning makes the model particularly suitable for the unique scenarios of Zone C. For example, it accurately identifies vehicles driving in the shaded area under the overpass and license plates when the light changes drastically at the tunnel entrances and exits. The fine-tuning also enables the dynamic frame-sampling model to learn the traffic flow patterns of Zone C. For instance, during morning and evening rush hours, when traffic is dense but slow, the model reduces the frame rate, focusing on identifying congestion and rear-end collision risks; while during off-peak hours, when traffic is fast, the model increases the frame rate, focusing on speeding and illegal lane changes. Through transfer fine-tuning, the global model is contextualized, perfectly adapting to the unique environment of Zone C, further improving the model's recognition accuracy and robustness in specific scenarios, and solving the performance degradation problem caused by environmental differences.

[0183] Taking a camera at a key intersection in Zone C as an example, its video stream is transmitted in real time to the edge computing gateway in Zone C for processing:

[0184] A dynamic frame-skipping model that inputs a video stream (25fps) into the scene:

[0185] Scenario 1: A car accelerates through an intersection when the yellow light is on. The dynamic frame extraction model of the scenario analyzes the changes in vehicle position, speed and traffic light status in several consecutive frames to determine that this is a highly dynamic and high-risk "running the yellow light" behavior. Therefore, the frame extraction density is reduced, and all frames in this critical 1-2 seconds (for example, extracting only 1 frame per second) are retained for subsequent detailed analysis.

[0186] Scenario 2: Vehicles are queuing normally at an intersection waiting for the red light. The dynamic frame extraction model of the scenario determines that the traffic flow is in a stationary or slow-moving state, with low risk. Therefore, the frame extraction density is greatly increased (for example, 1 frame is extracted every 3 frames) to save computing resources.

[0187] For each extracted image frame (such as the key frame of a vehicle running a yellow light), the static feature extraction module of the scene image recognition model extracts the single-frame static features of that frame (vehicle type, license plate, location, traffic light color, etc.), while the inter-frame dynamic feature extraction module extracts the optical flow field of the image data to be recognized in the current frame and the previous frame, and performs motion feature encoding based on the optical flow field to obtain the inter-frame dynamic features (such as the vehicle's motion vector, velocity change, acceleration, etc.).

[0188] The dynamic gating module generates a gating vector. Based on the gating vector, the static features of a single frame and the dynamic features between frames are concatenated or weighted and fused to form a fused feature. This fused feature includes both image content (vehicles, red lights) and target action (speeding towards the stop line). The image recognition module calls the Softmax function to output a single-frame image recognition probability vector, for example: {normal driving: 0.05, running a red light: 0.90, illegal lane changing: 0.03, other: 0.02}.

[0189] The probability vectors of single-frame images extracted from all five consecutive frames in the "running a yellow light" incident are input into the temporal integration module in chronological order. The temporal integration module analyzes the sequence of these five probability vectors and finds that the probability of normal driving is high when the vehicle is far from the stop line; as the vehicle approaches at high speed, the probability of running a red light rises sharply and stabilizes at a high level. Based on this temporal logical evolution, the temporal integration module finally outputs a high-confidence image recognition result: "vehicle ran a red light violation", and generates an evidence package containing license plate, time, location, violation screenshot and short video clip.

[0190] This invention also provides an image recognition device based on inter-frame features, referring to... Figure 3 The diagram illustrates the functional units of an image recognition device 300 based on inter-frame features according to the present invention. This device may include the following units:

[0191] The global model deployment unit 301 is used to build a global image recognition model and a global dynamic frame extraction model on a cloud server based on a dynamic weight federated learning mechanism, and deploy them to all edge computing gateways of the traffic monitoring system.

[0192] The migration fine-tuning unit 302 is used to perform migration fine-tuning on the global image recognition model and the global dynamic frame extraction model at the edge computing gateway to obtain the scene's image recognition model and the scene's dynamic frame extraction model.

[0193] The frame processing unit 303 is used to acquire the video stream from the camera, perform frame processing on the video stream, and obtain continuous frames of image data to be identified.

[0194] The feature extraction and fusion unit 304 is used for the image recognition model of the application scenario to extract and fuse features of each frame of image data to be recognized to obtain fused features;

[0195] The dynamic frame extraction unit 305 is used to dynamically extract frames from the image data to be identified in consecutive frames based on the fusion features and the dynamic frame extraction model of the application scenario, so as to obtain the image data to be identified in consecutive frames after frame extraction.

[0196] The image recognition unit 306 is used for the image recognition model of the application scenario to perform image recognition on the image data to be recognized in consecutive frames after frame extraction, and obtain the image recognition result.

[0197] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0198] Memory, used to store computer programs;

[0199] The processor, when executing a program stored in memory, implements the image recognition method based on inter-frame features of the present invention.

[0200] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0201] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0202] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image recognition method based on inter-frame features of the present invention.

[0203] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable vehicles (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0204] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0205] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0206] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0207] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0208] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image recognition method based on inter-frame features, characterized in that, The method includes: On the cloud server, a global image recognition model and a global dynamic frame extraction model are built based on a dynamic weighted federated learning mechanism and deployed to all edge computing gateways of the traffic monitoring system. The image recognition model includes a single-frame static feature extraction module based on the ResNet-50 algorithm, an inter-frame dynamic feature extraction module based on the PWC-Net-CNN algorithm, a dynamic gating module based on the GRU algorithm, an image recognition module based on the MLP algorithm, and a temporal integration module based on the LSTM algorithm. The dynamic frame extraction model includes a meta-policy learning module based on the MPO algorithm and a dynamic frame extraction policy generation module based on the DQN algorithm. The dynamic frame extraction policy generation module includes an agent and an experience replay pool. At the edge computing gateway, the global image recognition model and the global dynamic frame extraction model are fine-tuned through transfer learning to obtain the scene's image recognition model and the scene's dynamic frame extraction model, including: At the edge computing gateway, several scene image data are collected, and prediction labels are added to each scene image data to obtain a scene image dataset; Freeze the global model parameters of the global image recognition model and the global dynamic frame extraction model, set the initial LoRA adapter, and perform migration fine-tuning on the initial LoRA adapter based on the scene image dataset. During each iteration of migration fine-tuning, an improved optimization algorithm is used to search for the adapter parameters of the initial LoRA adapter to obtain optimized adapter parameters, including: Based on the LoRA adapter optimization problem, a multi-objective optimization function is set, and the multi-objective optimization function is used as the fitness function of the IPRGO algorithm; The adapter parameters of the LoRA adapter are encoded into individual vectors of the IPRGO algorithm, and the IPRGO population parameters and maximum number of iterations are set. Based on the IPRGO population parameters, the initial IPRGO population is obtained by initialization using the Tent chaotic mapping sequence; each IPRGO individual in the IPRGO population corresponds to a candidate adapter parameter. Based on the scene image dataset, the fitness function is used to obtain the fitness value of each initial IPRGO individual, and the initial IPRGO individual with the best fitness value is taken as the optimal solution. Based on the fiber root growth, lateral root growth and taproot growth of the IPRGO algorithm, the initial IPRGO population is iteratively updated to obtain the updated IPRGO population. Based on the scene image dataset, the fitness function is used to obtain the fitness value of each updated IPRGO individual, and the updated IPRGO individual with the best fitness value is updated as the optimal solution; When the number of iterations reaches the maximum number of iterations or the fitness value of the optimal solution meets the requirements, the iterative update of the IPRGO population is terminated, and the optimal solution of the current iteration is output. Decode the individual vectors of the IPRGO individuals corresponding to the optimal solution to obtain the optimized adapter parameters; Based on the optimized adapter parameters, the initial LoRA adapters of the global image recognition model and the global dynamic frame extraction model are optimized to obtain the scene's image recognition model and the scene's dynamic frame extraction model. The video stream from the camera is captured, and the video stream is processed frame by frame to obtain continuous frames of image data to be identified. The image recognition model for the application scenario extracts and fuses features from each frame of the image data to be recognized to obtain fused features; Based on the fusion features, the dynamic frame extraction model of the application scenario dynamically extracts frames from the image data to be identified in consecutive frames to obtain the image data to be identified in consecutive frames after frame extraction. The image recognition model for the application scenario performs image recognition on the image data to be recognized in consecutive frames after frame extraction, and obtains the image recognition result.

2. The image recognition method based on inter-frame features according to claim 1, characterized in that, On cloud servers, a global image recognition model and a global dynamic frame extraction model are built based on a dynamic weighted federated learning mechanism, and deployed to all edge computing gateways of the traffic monitoring system, including: On a cloud server, an original image recognition model is built using an image recognition algorithm, and an original dynamic frame extraction model is built using a reinforcement learning algorithm. The original image recognition model and the original dynamic frame extraction model are then distributed to several training servers in different scenarios. On the training server, a local image dataset is collected and input into the original image recognition model and the original dynamic frame extraction model for local training, resulting in a local image recognition model and a local dynamic frame extraction model. Collect the model parameter update amount, training data amount, and performance evaluation indicators of the local image recognition model and the local dynamic frame extraction model, and upload them to the cloud server. On the cloud server, dynamic federated weights are generated based on the amount of training data and performance evaluation metrics of each training server, using a dynamic federated weight mechanism. Based on the model parameter update amounts and corresponding dynamic federated weights of all training servers, the initial model parameters of the original image recognition model and the original dynamic frame extraction model are securely aggregated to obtain the global image recognition model and the global dynamic frame extraction model. Deploy the global image recognition model and the global dynamic frame extraction model to all edge computing gateways in the traffic monitoring system that communicate with the cloud server.

3. The image recognition method based on inter-frame features according to claim 2, characterized in that, The video stream from the camera is acquired, and frame processing is performed on the video stream to obtain continuous frames of image data to be identified, including: At the edge computing gateway, video cameras within the communication range are connected, and video streams from the cameras are captured. According to the preset time window, the video stream is frame-by-frame to obtain the raw image data of consecutive frames; Image preprocessing is performed on the raw image data of each frame to obtain image data to be recognized for consecutive frames, and the image data to be recognized for consecutive frames is then input into the image recognition model of the scene in sequence.

4. The image recognition method based on inter-frame features according to claim 3, characterized in that, The image recognition model for the application scenario extracts and fuses features from each frame of the image data to be recognized, resulting in fused features, including: The single-frame static feature extraction module of the image recognition model for the application scenario extracts the single-frame static features of each frame of the image data to be recognized. The inter-frame dynamic feature extraction module of the image recognition model for the application scenario extracts the optical flow field of the image data to be recognized in the current frame and the previous frame, and performs motion feature encoding based on the optical flow field to obtain the inter-frame dynamic features. The dynamic gating module of the image recognition model for the scene generates a gating vector. Based on the gating vector, it fuses the static features of a single frame and the dynamic features between frames, and then inputs the fused features into the dynamic frame extraction model of the scene.

5. The image recognition method based on inter-frame features according to claim 4, characterized in that, Based on the fusion features, a dynamic frame extraction model for the application scenario is used to dynamically extract frames from the image data to be identified in consecutive frames, resulting in the extracted image data to be identified in consecutive frames, including: Based on the fusion characteristics, the meta-policy learning module of the dynamic frame extraction model for the application scenario adjusts the agent of the dynamic frame extraction policy generation module to obtain an updated agent. The fused features of the image data to be identified in consecutive frames are written into the updated state space of the agent to obtain the updated state space. Based on the updated state space, the updated agent is generated using the dynamic frame-skipping strategy module. It makes dynamic frame-skipping decisions in the action space to obtain the dynamic frame-skipping strategy and collects the corresponding real-time experience. According to the dynamic frame extraction strategy, the image data to be identified in consecutive frames is dynamically extracted to obtain the image data to be identified in consecutive frames after frame extraction. Several historical experiences are randomly selected from the experience replay pool, and the dynamic frame-sampling model of the scene is continuously trained based on the real-time experience and several historical experiences to obtain an updated dynamic frame-sampling model.

6. The image recognition method based on inter-frame features according to claim 5, characterized in that, The image recognition model for the application scenario performs image recognition on the image data to be recognized in consecutive frames after frame extraction, and obtains the image recognition results, including: The single-frame static feature extraction module of the image recognition model for the application scenario extracts the single-frame static features of the image data to be recognized in each frame after frame extraction. The inter-frame dynamic feature extraction module of the image recognition model for the application scenario extracts the optical flow field of the image data to be recognized in the current frame and the previous frame after frame extraction, and performs motion feature encoding based on the optical flow field after frame extraction to obtain the inter-frame dynamic features after frame extraction. The dynamic gating module of the image recognition model for the application scenario generates a gating vector after frame extraction. Based on the gating vector after frame extraction, the static features of a single frame after frame extraction and the dynamic features between frames after frame extraction are fused to obtain the fused features after frame extraction. Based on the fused features after frame extraction, the image recognition module of the scene's image recognition model calls the Softmax function to perform image recognition and obtain a single-frame image recognition probability vector. Iterate through all frames of the image data to be recognized after frame extraction, and input the single-frame image recognition probability vector of all frames after frame extraction into the timing integration module. Based on the single-frame image recognition probability vectors of all frames after frame extraction, the temporal integration module of the scene's image recognition model is used to obtain the image recognition result.

7. An image recognition device based on inter-frame features, used to implement the image recognition method as described in any one of claims 1-6, characterized in that, The device includes: The global model deployment unit is used to build a global image recognition model and a global dynamic frame extraction model on a cloud server based on a dynamic weight federated learning mechanism, and deploy them to all edge computing gateways of the traffic monitoring system. The migration fine-tuning unit is used at the edge computing gateway to perform migration fine-tuning on the global image recognition model and the global dynamic frame extraction model to obtain the scene's image recognition model and the scene's dynamic frame extraction model. The frame processing unit is used to acquire the video stream from the camera, process the video stream into frames, and obtain continuous frames of image data to be recognized. The feature extraction and fusion unit is used in the image recognition model of the application scenario to extract and fuse features from each frame of the image data to be recognized to obtain fused features. The dynamic frame extraction unit is used to dynamically extract consecutive frames of the image data to be identified based on the fusion features and the dynamic frame extraction model of the application scenario, so as to obtain the image data to be identified in consecutive frames after frame extraction. The image recognition unit is used for image recognition models in the application scenario. It performs image recognition on the image data to be recognized in consecutive frames after frame extraction to obtain the image recognition result.

Images