Edge cloud collaborative vision system-oriented difficult attribute recognition domain self-adaption method
By selecting keyframes and performing data augmentation and model transfer learning in the edge-cloud collaborative vision system, the problems of low accuracy and high cost of difficult attribute recognition in complex environments by edge devices are solved, achieving efficient and low-cost difficult attribute recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU INNOVATION RES INST OF BEIJING UNIV OF AERONAUTICS & ASTRONAUTICS
- Filing Date
- 2026-02-09
- Publication Date
- 2026-04-24
AI Technical Summary
In edge-cloud collaborative vision systems, edge devices face complex and ever-changing environmental conditions that lead to a decrease in the accuracy of difficult attribute recognition. Existing methods are costly in terms of manpower and computing power, and do not fully utilize the spatiotemporal correlation and semantic consistency of video data, resulting in high annotation redundancy and low efficiency.
By receiving unlabeled video image sequences from edge devices, filtering key frames and labeling them, and using cloud-based data augmentation and model transfer learning, an updated model adapted to the target domain is generated, reducing manpower and computing costs and improving the accuracy of difficult attribute identification.
It significantly reduces the manpower and computing power costs in edge-cloud collaborative systems and improves the accuracy of difficult attribute identification by edge devices in diverse target domain scenarios.
Smart Images
Figure CN121921708A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of temporal adaptive technology for visual testing, and more specifically to a domain adaptive method for difficult attribute recognition in edge-cloud collaborative vision systems. Background Technology
[0002] With the rapid development of deep learning technology, visual artificial intelligence models have been widely applied in tasks such as image classification and object detection. However, edge devices in real-world deployment scenarios (such as edge cameras and mobile terminals) often face complex and variable environmental conditions, such as changes in lighting, differences in camera hardware, viewpoint shifts, and shifts in scene semantic distribution. This leads to a significant domain shift problem between the source domain data used in the training phase and the target domain data used in actual inference. This distribution difference has a particularly serious impact on difficult attribute recognition tasks with low visual feature discrimination (such as clothing color and subtle textures), causing a significant drop in the actual performance of the model on the edge.
[0003] To mitigate the domain drift problem, Test-Time Adaptation (TTA) technology has been proposed. Its core idea is to fine-tune the model online using unlabeled data from the target domain during the model deployment phase. Among these methods, pseudo-label-based approaches have attracted attention due to their low computational cost and lack of need for additional labeling. However, when the source and target domains differ significantly, the pseudo-labels generated by the model often contain a large amount of noise, which can mislead model updates and reduce generalization ability. To address this, some studies have introduced active learning mechanisms, selecting highly informative and representative samples for manual labeling to improve the adaptation effect. However, this strategy still faces challenges in edge-cloud collaborative systems: on the one hand, the large number and diverse scenarios of edge devices mean that independently performing labeling and training on each device would incur high manpower and computational costs; on the other hand, existing methods do not fully exploit the spatiotemporal correlation and semantic consistency of targets in video data, resulting in low sample selection efficiency and high labeling redundancy, making it difficult to support efficient and accurate domain adaptive transfer.
[0004] Therefore, under the edge-cloud collaborative architecture, how to improve the accuracy of end devices in identifying difficult attributes in diverse target domain scenarios is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of the above problems, the present invention proposes an adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems to overcome or at least partially solve the above problems.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] This invention provides an adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems, comprising the following steps: Receive video image sequences uploaded by edge devices that do not have a target domain labeled; Obtain the identity identifier and image sub-region sequence of each target instance from the video image sequence; Based on the identity identifier and the information value assessment results of each image sub-region, keyframes are selected from the image sub-region sequence; The keyframes are labeled to form an initial labeled query set with difficult attribute category labels; Based on the number of samples for each difficult attribute category in the initial labeled query set, an adaptive expansion factor is determined for each difficult attribute category; Based on the spatiotemporal correlation of the identity identifier of the key frame, and according to the augmentation factor of the difficult attribute category to which the key frame belongs, the key frame is augmented to generate an enhanced training data subset. The source domain visual pre-trained model is transferred to the cloud using the enhanced training data subset to obtain an updated model adapted to the target domain of the edge device, and the updated model is then sent to the edge device.
[0008] Furthermore, each image sub-region in the image sub-region sequence is determined by the x-coordinate of the centroid of the bounding box, the y-coordinate of the centroid of the bounding box, the width of the bounding box, and the height of the bounding box of the corresponding target instance.
[0009] Furthermore, the information value assessment result is obtained in the following way: The image sub-region is compressed and then input into the source domain visual pre-trained model for forward inference to obtain the inference score vector. From the reasoning scoring vector, correct and incorrect categories are preset according to the difficulty attribute categories, and the corresponding correct scores and incorrect scores are distinguished. Based on the correct and incorrect scores, the information value assessment result is calculated.
[0010] Furthermore, the information value assessment result is expressed as follows: info valid =weight c +weight w weight c =-log∑γ i s i weight w =-max(log (∑γ i s i -∑γ js j ),value min ) Among them, info valid Indicates the result of information value assessment; weight c Indicates the correct score harmonic value; weight w Indicates the harmonic value of the error rating; γ i s represents the weight of the i-th correct rating; i γ represents the i-th correct score; j s represents the weight of the j-th incorrect rating; j This represents the j-th incorrect rating; value min This represents the preset lower bound for incorrect ratings.
[0011] Furthermore, the step of selecting keyframes from the image sub-region sequence specifically includes: All image sub-regions are pre-classified according to the difficulty attribute category based on their inference results under the source domain visual pre-trained model, resulting in multiple category images; For each category of image, the first criterion is whether the identity identifier is repeated, and the second criterion is the ranking of the information value assessment results. N optimal image sub-regions are obtained as keyframes, where N is a preset positive integer.
[0012] Furthermore, the augmentation factor is negatively correlated with the number of samples of each difficult attribute category in the initial labeled query set.
[0013] Furthermore, the process of augmenting the keyframes based on their spatiotemporal correlation with the keyframes' identity identifiers and according to the augmentation factor of the difficult attribute category to which the keyframes belong, to generate an augmented training data subset, specifically includes: For each keyframe, other image sub-regions with the same identity identifier are searched from the video image sequence based on the corresponding identity identifier; From the found image sub-regions, filter out image sub-regions whose shooting time interval with the key frame is within a preset range; The selected image sub-regions are added to the augmented training data subset based on the augmentation factor determined by the augmentation factor of the difficult attribute category to which the keyframe belongs.
[0014] Furthermore, the transfer learning of the source domain vision pre-trained model using an enhanced subset of training data in the cloud specifically includes: freezing the preset network layer parameters in the source domain vision pre-trained model; and fine-tuning the network layers that are not frozen.
[0015] Furthermore, the step of obtaining the identity identifiers and image sub-region sequences of each target instance from the video image sequence is specifically achieved through detection, tracking, and relocation algorithms.
[0016] Furthermore, the source domain vision pre-trained model is trained using the YOLO architecture.
[0017] As can be seen from the above technical solution, compared with the prior art, the present invention discloses an adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems, which has the following beneficial effects: This invention addresses the challenges in intelligent systems where the large number of edge devices and diverse scene characteristics lead to a significant decrease in the accuracy of difficult attribute identification, resulting in high levels of human intervention and large data demands during retraining. Leveraging spatiotemporal correlation characteristics, a method for rapid data acquisition and efficient use is designed. This significantly reduces the human and computational costs required for data filtering and model training in edge-cloud collaborative systems, while also significantly reducing the time spent deploying high-performance target domain models. Furthermore, within the edge-cloud collaborative architecture, it improves the accuracy of edge devices in identifying difficult attributes across diverse target domain scenarios. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0019] Figure 1 This is a schematic diagram of the adaptive method for difficult attribute recognition domain in an edge-cloud collaborative vision system provided in an embodiment of the present invention. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] This invention discloses an adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems, such as... Figure 1 As shown, it includes the following steps: S1. Receive video image sequences of unlabeled target domains uploaded by edge devices; S2. Obtain the identity identifier and image sub-region sequence of each target instance from the video image sequence; S3. Based on the identity identifier and the information value assessment results of each image sub-region, keyframes are selected from the image sub-region sequence; S4. Annotate the keyframes to form an initial annotation query set with difficult attribute category labels; S5. Based on the number of pre-classified samples for each difficult attribute category in the initial labeled query set, determine an adaptive expansion factor for each difficult attribute category; S6. Based on the spatiotemporal correlation of key frame identity, and according to the augmentation factor of the difficult attribute category to which the key frame belongs, the key frame data is augmented to generate an enhanced training data subset. S7. In the cloud, use the enhanced training data subset to perform transfer learning on the source domain visual pre-trained model to obtain an updated model adapted to the target domain of the edge device, and then send the updated model to the edge device to complete the domain adaptation.
[0022] This method addresses the performance degradation of source domain models in identifying difficult attributes across multiple edge devices in different application scenarios. It compresses, labels, and augments a large amount of unlabeled data acquired from different edge devices. By leveraging the spatiotemporal correlation and semantic relevance of perceptual information, and utilizing cloud devices to efficiently complete the adaptive migration of the test time domain for different edge devices, it enables accurate attribute identification of multiple edge devices in the edge-cloud collaborative system under different scenarios and operating conditions.
[0023] Next, each of the above steps will be explained in detail.
[0024] In this embodiment of the invention, an RGB camera is used as the image sensor; a Rockchip RK3588 is used as the edge computing platform; an Intel Core i9 and an Nvidia RTX 4090 are used as the CPU and GPU of the cloud computing platform; DeepSORT is used as the visual detection and tracking algorithm model for obtaining the target sub-region sequence of the attribute to be identified; and the eight colors of the clothing worn by pedestrians—black, blue, brown, green, gray, purple, red, and white—are used as the attributes to be identified. The source domain consists of pedestrian images from the 3DPeS, MIT, TownCentre, and VIPeR public datasets, and the target domain consists of pedestrians in the suburbs at dusk whose main color of their clothing belongs to one of the above eight colors.
[0025] In step S1 above, a sequence of video images of the target domain that are not labeled is received from the edge device; In actual operation, the edge device and the cloud device are connected through the network and placed in the target domain environment. Scene video is collected for several hours at a frame rate of 10 frames per second to obtain unlabeled video image sequences in the target domain.
[0026] In step S2 above, the identity identifier and image sub-region sequence of each target instance are obtained from the video image sequence; In practice, detection, tracking, and relocalization methods such as DeepSORT can be used. Specifically, by first detecting and then tracking, and leveraging the correlation between target visual features and consecutive frames in the image sequence, semantic information and sub-region image information of each target instance (e.g., a pedestrian) are obtained and saved in each frame of the scene video. Among these: Semantic information refers to the identity information assigned to target instances in a video clip by the DeepSORT algorithm. Specifically, each target is assigned a unique ID as identity information in consecutive video clips. i Target identity information ID clipi It can be represented as: id clipi ={id1,id2,...,id n} Wherein, the set element id represents the ID identity identifier of the target entity, and its subscript represents the index of the ID identity identifier in the set; The sub-region image information refers to the location information extracted from the scene video using the DeepSORT algorithm. Specifically, the bounding box position information of the target instance corresponding to each ID in different frames is used as an image sub-region sequence. Each image sub-region p in the image sub-region sequence... id Represented as: p id =(x,y,w,h) Where x represents the horizontal coordinate of the centroid of the bounding box; y represents the vertical coordinate of the centroid of the bounding box; w represents the width of the bounding box; and h represents the height of the bounding box.
[0027] In step S3 above, keyframes are selected from the image sub-region sequence based on the identity identifier and the information value assessment results of each image sub-region; wherein: (1) The information value assessment results are obtained in the following ways: 1) After compressing the image sub-regions, input them into the source domain visual pre-trained model for forward inference to obtain the inference score vector; whereby the source domain visual pre-trained model is an image recognition model trained by single- or dual-stage model architectures such as YOLO; the obtained inference score vector is expressed as: v inf =(s1,s2,...,s k ) Among them, s k This represents the confidence score for the k-th difficult attribute category; 2) Based on the inference scoring vector v infCalculate the information value assessment results of image sub-regions. valid Specifically, based on the category characteristics of the difficult attribute, the ranges of the correct and incorrect categories are predefined, and the inference scoring vector v is selected. inf The information value assessment result is calculated using correct and incorrect ratings. valid The information value assessment result (info) valid It can be described as the sum of the correct rating harmonics and the incorrect rating harmonics, expressed as: info valid =weight c +weight w Where, weight c Indicates the correct score harmonic value; weight w The harmonic value represents the incorrect rating; since the number of correct ratings may be greater than one, in this embodiment of the invention, a logarithmic expression that can be represented as a weighted average is used to describe the two harmonic values: weight c =-log∑γ i s i weight w =-max(log (∑γ i s i -∑γ j s j ),value min ) Where, γ i s represents the weight of the i-th correct rating; i γ represents the i-th correct score; j s represents the weight of the j-th incorrect rating; j This represents the j-th incorrect rating; value min Indicates the lower bound of incorrect scoring; (2) Selecting keyframes from the image sub-region sequence, specifically including: 1) Based on the inference results of the source domain visual pre-trained model, pre-classify all image sub-regions according to the difficulty attribute category to obtain multiple category image sets pre. set , is represented as: pre set ={cls set1 ,cls set2 ,...,cls setk} Among them, cls setk This represents the image sub-region corresponding to the k-th difficult attribute category; 2) For each type of image, for each category of image, the first criterion is whether the identity identifier is repeated, and the second criterion is the ranking of the information value assessment results. N optimal image sub-regions are obtained as keyframes, where N is a preset positive integer. Based on the balanced distribution of samples, N is distributed as evenly as possible to each category.
[0028] In practice, the execution order of the first and second judgment conditions can be specified according to actual needs.
[0029] ① Taking the case where the first condition is checked first, and then the second condition is checked, as an example: Image sub-regions are filtered based on identity identifiers, and the highest-valued image sub-regions for each identity identifier are retained based on the query set size. This step involves compressing the image sets for each category to facilitate sample classification balance. Then, the filtered image sub-regions (cls) are... seti Sort the images in descending order of information value assessment results; finally, select the top N image sub-regions with the highest information value assessment results from the sorted results as keyframes.
[0030] ② Taking the case where the second condition is checked first, followed by the first condition, as an example: Image sub-region cls seti Sort the images in descending order of information value assessment results, and select the image sub-regions with the highest information value assessment results from the sorted results; then, filter the selected image sub-regions according to the identity identifier, and retain the top N image sub-regions with the highest information value assessment results under each identity identifier as keyframes.
[0031] The above image sub-region cls seti Represented as: cls seti ={v inf1 , v inf2 , ..., v infm} Among them, cls seti This represents the sorting result of the i-th class of images; v infm This represents the information value assessment result corresponding to the m-th image sub-region; This step selects key image data that meets the preset quantity requirements for each image sub-region sequence, based on whether the ID identifier is repeated as the first criterion and the information value assessment result ranking as the second criterion.
[0032] In step S4 above, keyframes are annotated by the prompter to form an initial annotation query set with difficult attribute category labels.
[0033] In step S5 above, based on the number of samples for each difficult attribute category in the initial labeled query set, an adaptive augmentation factor is determined for each difficult attribute category to ensure a roughly balanced sample distribution; then, the augmentation factor set is saved. This augmentation factor set is represented as: factor set ={factor1, factor2, ..., factor k} Where, factor n This represents the augmentation factor corresponding to the k-th difficult attribute category. The augmentation factor is negatively correlated with the number of samples for each difficult attribute category in the initial labeled query set.
[0034] In step S6 above, based on the spatiotemporal correlation of the keyframe's identity identifier and according to the augmentation factor of the hard attribute category to which the keyframe belongs, the keyframe is augmented to generate an augmented training data subset; taking a single category of hard attribute as an example, it specifically includes: For each keyframe, based on the corresponding identity identifier, other image sub-regions with the same identity identifier are searched from the video image sequence; From the found image sub-regions, filter out image sub-regions whose shooting time interval with the keyframe is within a preset range; The selected image sub-regions are added to the augmented training data subset based on the augmentation factor (e.g., 4) determined by the augmentation factor of the difficult attribute category to which the keyframe belongs.
[0035] In step S7 above, the source domain vision pre-trained model is transferred to the cloud using an enhanced subset of training data. Specifically, the network parameters of a preset number of layers (e.g., 5 layers) in the source domain vision pre-trained model near the input are frozen; only the unfrozen network layers are fine-tuned to obtain an updated model adapted to the target domain of the edge device, and the updated model is sent to the edge device to complete the domain adaptation.
[0036] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0037] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems, characterized in that, Includes the following steps: Receive video image sequences uploaded by edge devices that do not have a target domain labeled; Obtain the identity identifier and image sub-region sequence of each target instance from the video image sequence; Based on the identity identifier and the information value assessment results of each image sub-region, keyframes are selected from the image sub-region sequence; The keyframes are labeled to form an initial labeled query set with difficult attribute category labels; Based on the number of samples for each difficult attribute category in the initial labeled query set, an adaptive expansion factor is determined for each difficult attribute category; Based on the spatiotemporal correlation of the identity identifier of the key frame, and according to the augmentation factor of the difficult attribute category to which the key frame belongs, the key frame is augmented to generate an enhanced training data subset. The source domain visual pre-trained model is transferred to the cloud using the enhanced training data subset to obtain an updated model adapted to the target domain of the edge device, and the updated model is then sent to the edge device.
2. The adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems as described in claim 1, characterized in that, Each image sub-region in the image sub-region sequence is determined by the x-coordinate of the centroid of the bounding box, the y-coordinate of the centroid of the bounding box, the width of the bounding box, and the height of the bounding box of the corresponding target instance.
3. The adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems as described in claim 1, characterized in that, The information value assessment results are obtained in the following way: The image sub-region is compressed and then input into the source domain visual pre-trained model for forward inference to obtain the inference score vector. From the reasoning scoring vector, correct and incorrect categories are preset according to the difficulty attribute categories, and the corresponding correct scores and incorrect scores are distinguished. Based on the correct and incorrect scores, the information value assessment result is calculated.
4. The adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems as described in claim 3, characterized in that, The information value assessment result is expressed as follows: info valid =weight c +weight w weight c =-log∑γ i s i weight w =-max(log (∑γ i s i -∑γ j s j ),value min ) Among them, info valid Indicates the result of information value assessment; weight c Indicates the correct score harmonic value; weight w Indicates the harmonic value of the error rating; γ i s represents the weight of the i-th correct rating; i γ represents the i-th correct score; j s represents the weight of the j-th incorrect rating; j This represents the j-th incorrect rating; value min This represents the preset lower bound for incorrect ratings.
5. The adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems as described in claim 1, characterized in that, The step of selecting keyframes from the image sub-region sequence specifically includes: All image sub-regions are pre-classified according to the difficulty attribute category based on their inference results under the source domain visual pre-trained model, resulting in multiple category images; For each category of image, the first criterion is whether the identity identifier is repeated, and the second criterion is the ranking of the information value assessment results. N optimal image sub-regions are obtained as keyframes, where N is a preset positive integer.
6. The adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems as described in claim 1, characterized in that, The augmentation factor is negatively correlated with the number of samples for each difficult attribute category in the initial labeled query set.
7. The adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems as described in claim 1 or 6, characterized in that, The process of establishing spatiotemporal correlation based on the identity identifier of the keyframes and augmenting the keyframes according to the augmentation factor of the difficult attribute category to which the keyframes belong, to generate an augmented training data subset, specifically includes: For each keyframe, other image sub-regions with the same identity identifier are searched from the video image sequence based on the corresponding identity identifier; From the found image sub-regions, filter out image sub-regions whose shooting time interval with the key frame is within a preset range; The selected image sub-regions are added to the augmented training data subset based on the augmentation factor determined by the augmentation factor of the difficult attribute category to which the keyframe belongs.
8. The adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems as described in claim 1, characterized in that, The method of performing transfer learning on the source domain vision pre-trained model using an enhanced subset of training data in the cloud specifically includes: freezing the preset network layer parameters in the source domain vision pre-trained model; and fine-tuning the network layers that are not frozen.
9. The adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems as described in claim 1, characterized in that, The process of obtaining the identity identifiers and image sub-region sequences of each target instance from the video image sequence is specifically achieved through detection, tracking, and relocation algorithms.
10. The adaptive method for difficult attribute recognition domains in edge-cloud collaborative vision systems as described in claim 1, characterized in that, The source domain vision pre-trained model was trained using the YOLO architecture.