Target detection model training method based on graph representation domain adaptation

Abstract

The application discloses a target detection model training method based on graph representation domain adaptation. First, a target detection model is pre-trained based on a data set of a source domain, and the ability of accurately predicting target categories and boundary boxes is learned in the source domain. Then, a feature map is converted into a node group based on graph representation, and a contrast distribution loss and a node covariance alignment loss are used to refine and strengthen the nodes, so that the domain characteristics and the invariance characteristics of categories can be decoupled, the domain style elements can be selectively removed, and the generalization ability of the target detection model can be improved. For abnormal nodes in the graph, the graph is optimized by using an abnormal node filtering loss, the proportion of strange semantic nodes is reduced, and the detection precision of the trained target detection model is improved. The trained target detection model has good robustness and generalization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and particularly relates to a traffic scene target detection model training method based on graph representation domain adaptation. BACKGROUND

[0002] In recent years, with the development of artificial intelligence technology, automatic driving has made great progress. The detection and recognition of traffic targets, as a crucial part of the automatic driving perception system, play an important auxiliary role in vehicle driving decision-making in intelligent transportation systems. The application scenarios of automatic driving are relatively complex, and the target detection model is required to adapt to detection tasks in different scenarios, that is, the target detection model needs to have good domain adaptation ability and generalization ability.

[0003] In order to solve this difficulty, the industry has researched unsupervised domain adaptation technology to transfer the target detector from the labeled source domain to the unlabeled target domain. Previous studies usually describe the semantic space from the class level, and use the center of the class to represent the semantic knowledge of the class. However, these methods represented based on the class center have very big defects. First of all, they ignore the intra-class variance and directly manually align the center of the class. Because the size and appearance of the object to be detected are different, the intra-class variance covers the basic information representing the class conditional distribution, such as scale and shape, which should be aligned in domain adaptation. Ignoring the intra-class variance may lead to a large number of inadapted target instances, as well as potential overlap and classification errors of different class conditional distributions. Moreover, these works follow the Gaussian assumption to describe the features step by step, which is not optimal in the non-convex deep feature space. Although some current works embed image features into non-Euclidean space for expression to perform adaptive alignment, they still ignore the noise introduction when sampling nodes, resulting in poor domain adaptation performance.

[0004] Therefore, the generalization ability of the target detection model trained by the existing training method is not good, and the target detection model cannot obtain high detection precision in the target domain, which limits the application of the target detection model in real traffic scenes. SUMMARY

[0005] The main purpose of the present application is to provide a traffic scene target detection model training method based on graph representation domain adaptation, an intelligent terminal, a computer readable storage medium and a domain adaptation target detection method for complex traffic scenes, which aims to solve the problem that the generalization ability of the target detection model after training is not good and cannot obtain high detection precision in the target domain.

[0006] In order to achieve the above purpose, the first aspect of the present application provides a traffic scene target detection model training method based on graph representation domain adaptation, comprising:

[0007] Pre-training a target detection model based on a data set of a source domain;

[0008] inputting the source domain image and the target domain image into the pre-trained target detection model respectively to obtain a source domain feature map and a target domain feature map;

[0009] converting the source domain feature map and the target domain feature map into node groups based on graph representation respectively to obtain a first source domain node group and a first target domain node group;

[0010] complementing nodes for missing semantics in the first source domain node group and the first target domain node group based on node-based contrastive learning to obtain a second source domain node group and a second target domain node group;

[0011] calculating a contrastive distribution loss based on the first source domain node group, the first target domain node group, the second source domain node group and the second target domain node group;

[0012] extracting a covariance matrix of the second source domain node group and the second target domain node group respectively, and obtaining a variance matrix according to all the covariance matrices;

[0013] calculating a node covariance alignment loss according to the variance matrix, the second source domain node group and the second target domain node group;

[0014] converting the second source domain node group and the second target domain node group into graph structures and performing graph optimization to obtain a source domain graph node set, a source domain graph structure, and a target domain graph node set, a target domain graph structure;

[0015] representing distribution offset of the graph by using an environmental variable, and calculating an abnormal node filtering loss according to the source domain graph node set, the source domain graph structure, the target domain graph node set and the target domain graph structure;

[0016] optimizing the pre-trained target detection model according to the contrastive distribution loss, the node covariance alignment loss and the abnormal node filtering loss to obtain a trained target detection model for traffic target detection and identification.

[0017] Optionally, the extracting the covariance matrix of the second source domain node group and the second target domain node group respectively, and obtaining the variance matrix according to all the covariance matrices, comprises:

[0018] extracting a covariance matrix of the second source domain node group and the second target domain node group respectively to obtain a source domain covariance matrix and a target domain covariance matrix;

[0019] splicing the source domain covariance matrix and the target domain covariance matrix to obtain a spliced covariance matrix;

[0020] Calculate the mean and variance of the spliced covariance matrix to obtain the variance matrix.

[0021] Optionally, according to the variance matrix, the second source domain node group and the second target domain node group, a node covariance alignment loss is calculated, including:

[0022] The upper triangular elements of the variance matrix are clustered to obtain an upper triangular matrix.

[0023] Splice the second source domain node group and the second target domain node group to obtain a spliced node group.

[0024] Extract the covariance matrix of the spliced node group to obtain a spliced covariance matrix.

[0025] According to the upper triangular matrix and the spliced covariance matrix, the node covariance alignment loss is calculated.

[0026] Optionally, the node-based contrastive learning includes, in the first source domain node group and the first target domain node group, generating a supplementary node for a missing semantic to obtain a second source domain node group and a second target domain node group, including:

[0027] The nodes of the first source domain node group and the first target domain node group are contrastively learned by using a global node memory pool, and a supplementary node is generated for a missing semantic in the first source domain node group and the first target domain node group to obtain a supplemented source domain node group and a target domain node group.

[0028] The supplemented source domain node group and the target domain node group are respectively input into a shared self-attention module to obtain the second source domain node group and the second target domain node group.

[0029] Optionally, the contrastive distribution loss is calculated based on the first source domain node group, the first target domain node group, the second source domain node group and the second target domain node group, including:

[0030] Map the first source domain node group and the first target domain node group to an implicit feature space to obtain a first source feature space and a first target feature space.

[0031] Map the second source domain node group and the second target domain node group to an implicit feature space to obtain a second source feature space and a second target feature space.

[0032] According to the similarity between the first source feature space and the second source feature space, and the similarity between the first target feature space and the second target feature space, the contrastive distribution loss is obtained.

[0033] Optionally, after obtaining the source domain graph node set and the target domain graph node set, the method further comprises:

[0034] calculating a first node classification loss value of each graph node in the source domain graph node set;

[0035] calculating a second node classification loss value of each graph node in the target domain graph node set;

[0036] accumulating all the first node classification loss values and all the second node classification loss values to obtain a node classification loss value for optimizing the pre-trained target detection model.

[0037] Optionally, after obtaining the source domain graph structure and the target domain graph structure, the method further comprises:

[0038] obtaining a similarity matrix according to semantic relationships based on the source domain graph node set and the target domain graph node set;

[0039] calculating a structure loss value according to the similarity matrix, the source domain graph structure and the target domain graph structure for optimizing the pre-trained target detection model.

[0040] The second aspect of the present application provides a domain adaptation target detection method for complex traffic scenes, wherein the method comprises:

[0041] training a target detection model by using any one of the above traffic scene target detection model training methods based on graph representation domain adaptation to obtain a trained target detection model;

[0042] inputting a traffic image into the trained target detection model for target detection to obtain a target detection result.

[0043] The third aspect of the present application provides an intelligent terminal, which comprises a memory, a processor, and a traffic scene target detection model training program based on graph representation domain adaptation stored in the memory and executable on the processor, wherein the traffic scene target detection model training program based on graph representation domain adaptation implements the steps of any one of the above traffic scene target detection model training methods based on graph representation domain adaptation when executed by the processor.

[0044] The fourth aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a traffic scene target detection model training program based on graph representation domain adaptation, and the traffic scene target detection model training program based on graph representation domain adaptation implements the steps of any one of the above traffic scene target detection model training methods based on graph representation domain adaptation when executed by a processor.

[0045] From the above, the application firstly pre-trains the target detection model based on the data set of the source domain, learns the ability of accurately predicting the target category and the boundary box in the source domain, then refines and strengthens the nodes by converting the feature map into the node group based on the graph representation, adopts the contrast distribution loss and the node covariance alignment loss, can decouple the domain characteristics and the invariance characteristics of the category, selectively removes the domain style elements, and improves the generalization ability of the target detection model. For the abnormal nodes in the graph, the abnormal node filtering loss is used to optimize the graph, reduces the proportion of strange semantic nodes, and improves the detection accuracy of the trained target detection model. The trained target detection model has good robustness and generalization. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0047] Figure 1 is a traffic scene target detection model training method flowchart based on graph representation domain adaptation provided by the embodiment of the present application;

[0048] Figure 2 is a feature pyramid network schematic diagram provided by the embodiment of the present application;

[0049] Figure 3 is a training module schematic diagram provided by the embodiment of the present application;

[0050] Figure 4 is a training network schematic diagram provided by the embodiment of the present application;

[0051] Figure 5 is a contrast node supplement method flowchart provided by the embodiment of the present application;

[0052] Figure 6 is an internal structure principle block diagram of a smart terminal provided by the embodiment of the present application. DETAILED DESCRIPTION

[0053] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application, but it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0054] It should be understood that the word "comprising" when used in the specification and claims herein, specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0055] It should also be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the description of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0056] It should further be understood that the term "and / or" as used in the specification and in the claims, means any one of the items, any combination of the items, and all possible combinations of the items in the related list.

[0057] As used in the description of the present application and the appended claims, the term "if' can be interpreted as meaning "when" or "once" or "in response to a determination" or "in response to a detection" depending on the context. Similarly, the phrase "if determined" or "if detected [the described condition or event]" can be interpreted as meaning "once determined" or "in response to a determination" or "once detected [the described condition or event]" or "in response to a detection [the described condition or event]" depending on the context.

[0058] The technical solutions in the embodiments of the present application are clearly and completely described below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0059] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other manners different from those described herein, and a person of ordinary skill in the art can make similar generalizations without departing from the spirit and scope of the present application, and therefore the present application is not limited to the specific embodiments disclosed below.

[0060] Since the generalization ability of the target detection model trained by using the existing training method is not good, the target detection model cannot obtain high target detection precision in various scenes, which limits the application of the target detection model in real traffic scenes, especially in complex traffic scenes.

[0061] To solve the above technical problems, the application provides a traffic scene target detection model training method based on graph representation domain adaptation. The method converts a feature map into a node group based on graph representation, refines and strengthens the nodes by using node-based contrast learning and graph covariance, can decouple domain features and class invariance features, and selectively removes domain style elements to improve the generalization ability of the target detection model. For abnormal nodes generated in non-Euclidean space and abnormal nodes generated during sampling, the graph is optimized based on knowledge sharing and distribution generalization technology to reduce the proportion of strange semantic nodes in the generated graph and improve the detection accuracy of the trained target detection model. The trained target detection model has good robustness and generalization.

[0062] Exemplary method

[0063] The embodiment provides a traffic scene target detection model training method based on graph representation domain adaptation, which is deployed on an electronic terminal and is suitable for road automatic driving, unmanned vehicle park delivery and other scenes. The training process of the target detection model includes a pre-training stage and an unsupervised domain adaptation stage. Self-supervised learning and domain adaptation technology are used to enable the target detection model to adapt to the target domain and perform target detection without target labels.

[0064] As shown in Figure 1 , the method specifically comprises the following steps:

[0065] Step S100: pre-training a target detection model based on a data set of a source domain;

[0066] This step belongs to the pre-training stage. The domain is a specific scene of automatic driving, the source domain refers to a traffic scene providing a data set required for pre-training the target detection model, and the target domain refers to a traffic scene in which the trained target detection model is expected to generalize. The target detection model is a model used to detect traffic targets such as vehicles and pedestrians in collected traffic images, and the detection result includes framing the traffic targets to indicate the area and position of the traffic targets and identifying the type of the traffic targets. The target detection model in the embodiment is an FCOS (Fully Convolutional One-Stage Object Detection: fully convolutional one-stage object detection) detector, but is not limited to this model.

[0067] Generally, sufficient training data can be obtained in the source domain, while the training data in the target domain is less. Therefore, the target detection model needs to be pre-trained based on the data set of the source domain to learn the ability to accurately predict the target class and the bounding box in the source domain. Then, the unsupervised domain adaptation stage is used to enable the target detection model to adapt to the data distribution of the target domain and realize target detection in the target domain.

[0068] Specifically, the data of the source domain contains labeled images and corresponding bounding box labels. The data of the target domain only contains unlabeled images. In the pre-training stage, the labeled data of the source domain is used for the conventional FCOS model training to learn the ability to accurately predict the target class and the bounding box in the source domain. The specific process involves feature extraction, feature fusion, classification and regression loss, etc.

[0069] In the feature extraction part, a pre-trained convolutional neural network is used as a feature extractor, and the commonly used choices are ResNet, ResNeXt, etc. The embodiment selects a ResNet network as the feature extractor, which has multiple convolutional layers and pooling layers, and the final output is the feature map. Assuming that the input image is x, the feature extraction network is Backbone, and the output feature map of the feature extraction part is F backbone (x), whose size is H x W x C backbone , where H and W are the height and width of the feature map, and C backbone is the number of channels.

[0070] As shown in Figure 2 , the feature fusion part uses the idea of Feature Pyramid Network (FPN) to fuse features from different levels to detect targets of different scales at the same time. Assuming that the features output by different stages of ResNet are C2, C3, C4, and C5, which correspond to different resolutions and receptive fields. FCOS uses an upsampling method to upsample the features of the deeper level to the same resolution as the features of the shallower level, and then fuses them. Let F C2 , F C3 , F C4 , and F C5 be the feature maps output by different stages of ResNet, whose sizes are H / 4 x W / 4 x C2, H / 8 x W / 8 x C3, H / 16 x W / 16 x C4, and H / 32 x W / 32 x C5, respectively. Among them, H / 4, H / 8, H / 16, H / 32, and W / 4, W / 8, W / 16, W / 32 are the height and width of the corresponding feature map. First, a 1 x 1 convolution kernel is applied to F C5 to obtain an additional feature map F P5 , which is used to predict the largest scale target. Then, for the feature maps F C4 and F C3 , a 1 x 1 convolution kernel is applied to obtain additional feature maps F P4 and F P3 , which are used to predict medium-sized targets and smaller targets. Then, feature fusion is performed. F P5 does not need to be fused, and F P4 and F P3Need to be fused with the shallower level feature map F C3 and F C2 . The way of fusion is through upsampling and channel number adjustment (1x1 convolution kernel), so that F P4 and F P3 have the same size as F C3 and F C2 , and then element-wise addition to obtain the fused feature map F P4′ and F P3′ . The obtained F P4′ and F P3′ will be used to predict medium and small scale targets. Finally, the feature fused feature map is F fusion =F P3′ , F P4′ , F P5 , where F P3′ , F P4′ , F P5 have the same size H / 8xW / 8xC3, H / 16xW / 16xC4, H / 32xW / 32xC5. Finally, the feature fused network output F fusion is a set containing multiple scale feature maps, each of which is used to predict targets of different scale sizes. These feature maps will be used for subsequent target classification and bounding box regression tasks.

[0071] The following steps belong to the unsupervised domain adaptation stage, in which the target detection model is trained using a self-supervised learning method.

[0072] Step S200: input the source domain image and the target domain image into the pre-trained target detection model respectively to obtain the source domain feature map and the target domain feature map;

[0073] Referring to Figure 3 , after pre-training, the collected source domain image and target domain image are respectively input into the pre-trained target detection model, and the features of the target domain image and the source domain image are extracted using the above feature extraction and feature fusion part to obtain the source domain feature map and the target domain feature map.

[0074] For the target domain feature map, when using the pre-trained target detection model to perform target detection prediction, since the target domain has no real label, the classification and regression loss cannot be directly calculated. Therefore, the predicted bounding box and score are used to generate pseudo labels: for each position (x, y), if the predicted score is higher than the preset threshold, it is considered that there is a target at this position, and the predicted class score and bounding box prediction parameters are recorded.

[0075] Step S300: convert the source domain feature map and the target domain feature map into graph representation-based node groups respectively to obtain a first source domain node group and a first target domain node group;

[0076] For the source domain feature map, the spatial uniform sampling method is used again to collect the pixels in the real bounding box to obtain foreground nodes of the attention class and a small amount of background nodes of the attention background outside the bounding box, and the foreground nodes and the background nodes constitute the node group of the source domain.

[0077] For the target domain feature map, forward propagation is performed again to obtain a pseudo-score map Then, the pixels of are taken as foreground nodes, and the pixels of are taken as background nodes, and the foreground nodes and the background nodes constitute the node group of the target domain.

[0078] Then, the node group of the source domain and the node group of the target domain are transformed into a graph space by using a nonlinear transformation to obtain a first source domain node group and a first target domain node group The node alignment loss is calculated at the same time to prevent the node from overfitting the feature map.

[0079] Step S400: based on the node-based contrastive learning, the missing semantics are supplemented in the first source domain node group and the first target domain node group to obtain a second source domain node group and a second target domain node group.

[0080] The spatial uniform sampling roughly models the semantic sampling of the graph on the real bounding box, which inevitably leads to biased domain adaptation. The real bounding box usually contains non-informational background pixels, and the graph of the target domain introduces a lot of noise in the sampling process, such as misclassified labels. The existing node sampling method relies on these bounding boxes without processing non-informational background pixels. It is difficult to establish an implicit probability model that can well adapt to the distribution of a specific class by aligning these noises across domains in each training batch, which leads to poor semantic alignment effect and makes the target detection model quickly fall into a temporary local optimal state in the process of random gradient optimization.

[0081] The present application supplements the missing semantics of the source domain and the target domain by processing non-informational noise, and uses contrastive learning to pull the initial distribution of the sampling nodes and the distribution after the node supplement. Then, the node group is optimized through a shared attention layer to optimize the node quality. As shown in Figure 4 , the method is also called a contrastive node supplement method (CNC).

[0082] Specifically, the specific steps of the contrastive node supplement method in the embodiment are as shown in Figure 5 , which include:

[0083] Step S410: Comparative learning is performed on the nodes in the first source domain node group and the first target domain node group by using the global node memory pool, to generate a supplementary node for missing semantics in the first source domain node group and the first target domain node group, and to obtain a supplemented source domain node group and a supplemented target domain node group;

[0084] The missing nodes are supplemented by using a global node memory pool (GMB), and specifically, a momentum update method is used to iteratively update the node features, so that the node set has less noise. The momentum update method has the following form:

[0085]

[0086] wherein, is a GMB module containing node features, b s / t is a node that has been refined sin(·) is a similarity function.

[0087] In order to realize the supplement of node relationships based on the global node memory pool, the features of the original nodes are reorganized. The DNC (Domain-guided Node Completion) module and the GMB module are used to generate corresponding supplementary nodes for missing semantics so as to realize the transformation from the first source domain node group to the supplemented source domain node group the first target domain node group to the supplemented target domain node group . The DNC module is a node mutual supplement module, which is used to supplement the nodes missing in the source domain relative to the target domain and the nodes missing in the target domain relative to the source domain according to the semantic categories.

[0088] Step S420: The supplemented source domain node group and the supplemented target domain node group are respectively input into a shared self-attention module, to obtain a second source domain node group and a second target domain node group.

[0089] In order to reorganize the features of the supplemented target domain node group the supplemented source domain node group a self-attention module is used to apply to the supplemented node group, to obtain a second source domain node group and a second target domain node group

[0090] Through the node supplement based on comparative learning, the graph representation modeling strategy is more efficient and can stably represent the domain-level semantic space.

[0091] Step S500: calculating a contrastive distribution loss based on the first source domain node group, the first target domain node group, the second source domain node group and the second target domain node group;

[0092] After the node supplement process, although the relationship between the supplement nodes and the original nodes is constructed, the low-order information of the first source domain node group and the first target domain node group is lost due to noise, which is very important for semantic perception tasks such as identification and detection; these noises can even become the main component of the node group covariance. In order to improve the quality of the sampled nodes and constrain the noise in the sampled nodes, the first source domain node group and the first target domain node group are mapped to an implicit feature space by using an implicit feature extraction module (IFE) to obtain a first source feature space and a first target feature space The second source domain node group and the second target domain node group are mapped to the implicit feature space to obtain a second source feature space and a second target feature space Then, the contrastive distribution loss is calculated according to the similarity between the first source feature space and the second source feature space and the similarity between the first target feature space and the second target feature space . The contrastive distribution loss is used to constrain (i.e., the second source domain node group and the second target domain node group ) to be close to (i.e., the first target domain node group and the first source domain node group ). Specifically, referring to Figure 4 , the implicit feature extraction module (IFE) is used to map to the implicit feature space respectively. The contrastive distribution loss is used to pull and The expression of the contrastive distribution loss is as follows:

[0093]

[0094] wherein, τ is a temperature coefficient, and sim(·) is a similarity function.

[0095] By adopting the contrastive learning node supplement method, the sampled noise will be greatly reduced, thereby continuously improving the sampling quality of the node.

[0096] Step S600: Extract the covariance matrices of the second source domain node group and the second target domain node group respectively, and obtain the variance matrix according to all the covariance matrices;

[0097] Since the generated nodes inevitably carry domain information, this affects their representation and class-invariant feature extraction, resulting in poor semantic alignment. Some studies show that feature covariance contains domain-specific styles, such as color. Therefore, the present application migrates this conclusion to the graph representation to solve the problem of poor semantic alignment.

[0098] Specifically, the node group with complete graph information can be regarded as a highly concentrated semantic feature map. First, in the first n training rounds, the covariance information of is collected. Then the covariance matrices of the second source domain node group and the second target domain node group are extracted respectively to obtain the source domain covariance matrix ) and the target domain covariance matrix ); the source domain covariance matrix and the target domain covariance matrix are spliced to obtain the spliced covariance matrix; the mean and variance of the spliced covariance matrix are calculated to obtain the variance matrix.

[0099] The variance matrix V ∈ R C×C is defined as: Wherein, the mean and the variance The calculation formula is:

[0100]

[0101]

[0102] Wherein, [:] represents the splicing operation, N represents the total training batch, Σ c (·) represents the operation of extracting the covariance matrix.

[0103] After the above operation, the variance matrix V is composed of the variance of each covariance element in each domain. And the covariance of the node group is used to decouple the domain features and the class-invariant features, and selectively removes the domain style elements. A better node group

[0104] By reconstructing the features and structural relationships of the sampled nodes based on the global node memory pool, and then decoupling the covariance of the node group to remove the domain-specific elements in the nodes and obtain the key invariant features of each class, more fine-grained node semantic alignment is achieved.

[0105] Step S700: Calculating the node covariance alignment loss based on the variance matrix, the second source domain node group, and the second target domain node group;

[0106] Assume that the variance matrix V represents the sensitivity of the corresponding covariance to domain shift, which means that the covariance elements with high variance values ​​contain domain-specific styles. To identify these elements, we first simply apply the k-means clustering method to cluster the upper triangular elements of the variance matrix V. The k clusters are divided into two groups, G low ={c1,…,c m} is composed of low variance elements, G high ={c m+1 ,…,C k} is composed of high variance elements. Then the second source domain node group is spliced and the second target domain node group And extract the covariance matrix to obtain the spliced ​​covariance matrix Then according to the upper triangular matrix And the covariance matrix after splicing, calculate the node covariance alignment loss To remove domain style specific elements. The loss function is expressed as:

[0107]

[0108] in, represents the upper triangular matrix of color,

[0109]

[0110] Step S800: converting the second source domain node group and the second target domain node group into a graph structure and performing graph optimization to obtain a source domain graph node set and a source domain graph structure and a target domain graph node set and a target domain graph structure;

[0111] Due to the pixel feature representation in non-Euclidean space, abnormal nodes are easily generated in the graph construction process. Abnormal nodes are composed of meaningless nodes from ambiguous foreground semantics and out-of-distribution nodes sampled from clear foreground pixels in the node generation process. First, due to sampling to meaningless semantics, spatial sampling methods will generate some abnormal nodes in the real bounding box, such as ambiguous pixels of objects under extreme weather or mutual occlusion. Second, most graph construction strategies are based on the assumption that training and testing data are sampled from the same graph distribution. Unlike Euclidean data, the domain adaptation target detection problem involves a prediction task for each node. Nodes are connected to each other through the graph structure, and the node refinement process will introduce non-identical distribution characteristics even in the same environment. For example, abnormal nodes are usually sampled from pixels far from the center of the object, overlapping objects, or ambiguous objects. These abnormal nodes will confuse our target detection model and cause misclassification. This problem is particularly serious when the model generates false pseudolabels in the target domain, which will lead to incorrect semantic node sampling in the target graph.

[0112] Many current works model the corresponding domain graph through real bounding boxes, which will inevitably generate various abnormal nodes containing strange semantics due to the characteristics of node sampling and graph generation process. Aligning abnormal nodes with normal nodes will significantly weaken the ability of the target detection model to represent object class knowledge. This problem will cause the domain adaptation model to take a large step in the wrong gradient direction or overfitting phenomenon. Because each node stores multiple neighboring node features and graph structure knowledge, traditional distribution generalization methods are hindered from optimizing abnormal nodes. Identifying these abnormal nodes (or out-of-distribution samples) in non-Euclidean data is inherently more difficult.

[0113] Therefore, the present application adopts a precise graph optimization method (PGO) based on knowledge sharing and distribution generalization technology to convert nodes into high-quality graph representations. First, a designed global node enhancement module (GNE) is used to obtain the generated graph and optimization. Since precise graph optimization requires the source graph and the target graph to fully understand the domain knowledge of each other, message passing between graphs is crucial to improve the model's ability to perceive abnormal semantic nodes. Therefore, a global node enhancement module (GNE) is used to achieve efficient message passing, promote cross-graph information propagation, and improve the model's ability to perceive abnormal semantic nodes, obtaining the source domain graph node set Source domain graph structure ε s And the target domain graph node set Target domain graph structure ε t .

[0114] The global node enhancement module is represented as:

[0115]

[0116] wherein, and LN is layer normalization for the global-aware graph node set, is a learnable parameter.

[0117] In one example, a source domain graph node set and a target domain graph node set Afterwards, in order to enhance the semantic information of the graph, a node classification loss Specifically, a first node classification loss value of each graph node in the source domain graph node set is calculated; a second node classification loss value of each graph node in the target domain graph node set is calculated; all the first node classification loss values and all the second node classification loss values are accumulated to obtain a node classification loss value The specific expression is:

[0118]

[0119] wherein, F cls is a classifier, Y i is a ground truth label of the source domain node and a pseudo label (derived from a pseudo score graph ) of the target domain node, and n represents a training batch.

[0120] The node classification loss value can be used as a loss value to optimize the pre-trained target detection model. Through this message passing mechanism, each node feature is enhanced according to the knowledge of other nodes, thereby facilitating accurate graph optimization.

[0121] After obtaining a source domain graph structure ε s and a target domain graph structure ε t , a similarity matrix (also referred to as a semantic-aware node affinity matrix: SNA) is obtained according to the semantic relationship based on the source domain graph node set and the target domain graph node set; a structure loss value is calculated according to the similarity matrix the source domain graph structure ε s and the target domain graph structure ε t , and the structure loss value is used as a loss value to optimize the pre-trained target detection model. Specifically, a semantic-aware similarity matrix is learned using the class-level semantics with inherent relationships, and then is used to calculate a structure loss function The specific expression is:

[0122]

[0123] Step S900: representing the distribution shift of the graph by using the environment variable, calculating the abnormal node filtering loss according to the source domain graph node set, the source domain graph structure, the target domain graph node set and the target domain graph structure;

[0124] Suppose that the graph structure is generated based on the node feature V and the environment variable e, and the generation process is modeled to capture the probability of the training distribution shift:

[0125]

[0126] Where y represents the label of the node, p(e) and are dependent distributions. Then, the parameter θ is used to parameterize and The generation model is defined as a mixture of two parts. Where the environment variable e∈{0,1} represents whether the graph is in the distribution (ID, In the Distribution) e=1, or out of the distribution (OOD, Out of Distribution) e=0, which is formalized as follows:

[0127]

[0128]

[0129] Where, are observable, and then the variational inference is used to make the perception model approximate

[0130]

[0131] Finally, all the optimization objectives can be written as:

[0132]

[0133] By using variational inference to infer the latent variable of the current environment, the abnormal nodes (ASF) are filtered out, and a high-quality graph is obtained.

[0134] By modeling a generation process and introducing an environment latent variable as an indicator variable to describe the distribution shift of the graph, the proportion of strange semantic nodes in the generated graph can be reduced, and the quality of the constructed graph can be further improved.

[0135] Step S1000: optimizing the pre-trained target detection model according to the comparative distribution loss, the node covariance alignment loss and the abnormal node filtering loss, to obtain a trained target detection model for traffic target detection and recognition.

[0136] The overall loss function of the domain adaptation stage is:

[0137]

[0138] wherein, is the loss of global alignment, is the node alignment loss, is the classification and regression loss of the training source domain; is the abnormal node filtering loss; is the node covariance alignment loss; is the contrast distribution loss; λ 1 / 2 / 3 is set to 1 in the specific training process to control the amplitude of domain adaptation. It should be noted that obtaining the loss of global alignment, the node alignment loss, and the classification and regression loss of the training source domain are general technical means for training the FCOS model, and will not be described here.

[0139] The parameters in the pre-trained target detection model are updated using optimization algorithms such as gradient descent to minimize the unsupervised loss function on the target domain The above steps of the unsupervised adaptation stage are iterated until the target detection model is adapted to the target domain, and the trained target detection model for traffic target detection and recognition is obtained. Generally, the validation set on the target domain can be used to select the optimal model.

[0140] In summary, by introducing the idea of graph representation, through the intensive node refinement module (INR) and the precise graph optimization module (PGO), the embodiment can well cope with problems such as light changes, weather changes, and target occlusions in complex scenes, and can well apply to traffic target detection in real complex traffic scenes. It has good robustness and generalization, can achieve high precision and recall rate for traffic targets of different scales. It is suitable for perception layer in autonomous driving. It provides correct driving decisions for autonomous driving systems, greatly improving their safety and functionality.

[0141] In one example, in the face of other more complex scenes such as streets, a more powerful deep backbone network (such as Swin-T) can be replaced to achieve better recognition performance.

[0142] In one example, in the face of simple scenes (such as communities), a lighter backbone network (such as Mobilenet v3) can be replaced to obtain faster inference speed and higher edge computing efficiency.

[0143] The embodiment of the present application further provides a domain adaptation target detection method for a complex traffic scene. First, the traffic scene target detection model is trained by using the traffic scene target detection model training method based on the graph representation domain adaptation, and then the traffic image is input into the trained target detection model to perform target detection and obtain a target detection result.

[0144] Based on the above embodiment, the present application further provides an intelligent terminal, and a principle block diagram thereof can be shown in the accompanying drawings. Figure 6 The intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. The processor of the intelligent terminal is used to provide calculation and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a traffic scene target detection model training program based on the graph representation domain adaptation. The internal memory provides an environment for the operating system and the traffic scene target detection model training program based on the graph representation domain adaptation in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with external terminals through network connection. The traffic scene target detection model training program based on the graph representation domain adaptation is executed by the processor to implement the steps of any one of the traffic scene target detection model training methods based on the graph representation domain adaptation. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0145] Those skilled in the art can understand that Figure 6 The principle block diagram shown in the accompanying drawings is only a block diagram of part of the structure related to the present application, and does not constitute a limitation on the intelligent terminal to which the present application is applied. The specific intelligent terminal can include more or fewer components than those shown in the drawings, or combine certain components, or have a different component arrangement.

[0146] The embodiment of the present application further provides a computer readable storage medium. The computer readable storage medium stores a traffic scene target detection model training program based on the graph representation domain adaptation. The traffic scene target detection model training program based on the graph representation domain adaptation is executed by the processor to implement the steps of any one of the traffic scene target detection model training methods based on the graph representation domain adaptation provided by the embodiment of the present application.

[0147] It should be understood that the sequence numbers of the steps in the above embodiments do not mean the execution order, and the execution order of the processes should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example for description, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for convenient distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0149] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0150] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or in combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different ways to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0151] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented by other ways. For example, the above-described apparatus / terminal device embodiments are only schematic, and for example, the division of the above modules or units is only a logical function division, and there can be another division way in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0152] The above integrated modules / units, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the above-mentioned computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be realized. The above-mentioned computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The above-mentioned computer readable medium can include any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the above-mentioned computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0153] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand; it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not deviate from the spirit and scope of the corresponding technical solutions, and should be included in the protection scope of the present application.

Claims

1. The object detection model training method based on graph representation domain adaptation is characterized by: include: Pre-training the target detection model based on the dataset of the source domain; Input the source domain image and target domain image into the pre-trained object detection model respectively to obtain the source domain feature map and target domain feature map; Converting the source domain feature graph and the target domain feature graph into node groups based on graph representation, respectively, to obtain a first source domain node group and a first target domain node group; Based on node comparative learning, nodes with missing semantics are supplemented in the first source domain node group and the first target domain node group to obtain a second source domain node group and a second target domain node group; Calculating a comparative distribution loss based on the first source domain node group, the first target domain node group, the second source domain node group, and the second target domain node group; Extracting covariance matrices of the second source domain node group and the second target domain node group respectively, and obtaining a variance matrix based on all covariance matrices; Calculating a node covariance alignment loss according to the variance matrix, the second source domain node group, and the second target domain node group; Converting the second source domain node group and the second target domain node group into a graph structure and performing graph optimization to obtain a source domain graph node set and a source domain graph structure and a target domain graph node set and a target domain graph structure; Using environmental variables to characterize the distribution offset of the graph, and calculating the abnormal node filtering loss according to the source domain graph node set, the source domain graph structure, the target domain graph node set, and the target domain graph structure; The pre-trained target detection model is optimized according to the contrast distribution loss, the node covariance alignment loss, and the abnormal node filtering loss to obtain a trained target detection model for traffic target detection and recognition.

2. The object detection model training method based on graph representation domain adaptation according to claim 1, characterized in that The extracting covariance matrices of the second source domain node group and the second target domain node group respectively, and obtaining a variance matrix according to all the covariance matrices, includes: Extracting the covariance matrices of the second source domain node group and the second target domain node group respectively to obtain a source domain covariance matrix and a target domain covariance matrix; Concatenating the source domain covariance matrix and the target domain covariance matrix to obtain a concatenated covariance matrix; The mean and variance of the spliced ​​covariance matrix are calculated to obtain the variance matrix.

3. The object detection model training method based on graph representation domain adaptation according to claim 1, characterized in that Calculating a node covariance alignment loss according to the variance matrix, the second source domain node group, and the second target domain node group includes: Clustering the upper triangular elements of the variance matrix to obtain an upper triangular matrix; Splicing the second source domain node group and the second target domain node group to obtain a spliced ​​node group; Extracting the covariance matrix of the spliced ​​node group to obtain a spliced ​​covariance matrix; The node covariance alignment loss is calculated according to the upper triangular matrix and the concatenated covariance matrix.

4. The object detection model training method based on graph representation domain adaptation according to claim 1, characterized in that The node-based comparative learning, supplementing nodes for missing semantics in the first source domain node group and the first target domain node group to obtain a second source domain node group and a second target domain node group, includes: Using a global node memory pool to perform comparative learning on nodes of the first source domain node group and the first target domain node group, generating supplementary nodes for missing semantics in the first source domain node group and the first target domain node group, and obtaining supplemented source domain node group and target domain node group; The supplemented source domain node group and target domain node group are respectively input into the shared self-attention module to obtain the second source domain node group and the second target domain node group.

5. The object detection model training method based on graph representation domain adaptation according to claim 4, characterized in that The calculating the comparative distribution loss based on the first source domain node group, the first target domain node group, the second source domain node group, and the second target domain node group includes: Mapping the first source domain node group and the first target domain node group to an implicit feature space to obtain a first source feature space and a first target feature space; Mapping the second source domain node group and the second target domain node group to an implicit feature space to obtain a second source feature space and a second target feature space; The contrast distribution loss is obtained according to the similarity between the first source feature space and the second source feature space and the similarity between the first target feature space and the second target feature space.

6. The object detection model training method based on graph representation domain adaptation according to claim 1, characterized in that After obtaining the source domain graph node set and the target domain graph node set, it also includes: Calculating a first node classification loss value for each graph node in the source domain graph node set; Calculating a second node classification loss value for each graph node in the target domain graph node set; Accumulate all of the first node classification loss values ​​and all of the second node classification loss values ​​to obtain a node classification loss value for optimizing the pre-trained object detection model.

7. The object detection model training method based on graph representation domain adaptation according to claim 1, characterized in that After obtaining the source domain graph structure and the target domain graph structure, it also includes: Based on the source domain graph node set and the target domain graph node set, obtaining a similarity matrix according to semantic relationships; A structural loss value is calculated according to the similarity matrix, the source domain graph structure, and the target domain graph structure to optimize the pre-trained object detection model.

8. A domain-adaptive object detection method, characterized in that: The method comprises: Train the target detection model using the target detection model training method based on graph representation domain adaptation according to any one of claims 1 to 7 to obtain a trained target detection model; The traffic image is input into the trained target detection model to perform target detection and obtain a target detection result.

9. An intelligent terminal, characterized in that: The invention comprises a memory, a processor, and a target detection model training program based on graph representation domain adaptation stored in the memory and executable on the processor. When the target detection model training program based on graph representation domain adaptation is executed by the processor, the steps of the target detection model training method based on graph representation domain adaptation as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a target detection model training program based on graph representation domain adaptation. When the target detection model training program based on graph representation domain adaptation is executed by the processor, the steps of the target detection model training method based on graph representation domain adaptation as described in any one of claims 1 to 7 are implemented.

Images