Target recognition method, target recognition device and computer storage medium
Through the dual-flow object detection model and multi-scale spatiotemporal attention fusion network, combining the local weights of the location and key points, the problem of missing sequence dynamic features in the existing technology is solved, and the accuracy of target recognition is improved. robustness.
Patent Information
- Application Number
- CN202411177400.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-26
AI Technical Summary
In the prior art, only key point or site features are used for target recognition, resulting in the overall dynamic features of the sequence, poor model robustness, and easy to be affected by difficult samples.
The dual-stream object detection model is adopted to predict the local weights of the network and key points through the local weights of the site and the local weights of the network. Combined with the short-time space-time attention network and the long-time space-time attention network, the characteristics of the location and key points are extracted. , and identify the characteristics of multi-scale space-time attention fusion network fusion.
It improves the accuracy and robustness of target recognition, can adaptively capture significant local areas in the target, extract better target features, and enhances attention to the spatial and temporal information of the entire sequence.
Smart Images

Figure CN118691910B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a target recognition method, a target recognition device, and a computer storage medium. Background Art
[0002] In the existing technology, only key points or part features are used for target recognition, and more detailed features are extracted by analyzing the parts. However, there will be a certain lack of overall dynamic characteristics of the sequence. Paying too much or too little attention to the local part will make it more susceptible to the influence of difficult samples, and the robustness of the model will be poor. Summary of the invention
[0003] In order to solve the above technical problems, the present application proposes a target recognition method, a target recognition device and a computer storage medium.
[0004] To solve the above technical problems, the present application proposes a target recognition method, which includes: based on the target data to be identified, obtaining a part contour map sequence and a key point map sequence; inputting the part contour map sequence into the part local weight prediction network of the dual-stream target detection model to extract the part local weights; inputting the key point map sequence into the key point map local weight prediction network of the dual-stream target detection model to extract the key point local weights; after processing the part contour map sequence using the part local weights, the part features are extracted through the first feature extraction network of the dual-stream target detection model; after processing the key point map sequence using the key point local weights, the key point features are extracted through the second feature extraction network of the dual-stream target detection model; through the classification network of the dual-stream target detection model, the classification features fused from the part features and the key point features are identified to obtain the target recognition result of the target data to be identified.
[0005] Among them, the key point image sequence is a key point heat map sequence; the target recognition method also includes: based on the target data to be identified, obtaining a key point set of each target image; based on each key point in the key point set and its adjacent pixel points, generating a key hotspot; according to the key hotspots of each target image, generating a key point heat map.
[0006] Among them, the part contour map sequence is input into the part local weight prediction network of the dual-stream target detection model to extract the part local weight, including: weighted averaging the part contour map sequence through the part local weight prediction network to obtain a predicted part contour map; feature extraction and dynamic generation of local weights of the predicted part contour map through the part local weight prediction network to obtain the dynamic weight of each part area in the predicted part contour map to constitute the part local weight.
[0007] Among them, the first feature extraction network includes a short-term spatiotemporal attention network, and the part feature is a short-term part feature output by the short-term spatiotemporal attention network; the second feature extraction network includes a long-term spatiotemporal attention network, and the key point feature is a long-term key point feature output by the long-term spatiotemporal attention network.
[0008] Among them, the target recognition method also includes: inputting the short-time part features and the long-time key point features into a multi-scale spatiotemporal attention fusion network respectively, extracting the part spatiotemporal features of the short-time part features, the key point spatiotemporal features of the long-time key point features, and the fused spatiotemporal features of the short-time part features and the long-time key point features; inputting the combined features obtained by merging the part spatiotemporal features, the key point spatiotemporal features and the fused spatiotemporal features into the attention module of the multi-scale spatiotemporal attention fusion network to obtain weighted spatiotemporal features; fusing the combined features with the weighted spatiotemporal features to obtain the classification features.
[0009] Among them, the target recognition method also includes: obtaining labeled target data and unlabeled target data to be trained; using the labeled target data to train the target recognition model to obtain initial weight parameters; according to the initial weight parameters, obtaining a teacher model and a student model; using the labeled target data to train the teacher model to obtain a supervised loss; using the unlabeled target data to train the teacher model and the student model to obtain an unsupervised loss; using the supervised loss and the unsupervised loss to train the teacher model to obtain the dual-stream target detection model.
[0010] Among them, the method of using the supervised loss and the unsupervised loss to train the teacher model to obtain the dual-stream target detection model includes: using the supervised loss and the unsupervised loss to train the teacher model to obtain first model parameters; using the supervised loss and the unsupervised loss to train the student model to obtain second model parameters; using the first model parameters and the second model parameters to generate model parameters of the dual-stream target detection model.
[0011] Among them, using the unlabeled target data to train the teacher model and the student model to obtain the unsupervised loss includes: obtaining a first target sequence and a second target sequence of the unlabeled target data; performing weak data enhancement on the first target sequence to generate a third target sequence; performing strong data enhancement on the second target sequence to generate a fourth target sequence; inputting the third target sequence into the teacher model to extract a first target feature; inputting the fourth target sequence into the student model to extract a second target feature; and obtaining the unsupervised loss based on the first target feature and the second target feature.
[0012] In order to solve the above technical problems, the present application proposes a target identification device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above target identification method.
[0013] In order to solve the above technical problems, the present application proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above target identification method.
[0014] Compared with the prior art, the beneficial effects of the present application are as follows: the present application proposes a target recognition method, a target recognition device and a computer storage medium, wherein the target recognition device obtains a part contour map sequence and a key point map sequence based on the target data to be recognized; the part contour map sequence is input into the part local weight prediction network of the dual-stream target detection model to extract the part local weight; the key point map sequence is input into the key point map local weight prediction network of the dual-stream target detection model to extract the key point local weight; after processing the part contour map sequence using the part local weight, the part features are extracted through the first feature extraction network of the dual-stream target detection model; after processing the key point map sequence using the key point local weight, the key point features are extracted through the second feature extraction network of the dual-stream target detection model; the classification features fused with the part features and the key point features are identified through the classification network of the dual-stream target detection model to obtain the target recognition result of the target data to be recognized. Through the above method, by processing local weights, the significant local areas in the target input can be adaptively captured to help extract better target features, thereby improving the accuracy of target recognition. At the same time, in addition to using the part map, the key point heat map is also used. Not only does the local weights focus on the fine features of each part, but also the spatial and temporal information of the entire sequence is extracted, which not only better completes the task of target recognition but also increases robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:
[0016] Figure 1 is a flowchart of a first embodiment of a target recognition method provided by the present application;
[0017] Figure 2 It is a schematic diagram of a dual-stream target detection model provided by this application;
[0018] Figure 3 It is a schematic diagram of the local weight prediction network provided by this application;
[0019] Figure 4 is a schematic diagram of the short-term spatiotemporal attention module provided by this application;
[0020] Figure 5 is a schematic diagram of the long-term spatiotemporal attention module provided by this application;
[0021] Figure 6 It is a schematic diagram of feature fusion provided by this application;
[0022] Figure 7 is a flow chart of a second embodiment of the target recognition method provided by the present application;
[0023] Figure 8 is a flowchart of a third embodiment of the target recognition method provided by the present application;
[0024] Fig. 9 is a schematic diagram of an embodiment of dual-stream target detection model training provided by the present application;
[0025] Fig.10 is a schematic diagram of another embodiment of dual-stream target detection model training provided by the present application;
[0026] Fig.11 It is a structural schematic diagram of an embodiment of a target recognition device provided by the present application;
[0027] Fig.12 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION
[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0029] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0030] Please refer to Figure 1 and Figure 2 , Figure 1 is a flowchart of the first embodiment of the target recognition method provided by the present application, Figure 2 It is a schematic diagram of the dual-stream target detection model provided in this application.
[0031] The target recognition method of the present application is applied to a target recognition device, wherein the target recognition device of the present application can be a server, a terminal device, or a system in which a server and a terminal device cooperate with each other. Accordingly, the various parts included in the target recognition device, such as various units, subunits, modules, and submodules, can all be set in the server, can all be set in the terminal device, or can be set in the server and the terminal device respectively.
[0032] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here.
[0033] like Figure 1 As shown, the specific steps are as follows:
[0034] Step S11: Based on the target data to be identified, a part contour map sequence and a key point map sequence are obtained.
[0035] Among them, the target recognition method in the embodiment of the present application is an image recognition method, and the image can be a single-frame image or a continuous or discontinuous multi-frame image.
[0036] Among them, the target data to be identified is an image containing the target object and its posture information. The part contour map is a contour map obtained by analyzing the various parts of the target object in the image. It can be identified using Parsing R-CNN (regional convolutional neural network algorithm based on contextual features), QANet (machine reading comprehension model), etc.
[0037] Key points are the key points of the target object extracted from the image, which can be identified by methods such as HRNet (2D pose estimation model) and Openpose (pose recognition).
[0038] Taking the target object as a robot or an animal as an example, the parts are the head, torso, upper arm, thigh, etc. of the robot or the animal, and the key points are the key points of each part, namely the head key point, torso key point, upper arm key point, thigh key point, etc.
[0039] In one embodiment of the present application, the key point image sequence is a key point heat map sequence. The target recognition device obtains a key point set of each target image based on the target data to be identified; generates key hot spots based on each key point in the key point set and its adjacent pixel points; and generates a key point heat map according to the key hot spots of each target image.
[0040] Specifically, the target key point heat map first extracts the key point information from the target object, and then takes the pixels around each key point as a supplement. The key point heat map is derived from the key point sequence, so it has less spatial information, but it can better represent the dynamic changes of the entire sequence.
[0041] Step S12: Input the part contour map sequence into the part local weight prediction network of the dual-stream target detection model to extract the part local weight.
[0042] In target recognition, local information determines the degree of recognition of target features. Therefore, in the embodiment of the present application, local weights are added to weight the local information of the local position. Figure 2 As shown, the target recognition device inputs the part contour map sequence into the part local weight prediction network of the dual-stream target detection model to extract the part local weight.
[0043] Please continue to see Figure 3 , Figure 3It is a schematic diagram of the local weight prediction network provided in the present application. The target recognition device performs weighted averaging of the part contour map sequence through the part local weight prediction network to obtain a predicted part contour map; the predicted part contour map is subjected to feature extraction and local weight dynamic generation through the part local weight prediction network to obtain the dynamic weight of each part area in the predicted part contour map to constitute the part local weight.
[0044] Among them, the local weight represents the importance of each local area in the image, and the local weight is applicable to each frame image in the part contour map sequence.
[0045] Specifically, the part contour map divides the target object into k areas, each of the k areas has a dynamic weight, and the target recognition device composes a part local weight according to the dynamic weight of each part area.
[0046] Among them, the dynamic weights are learned through convolutional neural networks.
[0047] Step S13: Input the key point graph sequence into the key point graph local weight prediction network of the dual-stream object detection model to extract the key point local weights.
[0048] Specifically, the local weights of key points and parts are obtained in the same way, which will not be described here.
[0049] Step S14: After processing the part contour map sequence using the part local weights, the part features are extracted through the first feature extraction network of the dual-stream target detection model.
[0050] In one embodiment of the present application, the method of processing the part contour map sequence using the part local weight is: for each frame of the part contour map, the part local weight is multiplied by the pixel value of the image. In other embodiments of the present application, a dynamic coefficient can also be added to control the influence of the part weight.
[0051] Specifically, in a specific embodiment of the present application, the first feature extraction network is a short-term spatiotemporal attention network, and the part feature is a short-term part feature output by the short-term spatiotemporal attention network.
[0052] It should be noted that in the embodiment of the present application, the part contour map contains more spatial information and a small amount of dynamic information. In the process of extracting features, the target recognition device uses a network combining 2D convolution and 3D convolution to perform preliminary feature extraction on the part contour map. First, two 2D convolutions are used to extract spatial features, and then two layers of 3D convolution are used to simultaneously extract spatiotemporal features.
[0053] After the initial extraction of spatiotemporal features, the spatiotemporal features are input into the short-term spatiotemporal attention module, such as Figure 4 As shown, Figure 4It is a schematic diagram of the short-time spatiotemporal attention module provided by this application. Because the part contour map has stronger spatial information and there is strong short-time dynamic information between adjacent frames, the short-time spatiotemporal attention module is used for the part contour map to extract the spatial information and dynamic information of the target information between short-time distance frames, such as between three or five adjacent target frames.
[0054] Step S15: After processing the key point graph sequence using the key point local weights, the key point features are extracted through the second feature extraction network of the dual-stream target detection model.
[0055] The method of processing the key point graph sequence using the key point local weights is the same as that in step S14, and will not be repeated here.
[0056] In the embodiment of the present application, the weight of each local part is predicted based on the input. The weight of the part contour map represents the importance of each part of the target object, while the weight of the key point heat map represents the importance of different key points. These two weights can help the model extract more discernible local features.
[0057] In a specific embodiment of the present application, the second feature extraction network is a long-term spatiotemporal attention network, and the key point feature is a long-term key point feature output by the long-term spatiotemporal attention network.
[0058] The key point heat map contains more dynamic information and a small amount of spatial information. For the key point heat map, the target detection device directly uses multi-layer 3D convolution to extract spatiotemporal features.
[0059] After the initial extraction of spatiotemporal features, the spatiotemporal features are fed into the long-term spatiotemporal attention module, e.g. Figure 5 As shown, Figure 5 It is a schematic diagram of the long-term spatiotemporal attention module provided by this application. The key point heat map is derived from the key point sequence, so it has only less spatial information, but it can better characterize the dynamic changes of the entire sequence. Therefore, using long-term spatiotemporal attention for the key point heat map can better focus on the spatial information and dynamic information of the target object, and the specific structure.
[0060] Furthermore, this application proposes an embodiment for obtaining classification features. For details, please refer to Figure 6-Figure 7 , Figure 6 is a schematic diagram of feature fusion provided by this application, Figure 7 It is a flowchart of the second embodiment of the target recognition method provided by the present application.
[0061] like Figure 7 As shown, the specific steps are as follows:
[0062] Step S21: Input the short-time part features and the long-time key point features into the multi-scale spatiotemporal attention fusion network respectively, extract the part spatiotemporal features of the short-time part features, the key point spatiotemporal features of the long-time key point features, and the fused spatiotemporal features of the short-time part features and the long-time key point features.
[0063] After the target recognition device obtains the spatiotemporal information of the two branches, it fuses them through multi-scale spatiotemporal attention and adds the features of the two branches to obtain another richer feature.
[0064] Step S22: The combined features obtained by merging the part spatiotemporal features, the key point spatiotemporal features and the fused spatiotemporal features are input into the attention module of the multi-scale spatiotemporal attention fusion network to obtain weighted spatiotemporal features.
[0065] like Figure 6 As shown, the part spatiotemporal features, key point spatiotemporal features and the fused spatiotemporal features obtained in step S21 are merged again to obtain the merged features, which are input into the attention module of the multi-scale spatiotemporal attention fusion network to obtain the weighted spatiotemporal features.
[0066] Step S23: Fusing the combined features and the weighted spatiotemporal features to obtain classification features.
[0067] Specifically, the target recognition feature fuses the combined feature and the weighted spatiotemporal feature to obtain the classification feature. The feature fusion method can be any feature fusion method, which is not specifically limited in this application.
[0068] Step S16: Using the classification network of the dual-stream target detection model, the classification features fused with the part features and the key point features are identified to obtain the target recognition result of the target data to be identified.
[0069] Among them, the classification features are obtained through steps S21 to S23.
[0070] The target recognition method proposed in this application can better extract and fuse the spatial and temporal features of the target object. The input of this application is the part contour map and the key point heat map. These two inputs contain certain temporal and spatial features and emphasize the influence of different local areas.
[0071] In the embodiment of the present application, different basic feature extraction modules and spatiotemporal feature extraction modules are designed according to the different spatial information and temporal information contained in different inputs, so as to ensure that both inputs can extract the corresponding spatial information and temporal information. Finally, multi-scale spatiotemporal fusion features are used to deeply fuse features of different scales to obtain more discriminative features for target recognition.
[0072] Furthermore, the present application proposes an embodiment for obtaining a dual-stream target detection model. For details, see Figure 8-Figure 10 , Figure 8 is a flow chart of the third embodiment of the target recognition method provided by the present application, Fig. 9 is a schematic diagram of an embodiment of dual-stream target detection model training provided by the present application, Fig.10 It is a schematic diagram of another embodiment of dual-stream target detection model training provided by the present application.
[0073] like Figure 8 As shown, the specific steps are as follows:
[0074] Step S31: Obtain labeled target data and unlabeled target data to be trained.
[0075] Specifically, Fig. 9 As shown, the data set includes a data set with labeled target data and a data set with unlabeled target data.
[0076] Step S32: Use the labeled target data to train the target recognition model to obtain initial weight parameters.
[0077] Specifically, the target recognition device first tries to use the labeled target data to train the target recognition model to obtain the initial weights of the preliminary dual-stream spatiotemporal feature aggregation network model.
[0078] Among them, the target recognition device uses labeled data to supervise the training of the dual-stream spatiotemporal feature aggregation network model, uses a mixture of labeled data and unlabeled data to train the dual-stream spatiotemporal feature aggregation network model, and uses labeled data to perform supervised fine-tuning on the dual-stream spatiotemporal feature aggregation network model.
[0079] Step S33: Obtain the teacher model and the student model according to the initial weight parameters.
[0080] Specifically, during the training process, the teacher model and the student model have the same model structure, and the initial values are the initial weight parameters obtained through supervised training.
[0081] In the embodiment of the present application, the model training process is divided into two parts: training using labeled target data and training using unlabeled target data.
[0082] Step S34: Use the labeled target data to train the teacher model to obtain the supervised loss.
[0083] Specifically, the target recognition device performs weak data enhancement on the labeled target data, inputs the labeled target data after weak data enhancement into the teacher model, and calculates the triplet loss (Tripletloss) and cross entropy loss (CEloss) according to the label information.
[0084] The weak data enhancement method for the labeled target data may be translation and / or flipping.
[0085] Among them, the method of calculating the triplet loss and the cross entropy loss according to the label information is to calculate according to the difference between the predicted label and the true label. The specific calculation of the triplet loss and the cross entropy loss is the same as the prior art and will not be repeated here.
[0086] Step S35: Use the unlabeled target data to train the teacher model and the student model to obtain the unsupervised loss.
[0087] Specifically, this application proposes a method to train a teacher model and a student model using unlabeled target data, and the specific method is as follows:
[0088] The target recognition device obtains a first target sequence and a second target sequence of unlabeled target data; performs weak data enhancement on the first target sequence to generate a third target sequence; performs strong data enhancement on the second target sequence to generate a fourth target sequence; inputs the third target sequence into the teacher model to extract the first target feature; inputs the fourth target sequence into the student model to extract the second target feature; and obtains an unsupervised loss based on the first target feature and the second target feature.
[0089] Specifically, for unlabeled data, the target recognition device selects two target subsequences with less overlap from a complete target sequence, performs weak data enhancement and strong data enhancement on the two target subsequences with less overlap respectively, and then inputs them into the student model and the teacher model, and calculates the contrast loss (InfoNCELoss) of the two target features obtained.
[0090] Among them, weak data enhancement includes but is not limited to translation and flipping, and strong data enhancement includes but is not limited to random cropping, pixel distortion, etc.
[0091] Step S36: Train the teacher model using supervised loss and unsupervised loss to obtain a dual-stream target detection model.
[0092] Specifically, the target recognition device uses supervised loss and unsupervised loss to train the teacher model to obtain first model parameters; uses supervised loss and unsupervised loss to train the student model to obtain second model parameters; uses the first model parameters and the second model parameters to generate model parameters of the dual-stream target detection model.
[0093] like Fig.10As shown, the target recognition device weightedly fuses the supervised loss and the unsupervised loss to obtain the final loss. In order to avoid the impact of special data on the model, in one embodiment of the present application, only the student model parameters are updated during the back propagation process, while the teacher model is updated using a smoother EMA method (exponential moving average). The exponential moving average refers to updating the parameters of the teacher network to the weighted sum of the parameters of the teacher network and the student network at the current moment. After the training is completed, the teacher model is used as the final model, and the labeled data is used for the final fine-tuning.
[0094] The semi-supervised hybrid training method, which alternately mixes labeled data and unlabeled data for training, can give full play to the role of labeled data and unlabeled data, so that a large number of unlabeled target sequences can be used for model training.
[0095] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0096] In order to realize the above target recognition method, this application also proposes a target recognition device, please refer to Fig.11 , Fig.11 It is a structural schematic diagram of an embodiment of a target recognition device provided by the present application.
[0097] The target recognition device 400 of this embodiment includes a processor 41 , a memory 42 , an input / output device 43 , and a bus 44 .
[0098] The processor 41 , the memory 42 , and the input / output device 43 are respectively connected to the bus 44 . The memory 42 stores program data, and the processor 41 is used to execute the program data to implement the target recognition method described in the above embodiment.
[0099] In the embodiment of the present application, the processor 41 may also be referred to as a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip having the ability to process signals. The processor 41 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 41 may also be any conventional processor, etc.
[0100] This application also provides a computer storage medium, please continue to refer to Fig.12 , Fig.12 It is a structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by a processor, it is used to implement the target recognition method of the above embodiment.
[0101] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
[0102] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A target recognition method, characterized in that: The target recognition method comprises: Based on the target data to be identified, a part contour map sequence and a key point map sequence are obtained; Inputting the part contour map sequence into the part local weight prediction network of the dual-stream target detection model to extract the part local weight; Inputting the key point graph sequence into the key point graph local weight prediction network of the dual-stream object detection model to extract the key point local weights; After processing the part contour map sequence using the part local weights, part features are extracted through a first feature extraction network of the dual-stream object detection model; After processing the key point graph sequence using the key point local weights, key point features are extracted through a second feature extraction network of the dual-stream object detection model; Identifying the classification features fused from the part features and the key point features through the classification network of the dual-stream target detection model to obtain a target recognition result for the target data to be identified; The first feature extraction network includes a short-term spatiotemporal attention network, and the part feature is a short-term part feature output by the short-term spatiotemporal attention network; The second feature extraction network includes a long-term spatiotemporal attention network, and the key point feature is a long-term key point feature output by the long-term spatiotemporal attention network; The target recognition method further includes: Inputting the short-term part features and the long-term key point features into a multi-scale spatiotemporal attention fusion network respectively, extracting the part spatiotemporal features of the short-term part features, the key point spatiotemporal features of the long-term key point features, and the fused spatiotemporal features of the short-term part features and the long-term key point features; Inputting the combined features obtained by combining the part spatiotemporal features, the key point spatiotemporal features and the fused spatiotemporal features into the attention module of the multi-scale spatiotemporal attention fusion network to obtain weighted spatiotemporal features; The combined feature and the weighted spatiotemporal feature are fused to obtain the classification feature.
2. The target recognition method according to claim 1, characterized in that: The key point map sequence is a key point heat map sequence; The target recognition method further includes: Based on the target data to be identified, obtaining a key point set of each target image; Generate a key hotspot based on each key point in the key point set and its adjacent pixels; Generate a key point heat map according to the key hot spots of each target image.
3. The target recognition method according to claim 1, characterized in that: The step of inputting the part contour map sequence into a part local weight prediction network of a dual-stream target detection model to extract the part local weight comprises: Performing weighted averaging on the sequence of part contour images through the part local weight prediction network to obtain a predicted part contour image; The predicted part contour map is subjected to feature extraction and local weight dynamic generation through the part local weight prediction network to obtain the dynamic weight of each part area in the predicted part contour map to form the part local weight.
4. The target recognition method according to claim 1, characterized in that: The target identification method further includes: Obtain labeled target data and unlabeled target data to be trained; Using the labeled target data to train a target recognition model to obtain initial weight parameters; According to the initial weight parameters, obtain the teacher model and student model; Training the teacher model using the labeled target data to obtain a supervised loss; Training the teacher model and the student model using the unlabeled target data to obtain an unsupervised loss; The teacher model is trained using the supervised loss and the unsupervised loss to obtain the dual-stream object detection model.
5. The target recognition method according to claim 4, characterized in that: The method of training the teacher model using the supervised loss and the unsupervised loss to obtain the dual-stream target detection model includes: Training the teacher model using the supervised loss and the unsupervised loss to obtain first model parameters; Training the student model using the supervised loss and the unsupervised loss to obtain second model parameters; The first model parameters and the second model parameters are used to generate model parameters of the dual-stream object detection model.
6. The target recognition method according to claim 4, characterized in that: The using the unlabeled target data to train the teacher model and the student model to obtain an unsupervised loss includes: Acquire a first target sequence and a second target sequence of the unlabeled target data; Performing weak data enhancement on the first target sequence to generate a third target sequence; Performing strong data enhancement on the second target sequence to generate a fourth target sequence; Inputting the third target sequence into the teacher model to extract the first target feature; Inputting the fourth target sequence into the student model to extract the second target feature; The unsupervised loss is obtained based on the first target feature and the second target feature.
7. A target recognition device, characterized in that: The target recognition device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the target recognition method according to any one of claims 1 to 6.
8. A computer storage medium, characterized in that The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the target recognition method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Image detection method, device and apparatus and storage medium
CN112241667A
Double-flow global-local action recognition method, system and equipment based on video input and storage medium
CN116311495A
