Target object recognition method and electronic device

Abstract

The application discloses a target object recognition method and an electronic device, and is used for enhancing the richness of training data and improving the recognition effect of a target in a night scene. The method comprises the following steps: acquiring a to-be-recognized image sequence containing a target object; inputting the to-be-recognized image sequence into a feature extraction network to output object features of the target object contained in the to-be-recognized image sequence; wherein a training set of the feature extraction network comprises a plurality of image sequence sets corresponding to a plurality of training objects respectively, the image sequence set comprises a plurality of image pairs, the image pair contains two images, one of which is obtained by converting the other, and the two images contained in the image pair are of different image types; and performing recognition on the object features of the target object contained in the to-be-recognized image sequence to determine a recognition result of the target object.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition, and in particular to a target object recognition method and electronic equipment. Background Art

[0002] With the advancement of deep learning technology, target recognition has also made great progress in recent years. However, target recognition, such as face recognition, has high requirements for the recognition image. Close-range recognition is required to ensure the accuracy of the recognition results. In addition, there are also high requirements for the size of the target in the image.

[0003] Although there are target recognition technologies that can circumvent the above shortcomings to a certain extent, the current target recognition technologies are highly dependent on texture features, have poor recognition effects in night scenes, and have limitations in application. Summary of the Invention

[0004] The present invention provides a target object recognition method and electronic equipment, which are used to enhance the richness of training data and improve the recognition effect of targets in night scenes by converting visible light images into infrared images or converting infrared images into visible light images.

[0005] In a first aspect, an embodiment of the present invention provides a target object recognition method, comprising:

[0006] Obtain an image sequence to be identified containing the target object;

[0007] Inputting the image sequence to be identified into a feature extraction network, and outputting object features of the target object contained in the image sequence to be identified; wherein a training set of the feature extraction network includes a set of image sequences corresponding to a plurality of training objects, the image sequences in the set of image sequences include a plurality of image pairs, the image pairs include two images, one of which is obtained by converting the other image, and the two images contained in the image pairs are of different image types;

[0008] Identify the object features of the target object contained in the to-be-identified image sequence and determine the identification result of the target object.

[0009] The present invention realizes mutual search between infrared and visible light by converting visible light images into infrared images or converting infrared images into visible light images during the training stage of the feature extraction network, thereby enhancing the richness of training data. Since it is in the training stage, it does not increase the time spent on target object recognition in the inference stage, and can improve the recognition effect of targets in night scenes without increasing time.

[0010] As an optional implementation manner, the image pair includes an original image and an image obtained by converting the original image; the image pair is determined by:

[0011] If it is determined that the original image is a visible light image, converting the visible light image into an infrared image, and determining an image pair based on the visible light image and the converted infrared image;

[0012] If it is determined that the original image is an infrared image, the infrared image is converted into a visible light image, and an image pair is determined based on the infrared image and the converted visible light image.

[0013] As an optional implementation,

[0014] If it is determined that the original image is a visible light image, converting the visible light image into an infrared image using a first conversion network; if it is determined that the original image is an infrared image, converting the infrared image into a visible light image using a second conversion network;

[0015] The first conversion network and the second conversion network are obtained by training based on a sample set, the sample set includes a plurality of sample pairs, and the training process of the first conversion network and the second conversion network includes:

[0016] If the image included in the sample pair is a visible light image, the sample pair is input into a first conversion network to output an infrared sample pair; the infrared sample pair is input into a second conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using reconstruction loss values ​​determined from the sample pair and the reconstructed sample pair;

[0017] If the image contained in the sample pair is an infrared image, the sample pair is input into the second conversion network to output a visible light sample pair; the visible light sample pair is input into the first conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using the reconstruction loss values ​​determined by the sample pair and the reconstructed sample pair.

[0018] As an optional implementation manner, the sample pair includes adjacent frame images; the optical flow represents inter-frame difference information of the adjacent frame images; and further includes:

[0019] If the image included in the sample pair is a visible light image, determining a first optical flow of the sample pair and a second optical flow of the infrared sample pair, wherein a first conversion network and a second conversion network are trained according to an inter-frame difference loss value determined by the first optical flow and the second optical flow;

[0020] If the image included in the sample pair is an infrared image, a third optical flow of the sample pair and a fourth optical flow of the visible light sample pair are determined, wherein the first conversion network and the second conversion network are trained according to the inter-frame difference loss value determined by the third optical flow and the fourth optical flow.

[0021] As an optional implementation, it also includes:

[0022] If the image included in the sample pair is a visible light image, training the first conversion network and the second conversion network according to the discriminant classification loss value determined by the infrared sample pair and the pre-acquired real infrared image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real visible light image;

[0023] If the image included in the sample pair is an infrared image, the first conversion network and the second conversion network are trained based on the discriminant classification loss value determined by the visible light sample pair and the pre-acquired real visible light image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real infrared image.

[0024] As an optional implementation, the feature extraction network includes a backbone network, a first extraction network, and a second extraction network;

[0025] Inputting the image sequence to be identified into a feature extraction network and outputting object features of the target object contained in the image sequence to be identified includes:

[0026] Inputting the image sequence to be recognized into the backbone network and outputting corresponding image semantic features;

[0027] Inputting the image semantic features into a first extraction network to output a first feature, and inputting the image semantic features into a second extraction network to output a second feature; wherein the first feature represents a sequence feature of an image sequence to be identified, and the second feature represents a gait feature of the target object;

[0028] An object feature of a target object included in the sequence of images to be identified is determined according to the first feature and the second feature.

[0029] As an optional implementation manner, determining the object feature of the target object contained in the sequence of images to be identified based on the first feature and the second feature includes:

[0030] Determine a spliced ​​feature obtained by splicing the first feature and the second feature as the object feature; or

[0031] A fusion feature obtained by adding the first feature and the second feature is determined as the object feature.

[0032] As an optional implementation, it further includes a sequence feature extraction network and a gait feature extraction network; the feature extraction network includes a backbone network, a first extraction network, and a second extraction network; the training process of the feature extraction network includes:

[0033] Input the image sequence set into the sequence feature extraction network to output the corresponding sequence feature set; input the image sequence set into the gait feature extraction network to output the corresponding gait feature set; input the image sequence set into the backbone network of the feature extraction network to output the image semantic feature set, and input the image semantic feature set into the first extraction network and the second extraction network to obtain the first feature set and the second feature set respectively;

[0034] Determine a first distillation loss value based on the sequence feature set and the first feature set, and determine a second distillation loss value based on the gait feature set and the second feature set;

[0035] The feature extraction network is trained according to the feature loss values ​​corresponding to the sequence feature set, the gait feature set, the first feature set, and the second feature set, as well as the first distillation loss value and the second distillation loss value.

[0036] As an optional implementation, the feature extraction network further includes an attribute classification network, and the attribute classification network is used to extract local features of the target object; the training process of the feature extraction network further includes:

[0037] Inputting the image semantic feature set into the attribute classification network, outputting a corresponding attribute classification feature set, and determining an attribute classification loss value based on the attribute classification feature set;

[0038] The feature extraction network is trained according to the attribute classification loss value.

[0039] In a second aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0040] Obtain an image sequence to be identified containing the target object;

[0041] Inputting the image sequence to be identified into a feature extraction network, and outputting object features of the target object contained in the image sequence to be identified; wherein a training set of the feature extraction network includes a set of image sequences corresponding to a plurality of training objects, the image sequences in the set of image sequences include a plurality of image pairs, the image pairs include two images, one of which is obtained by converting the other image, and the two images contained in the image pairs are of different image types;

[0042] Identify the object features of the target object contained in the to-be-identified image sequence and determine the identification result of the target object.

[0043] As an optional implementation manner, the image pair includes an original image and an image obtained by converting the original image; the processor is specifically configured to determine the image pair in the following manner:

[0044] If it is determined that the original image is a visible light image, converting the visible light image into an infrared image, and determining an image pair based on the visible light image and the converted infrared image;

[0045] If it is determined that the original image is an infrared image, the infrared image is converted into a visible light image, and an image pair is determined based on the infrared image and the converted visible light image.

[0046] As an optional implementation, the processor is specifically configured to execute:

[0047] If it is determined that the original image is a visible light image, converting the visible light image into an infrared image using a first conversion network; if it is determined that the original image is an infrared image, converting the infrared image into a visible light image using a second conversion network;

[0048] The first conversion network and the second conversion network are obtained by training based on a sample set, the sample set includes a plurality of sample pairs, and the training process of the first conversion network and the second conversion network includes:

[0049] If the image included in the sample pair is a visible light image, the sample pair is input into a first conversion network to output an infrared sample pair; the infrared sample pair is input into a second conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using reconstruction loss values ​​determined from the sample pair and the reconstructed sample pair;

[0050] If the image contained in the sample pair is an infrared image, the sample pair is input into the second conversion network to output a visible light sample pair; the visible light sample pair is input into the first conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using the reconstruction loss values ​​determined by the sample pair and the reconstructed sample pair.

[0051] As an optional implementation manner, the sample pair includes adjacent frame images; the optical flow represents inter-frame difference information of the adjacent frame images; and the processor is further configured to execute:

[0052] If the image included in the sample pair is a visible light image, determining a first optical flow of the sample pair and a second optical flow of the infrared sample pair, wherein a first conversion network and a second conversion network are trained according to an inter-frame difference loss value determined by the first optical flow and the second optical flow;

[0053] If the image included in the sample pair is an infrared image, a third optical flow of the sample pair and a fourth optical flow of the visible light sample pair are determined, wherein the first conversion network and the second conversion network are trained according to the inter-frame difference loss value determined by the third optical flow and the fourth optical flow.

[0054] As an optional implementation manner, the processor is further configured to execute:

[0055] If the image included in the sample pair is a visible light image, training the first conversion network and the second conversion network according to the discriminant classification loss value determined by the infrared sample pair and the pre-acquired real infrared image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real visible light image;

[0056] If the image included in the sample pair is an infrared image, the first conversion network and the second conversion network are trained based on the discriminant classification loss value determined by the visible light sample pair and the pre-acquired real visible light image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real infrared image.

[0057] As an optional implementation, the feature extraction network includes a backbone network, a first extraction network, and a second extraction network; the processor is specifically configured to execute:

[0058] Inputting the image sequence to be recognized into the backbone network and outputting corresponding image semantic features;

[0059] Inputting the image semantic features into a first extraction network to output a first feature, and inputting the image semantic features into a second extraction network to output a second feature; wherein the first feature represents a sequence feature of an image sequence to be identified, and the second feature represents a gait feature of the target object;

[0060] An object feature of a target object included in the sequence of images to be identified is determined according to the first feature and the second feature.

[0061] As an optional implementation, the processor is specifically configured to execute:

[0062] Determine a spliced ​​feature obtained by splicing the first feature and the second feature as the object feature; or

[0063] A fusion feature obtained by adding the first feature and the second feature is determined as the object feature.

[0064] As an optional embodiment, it further includes a sequence feature extraction network and a gait feature extraction network; the feature extraction network includes a backbone network, a first extraction network and a second extraction network; the processor is specifically configured to execute:

[0065] Input the image sequence set into the sequence feature extraction network to output the corresponding sequence feature set; input the image sequence set into the gait feature extraction network to output the corresponding gait feature set; input the image sequence set into the backbone network of the feature extraction network to output the image semantic feature set, and input the image semantic feature set into the first extraction network and the second extraction network to obtain the first feature set and the second feature set respectively;

[0066] Determine a first distillation loss value based on the sequence feature set and the first feature set, and determine a second distillation loss value based on the gait feature set and the second feature set;

[0067] The feature extraction network is trained according to the feature loss values ​​corresponding to the sequence feature set, the gait feature set, the first feature set, and the second feature set, as well as the first distillation loss value and the second distillation loss value.

[0068] As an optional implementation, the feature extraction network further includes an attribute classification network, and the attribute classification network is used to extract local features of the target object; the processor is further configured to execute:

[0069] Inputting the image semantic feature set into the attribute classification network, outputting a corresponding attribute classification feature set, and determining an attribute classification loss value based on the attribute classification feature set;

[0070] The feature extraction network is trained according to the attribute classification loss value.

[0071] In a third aspect, an embodiment of the present invention further provides a target object recognition device, comprising:

[0072] An image sequence acquisition unit is used to acquire an image sequence to be identified containing a target object;

[0073] an image feature extraction unit configured to input the to-be-identified image sequence into a feature extraction network and output object features of a target object contained in the to-be-identified image sequence; wherein a training set for the feature extraction network comprises a plurality of image sequence sets corresponding to respective training objects, an image sequence in the image sequence set comprising a plurality of image pairs, each of the image pairs comprising two images, one of which is obtained by converting the other image, and the two images contained in the image pairs are of different image types;

[0074] The recognition result determination unit is configured to identify the object features of the target object contained in the to-be-recognized image sequence and determine the recognition result of the target object.

[0075] As an optional implementation manner, the image pair includes an original image and an image obtained by converting the original image; and the image feature extraction unit is specifically configured to determine the image pair in the following manner:

[0076] If it is determined that the original image is a visible light image, converting the visible light image into an infrared image, and determining an image pair based on the visible light image and the converted infrared image;

[0077] If it is determined that the original image is an infrared image, the infrared image is converted into a visible light image, and an image pair is determined based on the infrared image and the converted visible light image.

[0078] As an optional implementation manner, the image feature extraction unit is specifically configured to:

[0079] If it is determined that the original image is a visible light image, converting the visible light image into an infrared image using a first conversion network; if it is determined that the original image is an infrared image, converting the infrared image into a visible light image using a second conversion network;

[0080] The first conversion network and the second conversion network are obtained by training based on a sample set, the sample set includes a plurality of sample pairs, and the training process of the first conversion network and the second conversion network includes:

[0081] If the image included in the sample pair is a visible light image, the sample pair is input into a first conversion network to output an infrared sample pair; the infrared sample pair is input into a second conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using reconstruction loss values ​​determined from the sample pair and the reconstructed sample pair;

[0082] If the image contained in the sample pair is an infrared image, the sample pair is input into the second conversion network to output a visible light sample pair; the visible light sample pair is input into the first conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using the reconstruction loss values ​​determined by the sample pair and the reconstructed sample pair.

[0083] As an optional implementation manner, the sample pair includes adjacent frame images; the optical flow represents inter-frame difference information of the adjacent frame images; and the image feature extraction unit is further configured to:

[0084] If the image included in the sample pair is a visible light image, determining a first optical flow of the sample pair and a second optical flow of the infrared sample pair, wherein a first conversion network and a second conversion network are trained according to an inter-frame difference loss value determined by the first optical flow and the second optical flow;

[0085] If the image included in the sample pair is an infrared image, a third optical flow of the sample pair and a fourth optical flow of the visible light sample pair are determined, wherein the first conversion network and the second conversion network are trained according to the inter-frame difference loss value determined by the third optical flow and the fourth optical flow.

[0086] As an optional implementation manner, the image feature extraction unit is further specifically configured to:

[0087] If the image included in the sample pair is a visible light image, training the first conversion network and the second conversion network according to the discriminant classification loss value determined by the infrared sample pair and the pre-acquired real infrared image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real visible light image;

[0088] If the image included in the sample pair is an infrared image, the first conversion network and the second conversion network are trained based on the discriminant classification loss value determined by the visible light sample pair and the pre-acquired real visible light image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real infrared image.

[0089] As an optional implementation, the feature extraction network includes a backbone network, a first extraction network, and a second extraction network; the image feature extraction unit is specifically configured to:

[0090] Inputting the image sequence to be recognized into the backbone network and outputting corresponding image semantic features;

[0091] Inputting the image semantic features into a first extraction network to output a first feature, and inputting the image semantic features into a second extraction network to output a second feature; wherein the first feature represents a sequence feature of an image sequence to be identified, and the second feature represents a gait feature of the target object;

[0092] An object feature of a target object included in the sequence of images to be identified is determined according to the first feature and the second feature.

[0093] As an optional implementation manner, the image feature extraction unit is specifically configured to:

[0094] Determine a spliced ​​feature obtained by splicing the first feature and the second feature as the object feature; or

[0095] A fusion feature obtained by adding the first feature and the second feature is determined as the object feature.

[0096] As an optional embodiment, it further includes a sequence feature extraction network and a gait feature extraction network; the feature extraction network includes a backbone network, a first extraction network and a second extraction network; the image feature extraction unit is specifically used to:

[0097] Input the image sequence set into the sequence feature extraction network to output the corresponding sequence feature set; input the image sequence set into the gait feature extraction network to output the corresponding gait feature set; input the image sequence set into the backbone network of the feature extraction network to output the image semantic feature set, and input the image semantic feature set into the first extraction network and the second extraction network to obtain the first feature set and the second feature set respectively;

[0098] Determine a first distillation loss value based on the sequence feature set and the first feature set, and determine a second distillation loss value based on the gait feature set and the second feature set;

[0099] The feature extraction network is trained according to the feature loss values ​​corresponding to the sequence feature set, the gait feature set, the first feature set, and the second feature set, as well as the first distillation loss value and the second distillation loss value.

[0100] As an optional implementation, the feature extraction network further includes an attribute classification network, and the attribute classification network is used to extract local features of the target object; the image feature extraction unit is further used to:

[0101] Inputting the image semantic feature set into the attribute classification network, outputting a corresponding attribute classification feature set, and determining an attribute classification loss value based on the attribute classification feature set;

[0102] The feature extraction network is trained according to the attribute classification loss value.

[0103] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, it is used to implement the steps of the method described in the first aspect above.

[0104] These and other aspects of the present application will be more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0106] Figure 1 A flowchart of an implementation method of a target object identification method provided by an embodiment of the present invention;

[0107] Figure 2 A schematic diagram of a network architecture during the training process of a feature extraction network provided by an embodiment of the present invention;

[0108] Figure 3 A schematic diagram of the architecture of a feature extraction network inference stage provided by an embodiment of the present invention;

[0109] Figure 4 A schematic diagram of a training process of a first conversion network and a second conversion network provided by an embodiment of the present invention;

[0110] Figure 5 A schematic diagram of a training process of a first conversion network and a second conversion network provided by an embodiment of the present invention;

[0111] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention;

[0112] Figure 7 A schematic diagram of a target object recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0113] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0114] In embodiments of the present invention, the term "and / or" describes the association relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0115] The application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Persons skilled in the art will appreciate that as new application scenarios emerge, the technical solutions provided by the embodiments of the present invention will also be applicable to similar technical problems. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0116] In the technical solution of this application, the collection, dissemination, and use of image, video and other data comply with the requirements of relevant national laws and regulations.

[0117] With advances in deep learning technology, facial recognition and person tracking (ReID) technologies have made significant progress in recent years, contributing significantly to public security and crime detection. ReID, also known as pedestrian re-identification, pedestrian re-identification, or cross-camera tracking, uses cameras integrated with person tracking technology to reconstruct a person's movement trajectory. Subsequently, as long as the person appears in any camera capable of human recognition, they can be identified. By restoring trajectories across multiple cameras and comparing individuals, security measures become more comprehensive. However, facial recognition requires high image quality, close proximity, and a high target size. While ReID technology can mitigate these shortcomings to some extent, its strong reliance on texture features makes it less effective in infrared scenes. Since crimes often occur at night, this limits the application of facial recognition and ReID. Deep learning performance generally depends on the richness of training data. Large-scale data can train models with better generalization performance, but collecting large amounts of paired infrared and visible light data is extremely difficult.

[0118] This embodiment provides a method for identifying a target object. By converting a visible light image into an infrared image or converting an infrared image into a visible light image during the training phase of a feature extraction network, mutual search between infrared and visible light is achieved, thereby enhancing the richness of training data. Since the method is performed during the training phase, the time consumption for target object identification during the inference phase is not increased, and the recognition effect of the target in night scenes can be improved without increasing the time consumption.

[0119] like Figure 1 As shown, the implementation process of a target object recognition method provided by this embodiment is as follows:

[0120] Step 100: Obtain a sequence of images to be identified containing a target object;

[0121] In practice, the target objects of this embodiment include various types of objects such as people, faces, bodies, cars, animals, etc., and this embodiment does not impose too many restrictions on this.

[0122] In this embodiment, the sequence of images to be identified can be a sequence of video frames continuously captured of one or more target objects. Each video frame is equivalent to an image, and the images in the sequence are arranged in chronological order of acquisition time. In this embodiment, the sequence of images to be identified includes multiple images to be identified, each of which can contain one or more target objects. Different images to be identified can contain the same target object or different target objects.

[0123] Optionally, the acquisition device for collecting the image sequence to be identified includes, but is not limited to, a camera device, an infrared device, a monitoring device, etc. The present embodiment does not make excessive limitation on the acquisition device for collecting the image sequence to be identified.

[0124] Step 101: inputting the image sequence to be identified into a feature extraction network, and outputting object features of a target object contained in the image sequence to be identified;

[0125] The training set of the feature extraction network includes a plurality of image sequence sets respectively corresponding to training objects, the image sequence set includes a plurality of image pairs, and the image pair contains two images, one of which is obtained by converting the other image. The two images contained in the image pair are of different image types.

[0126] Optionally, one of the images contained in the image pair is an original image, and the other image is obtained by converting the original image. In the implementation, when the original image in the image pair is converted, the conversion process is mainly the conversion of the image type, and the conversion manner is determined according to the image type of the original image and the image type of the image to be converted. The present embodiment does not make excessive limitation on this.

[0127] The image type includes, but is not limited to, a multi-channel image type and a single-channel image type. The multi-channel image type includes, but is not limited to, an RGB image type and a CMYK image type, such as a visible light image type. The single-channel image type includes a grayscale image type, an infrared image type, and an X-ray image type.

[0128] Step 102: identifying the object features of the target object contained in the image sequence to be identified, and determining an identification result of the target object.

[0129] Optionally, the feature extraction network in the present embodiment is obtained by training based on a training set. During the training, the learning starts from the random initialization of the weights of the feature extraction network. The training set is prepared according to the dimensions of the training objects. The training set includes a plurality of input sets. Each input set is equivalent to a batch. Each input set includes K image sequence sets respectively corresponding to K training objects. Each training object (ID) corresponds to an image sequence set. Each image sequence set includes T image sequences. Each image sequence includes a plurality of continuous original images of the same training object. K≥1, T≥1, and K and T are integers.

[0130] It should be noted that this embodiment will perform image conversion on the original images contained in the image sequence, thereby expanding the data set of the image sequence, converting the original images in each image sequence, and treating the original image and the converted image as an image pair, so that the image pairs contained in the image sequence can involve both visible light and infrared characteristic images.

[0131] In the implementation, the training set D is defined as {P1, P2, ..., P N}, where P i represents the set of image sequences with training object (ID) i, N represents the total number of training objects (ID) in the training set; 1≤i≤N, i is an integer;

[0132] M i Indicates P i The number of image sequences in , S ij Indicates P i The j-th image sequence in M, 1≤j≤M i , j is an integer, M i is an integer greater than or equal to 1.

[0133] First, select K training object (ID) image sequence sets from the training set D, and then select T image sequences from the image sequence set of each ID, a total of K*T image sequences as an original batch, for each image sequence S in the original batch ij (represents the jth image sequence in the image sequence set of the i-th training object), where each image sequence S ij Includes multiple consecutive frames of original images of the same training object.

[0134] In some embodiments, the image pair is determined by:

[0135] If it is determined that the original image is a visible light image, converting the visible light image into an infrared image, and determining an image pair based on the visible light image and the converted infrared image;

[0136] If it is determined that the original image is an infrared image, the infrared image is converted into a visible light image, and an image pair is determined based on the infrared image and the converted visible light image.

[0137] During implementation, this embodiment can convert between visible light images and infrared images through image algorithms, and can also use deep learning, machine learning and other algorithms to convert between visible light images and infrared images. This embodiment does not impose too many restrictions on this.

[0138] In some embodiments, the original image is converted as follows:

[0139] First, based on the channel dimension of the original image, determine whether the original image is a visible light image (RGB image) or an infrared image. If the channel dimension of the original image is 3, the original image is a visible light image. If the channel dimension of the original image is 1, the original image is an infrared image.

[0140] Secondly, if it is determined that the original image is a visible light image, the visible light image S is transformed into ij Convert to infrared image S i ′ j If the original image is determined to be an infrared image, the infrared image S is converted into ij Converted to visible light image S i ′ j ;

[0141] Finally, the original image S ij The converted image S i ′ j , added to the corresponding image sequence, the original image S ij And the image S obtained by conversion i ′ j As an image pair of the image sequence, there are a total of K*T*2 image sequences forming a new batch as the input set of the input feature extraction network to train the feature extraction network. When training, the feature extraction network is trained by using multiple batches as input.

[0142] In some embodiments, the process of training the feature extraction network also includes a sequence feature extraction network and a gait feature extraction network; wherein the sequence feature extraction network and the gait feature extraction network are two pre-trained networks, the sequence feature extraction network is used to extract video sequence features, such as ReID features, and the gait feature extraction network is used to extract gait features. In this embodiment, during the process of training the feature extraction network, by using the sequence feature extraction network and the gait feature extraction network, the feature extraction network can be equipped with the ability to simultaneously learn both video sequence features and gait features. This allows the features extracted by the final feature extraction network to be more representative, have better recognition effects, and possess both ReID features and gait features.

[0143] In some embodiments, the feature extraction network includes a backbone network, a first extraction network, and a second extraction network; the training process of the feature extraction network includes:

[0144] Process (1) inputs the image sequence set into the sequence feature extraction network and outputs the corresponding sequence feature set; inputs the image sequence set into the gait feature extraction network and outputs the corresponding gait feature set; inputs the image sequence set into the backbone network of the feature extraction network and outputs the image semantic feature set; inputs the image semantic feature set into the first extraction network and the second extraction network to obtain the first feature set and the second feature set respectively;

[0145] During the training process, the new batch composed of the above K*T*2 image sequences is input into the three branches respectively, and the sequence features are extracted using the sequence feature extraction network branch, and the sequence feature set F is output. rt , use the gait feature extraction network to extract gait features, input gait feature set F gt , use the first extraction network of the feature extraction network to extract the first feature and output the first feature set F hs1 , use the second extraction network of the feature extraction network to extract the second feature and output the second feature set F hs2 .

[0146] Among them, the sequence features in the sequence feature set and the input image sequence are in one-to-one correspondence. Similarly, the gait features in the gait feature set and the input image sequence are in one-to-one correspondence. The first features in the first feature set and the input image sequence are in one-to-one correspondence. The second features in the second feature set and the input image sequence are in one-to-one correspondence.

[0147] Process (2) determines a first distillation loss value based on the sequence feature set and the first feature set, and determines a second distillation loss value based on the gait feature set and the second feature set;

[0148] In practice, the distillation loss is the MSE loss L mse Function can be calculated by the following formula:

[0149]

[0150] In formula (1), C d Represents the dimensions of features F1 and F2, and i represents the i-th dimension.

[0151] The first distillation loss value is expressed as L mse (F rt ,F hs1 ), the second distillation loss value is expressed as L mse (F gt ,F hs2 ), where F rt represents the sequence feature set, F hs1 represents the first feature set, F gt represents the gait feature set, F hs2 Represents the second feature set.

[0152] Process (3) trains the feature extraction network according to the feature loss values ​​corresponding to the sequence feature set, the gait feature set, the first feature set, and the second feature set, as well as the first distillation loss value and the second distillation loss value.

[0153] In implementation, the feature loss value includes the triple loss L triplet (Triplet Loss) and classification loss L cls ; Among them, the triplet loss is calculated as follows:

[0154] TripletLoss=max*(max(d(a,p))-min(d(a,n))+m,0) Formula (2);

[0155] In formula (2), TripletLoss represents the triplet loss, a represents the feature extracted from any image sequence in a batch (image sequence set), p represents the feature set of the image sequence set that belongs to the same training object (ID) as a, n represents the feature set of the image sequence set that belongs to a different ID than a, and d(·) represents the distance metric function.

[0156] The classification loss is calculated as follows:

[0157] First, use the spatial mapping matrix to transform the dimension C d Features of image sequences Mapped to (where C id Represents the number of IDs in the training set), and then uses Softmax Loss to calculate the classification loss.

[0158] In implementation, the feature loss value is expressed by the following formula:

[0159]

[0160]

[0161]

[0162]

[0163] Among them, L rt The feature loss representing the sequence features, represents the triplet loss calculated for sequence features, represents the classification loss calculated by sequence features; L gt The feature loss representing the gait characteristics, represents the triplet loss of gait feature calculation, represents the classification loss of gait feature calculation; L hs1 represents the feature loss of the first feature, represents the triplet loss calculated for the first feature, represents the classification loss calculated by the first feature; L hs2 represents the feature loss of the second feature, represents the triplet loss calculated for the second feature, Represents the classification loss calculated for the second feature.

[0164] In some embodiments, the feature extraction network also includes an attribute classification network, which is used to extract local features of the target object; optionally, the local features in this embodiment include but are not limited to hairstyle, accessories (backpacks, hair accessories, scarves, hats, skateboards, etc.), gender, etc.

[0165] The training process of the feature extraction network also includes:

[0166] The image semantic feature set is input into the attribute classification network, a corresponding attribute classification result set is output, an attribute classification loss value is determined according to the attribute classification result set; and the feature extraction network is trained according to the attribute classification loss value.

[0167] Attribute classification loss is calculated by using a spatial mapping matrix to transform the dimension C d Features of image sequences Mapped to (where C id Represents the number of IDs in the training set), and then uses Softmax Loss to calculate the classification loss.

[0168] Optionally, the feature extraction network can be trained using the following total loss function:

[0169] L total =L mse (F rt ,F hs1 )+L mse (F gt ,F hs2 )+L attr +L rt +L gt +L hs1 +L hs2

[0170] Formula (7);

[0171] Among them, L total Represents the total loss function, L mse (F rt ,F hs1) represents the first distillation loss, L mse (F gt ,F hs2 ) represents the second distillation loss, L attr represents the attribute classification loss, L rt The feature loss of the sequence feature, L gt The feature loss representing the gait feature, L hs1 represents the feature loss of the first feature, L hs2 Represents the feature loss of the second feature.

[0172] like Figure 2 As shown, this embodiment provides a schematic diagram of a network architecture during the training process of a feature extraction network. The network architecture consists of three branches: a sequence feature extraction network branch, a gait feature extraction network branch, and a feature extraction network branch.

[0173] Among them, the sequence feature extraction network branch can use the common video ReID network to extract the sequence features of the input image sequence, and the gait feature extraction network can use the common gait information extraction network to use human body contour segmentation and temporal feature extraction network to extract gait features;

[0174] The structure of the feature extraction network branch includes a backbone network, a first extraction network, a second extraction network, and an attribute classification network. First, a backbone network (such as 3D convolution, 2D convolution, transformer, etc.) is used to extract semantic features. Then, the extracted semantic features are connected to three sub-branches to extract the first and second features respectively, as well as a multi-dimensional local feature extraction branch to extract attribute classification features.

[0175] The above network architecture enables the network to learn both ReID features and gait features simultaneously. Through distillation, the feature extraction network branch is able to learn both feature representations simultaneously. At the same time, the attribute classification network is used as a supervisory signal, allowing the feature extraction network to focus on subtle local attributes (hairstyle, accessories, gender, etc.), making the final fusion features more representative and achieving better recognition results. At the same time, it can have the characteristics of gait and ReID, and has the ability to recognize changes of clothing and long-distance recognition.

[0176] After the feature extraction network training is completed, only the backbone network, the first extraction network and the second extraction network of the feature extraction network are used during use, such as Figure 3 As shown, this embodiment provides a schematic diagram of the architecture of a feature extraction network inference stage. In some embodiments, the feature extraction network is used to extract object features of the image sequence to be identified through the following steps:

[0177] Step 1) inputting the image sequence to be identified into the backbone network and outputting the corresponding image semantic features;

[0178] Step 2) inputting the image semantic features into a first extraction network to output a first feature, and inputting the image semantic features into a second extraction network to output a second feature;

[0179] The first feature represents a sequence feature of the image sequence to be identified, and the second feature represents a gait feature of the target object;

[0180] Step 3) Determine the object features of the target object contained in the image sequence to be identified based on the first feature and the second feature.

[0181] In some embodiments, one approach is to determine a spliced ​​feature obtained by splicing the first feature and the second feature as the object feature; another approach is to determine a fused feature obtained by adding the first feature and the second feature as the object feature.

[0182] In practice, during the inference phase of the feature extraction network, the object features ultimately output by the feature extraction network are calculated using any of the following formulas:

[0183] F fuse =cat(F hs1 ,F hs2 ) formula (8);

[0184] F fuse =F hs1 +F hs2 Formula (9);

[0185] Among them, F fuse Indicates object features, cat() means splicing features, F hs1 represents the first feature, F hs2 Indicates the second feature.

[0186] In some embodiments, this embodiment uses a neural network to build a first conversion network and a second conversion network, and uses the first conversion network and the second conversion network to achieve image style domain migration. The first conversion network is used to convert visible light images into infrared images, and the second conversion network is used to convert infrared images into visible light images.

[0187] In some embodiments, the first and second conversion networks are trained based on a sample set, wherein the sample set includes multiple sample pairs, each of which includes adjacent frame images; optical flow represents the difference between adjacent frame images. Optical flow is the apparent movement of image brightness patterns, expressing image changes. Because it contains information about the movement of an object, it can be used by an observer to determine the object's motion.

[0188] The training process of the first conversion network and the second conversion network in this embodiment specifically includes:

[0189] In the first case, the sample pair contains images that are visible light images:

[0190] The sample pair is input into a first conversion network to output an infrared sample pair; the infrared sample pair is input into a second conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using the reconstruction loss value determined by the sample pair and the reconstructed sample pair.

[0191] In implementation, the reconstruction loss function is used to restore the image after one round of reconstruction to the input image.

[0192] In some embodiments, to enhance the quality of image migration, this embodiment further adds inter-frame difference loss so that the first conversion network and the second conversion network can preserve the temporal information between images. The specific implementation process is as follows:

[0193] A first optical flow of the sample pair and a second optical flow of the infrared sample pair are determined, wherein a first conversion network and a second conversion network are trained according to inter-frame difference loss values ​​determined by the first optical flow and the second optical flow.

[0194] A third optical flow of the sample pair and a fourth optical flow of the visible light sample pair are determined, wherein the first conversion network and the second conversion network are trained according to the inter-frame difference loss value determined by the third optical flow and the fourth optical flow.

[0195] During implementation, optical flow is used to represent the inter-frame differences between adjacent frame images contained in a sample pair, and the inter-frame difference loss is determined using the first and second optical flows, allowing the first conversion network to retain the temporal information between images. Similarly, the inter-frame difference loss is determined using the third and fourth optical flows, allowing the second conversion network to retain the temporal information between images.

[0196] In some embodiments, to enhance the quality of image transfer, this embodiment also adds a discriminant classification loss to determine whether the image is a true infrared / visible light image, making the converted image as realistic as possible. The specific implementation process is as follows:

[0197] The first conversion network and the second conversion network are trained according to the discriminant classification loss value determined by the infrared sample pair and the pre-acquired real infrared image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real visible light image.

[0198] In the second case, the sample pair contains an infrared image:

[0199] The sample pairs are input into a second conversion network to output a visible light sample pair; the visible light sample pairs are input into a first conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using reconstruction loss values ​​determined by the sample pairs and the reconstructed sample pairs.

[0200] In implementation, the reconstruction loss function is used to restore the image after one round of reconstruction to the input image.

[0201] In some embodiments, to enhance the quality of image migration, this embodiment further adds inter-frame difference loss so that the first conversion network and the second conversion network can preserve the temporal information between images. The specific implementation process is as follows:

[0202] A first optical flow of the sample pair and a second optical flow of the infrared sample pair are determined, wherein a first conversion network and a second conversion network are trained according to inter-frame difference loss values ​​determined by the first optical flow and the second optical flow.

[0203] A third optical flow of the sample pair and a fourth optical flow of the visible light sample pair are determined, wherein the first conversion network and the second conversion network are trained according to the inter-frame difference loss value determined by the third optical flow and the fourth optical flow.

[0204] During implementation, optical flow is used to represent the inter-frame differences between adjacent frame images contained in a sample pair, and the inter-frame difference loss is determined using the first and second optical flows, allowing the first conversion network to retain the temporal information between images. Similarly, the inter-frame difference loss is determined using the third and fourth optical flows, allowing the second conversion network to retain the temporal information between images.

[0205] In some embodiments, to enhance the quality of image transfer, this embodiment also adds a discriminant classification loss to determine whether the image is a true infrared / visible light image, making the converted image as realistic as possible. The specific implementation process is as follows:

[0206] The first conversion network and the second conversion network are trained according to the discriminant classification loss value determined by the visible light sample pair and the pre-acquired real visible light image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real infrared image.

[0207] In some embodiments, in order to train a good generator, this embodiment designs three sets of loss functions to enhance the quality of image transfer, including a discrimination and classification loss function, an inter-frame difference loss function, and a reconstruction loss function. Among them:

[0208] Classification loss functions include but are not limited to cross entropy loss function;

[0209] Reconstruction loss includes but is not limited to L2 loss, which is calculated as follows:

[0210]

[0211] L(t,t * ) represents the reconstruction loss value between the t-th frame image in the image pair input to the first conversion network and the t*-th frame image output after the t-th frame image is converted by the first conversion network and the second conversion network, M, N are the length resolution and width resolution of the image contained in the sample pair, i, j represent the pixel coordinates of the image, t represents the t-th frame image in the image pair input to the first conversion network, t* represents the t*-th frame image in the image pair output from the second conversion network, where the t-th frame image corresponds to the t*-th frame image. Similarly, L(t+1,t+1 * ) represents the reconstruction loss value between the t+1th frame image in the image pair input to the first conversion network and the t+1*th frame image output after the tth frame image is converted by the first conversion network and the second conversion network.

[0212] This embodiment also uses optical flow and inter-frame difference loss to train the first and second conversion networks, ensuring that the image frames generated by the first and second conversion networks maintain consistency in brightness, style, and other aspects. The first conversion network is used to convert color images into infrared images, and the second conversion network is used to convert two infrared frames into corresponding color images. The inter-frame difference loss uses the L2 loss function, and the optical flow algorithm uses network optical flow to facilitate the calculation of loss backpropagation.

[0213] like Figure 4 As shown, this embodiment also provides a schematic diagram of the training process of the first conversion network and the second conversion network. Taking the input sample pair containing visible light images as an example, the network structures of the first conversion network and the second conversion network can be the same or different. The first conversion network and the second conversion network can be convolutional neural networks or vision transformers. The discriminator in the figure is a binary classification neural network, mainly used to distinguish whether the image is a converted image or a real image.

[0214] like Figure 5As shown, this embodiment also provides a schematic diagram of the training process of the first conversion network and the second conversion network. Taking the input sample pair containing an infrared image as an example, the network structures of the first conversion network and the second conversion network can be the same or different. The first conversion network and the second conversion network can be convolutional neural networks or vision transformers. The discriminator in the figure is a binary classification neural network, mainly used to distinguish whether the image is a converted image or a real image.

[0215] In the inference phase, if it is necessary to use the video V = {t1, t2, ..., t N}, the image pair is input and the first inference uses the image pair {t1, t2} to get the transformed {t′ 1,1 ,t′ 1,2}, the second inference uses the image pair {t2, t3} to get the transformed {t′ 2,2 ,t′ 2,3}, and so on. Finally, take V′={t′ 1,1 ,t′ 1,2 ,t′ 2,3 ,…,t′ N-1,N} is the new video obtained by conversion.

[0216] When the first conversion network or the second conversion network is used for conversion, two original images of adjacent frames are input, and the two original images are converted at the same time to obtain converted original images.

[0217] This embodiment utilizes an infrared-to-visible light or visible light-to-infrared conversion network based on inter-frame information constraints to enhance the richness of training data, improve the quality of image transfer, make video transfer smoother, and improve the quality of video generation. Through a multi-feature fusion training structure and the use of distillation, multiple representations are learned simultaneously, enabling the network to identify long-distance targets at night and targets that have changed clothing.

[0218] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device. Since the electronic device is the electronic device in the method in the embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0219] like Figure 6 As shown, the electronic device includes a processor 600 and a memory 601, wherein the memory 601 is used to store a program executable by the processor 600, and the processor 600 is used to read the program in the memory 601 and perform the following steps:

[0220] Obtain an image sequence to be identified containing the target object;

[0221] Inputting the image sequence to be identified into a feature extraction network, and outputting object features of the target object contained in the image sequence to be identified; wherein a training set of the feature extraction network includes a set of image sequences corresponding to a plurality of training objects, the image sequences in the set of image sequences include a plurality of image pairs, the image pairs include two images, one of which is obtained by converting the other image, and the two images contained in the image pairs are of different image types;

[0222] Identify the object features of the target object contained in the to-be-identified image sequence and determine the identification result of the target object.

[0223] As an optional implementation manner, the image pair includes an original image and an image converted from the original image; the processor 600 is specifically configured to determine the image pair in the following manner:

[0224] If it is determined that the original image is a visible light image, converting the visible light image into an infrared image, and determining an image pair based on the visible light image and the converted infrared image;

[0225] If it is determined that the original image is an infrared image, the infrared image is converted into a visible light image, and an image pair is determined based on the infrared image and the converted visible light image.

[0226] As an optional implementation manner, the processor 600 is specifically configured to execute:

[0227] If it is determined that the original image is a visible light image, converting the visible light image into an infrared image using a first conversion network; if it is determined that the original image is an infrared image, converting the infrared image into a visible light image using a second conversion network;

[0228] The first conversion network and the second conversion network are obtained by training based on a sample set, the sample set includes a plurality of sample pairs, and the training process of the first conversion network and the second conversion network includes:

[0229] If the image included in the sample pair is a visible light image, the sample pair is input into a first conversion network to output an infrared sample pair; the infrared sample pair is input into a second conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using reconstruction loss values ​​determined from the sample pair and the reconstructed sample pair;

[0230] If the images contained in the sample pair are infrared images, the sample pair is input into the second conversion network to output a visible light sample pair; the visible light sample pair is input into the first conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained by using a reconstruction loss value determined by the sample pair and the reconstructed sample pair.

[0231] As an optional implementation, the sample pair includes adjacent frame images; the optical flow represents inter-frame difference information of the adjacent frame images; and the processor 600 is specifically configured to perform:

[0232] If the images contained in the sample pair are visible light images, a first optical flow of the sample pair and a second optical flow of the infrared sample pair are determined, and the first conversion network and the second conversion network are trained by using an inter-frame difference loss value determined by the first optical flow and the second optical flow;

[0233] If the images contained in the sample pair are infrared images, a third optical flow of the sample pair and a fourth optical flow of the visible light sample pair are determined, and the first conversion network and the second conversion network are trained by using an inter-frame difference loss value determined by the third optical flow and the fourth optical flow.

[0234] As an optional implementation, the processor 600 is specifically configured to perform:

[0235] If the images contained in the sample pair are visible light images, the first conversion network and the second conversion network are trained by using a discriminative classification loss value determined by the infrared sample pair and a real infrared image obtained in advance, and a discriminative classification loss value determined by the reconstructed sample pair and a real visible light image obtained in advance;

[0236] If the images contained in the sample pair are infrared images, the first conversion network and the second conversion network are trained by using a discriminative classification loss value determined by the visible light sample pair and a real visible light image obtained in advance, and a discriminative classification loss value determined by the reconstructed sample pair and a real infrared image obtained in advance.

[0237] As an optional implementation, the feature extraction network includes a backbone network, a first extraction network and a second extraction network; and the processor 600 is specifically configured to perform:

[0238] The image sequence to be recognized is input into the backbone network to output corresponding image semantic features;

[0239] The image semantic features are input into the first extraction network to output first features, and the image semantic features are input into the second extraction network to output second features; wherein the first features represent sequence features of the image sequence to be recognized, and the second features represent gait features of the target object.

[0240] According to the first feature and the second feature, an object feature of a target object contained in the image sequence to be identified is determined.

[0241] As an optional implementation, the processor 600 is specifically configured to perform:

[0242] The spliced feature obtained by splicing the first feature and the second feature is determined as the object feature; or,

[0243] The fusion feature obtained by adding the first feature and the second feature is determined as the object feature.

[0244] As an optional implementation, the feature extraction network further includes a sequence feature extraction network and a gait feature extraction network; the feature extraction network includes a backbone network, a first extraction network and a second extraction network; the processor 600 is specifically configured to perform:

[0245] The image sequence set is input into the sequence feature extraction network to output a corresponding sequence feature set; the image sequence set is input into the gait feature extraction network to output a corresponding gait feature set; the image sequence set is input into the backbone network of the feature extraction network to output an image semantic feature set, and the image semantic feature set is input into the first extraction network and the second extraction network to obtain a first feature set and a second feature set, respectively;

[0246] A first distillation loss value is determined according to the sequence feature set and the first feature set, and a second distillation loss value is determined according to the gait feature set and the second feature set;

[0247] The feature extraction network is trained according to feature loss values corresponding to the sequence feature set, the gait feature set, the first feature set and the second feature set, and the first distillation loss value and the second distillation loss value.

[0248] As an optional implementation, the feature extraction network further includes an attribute classification network, and the attribute classification network is used to extract a local feature of the target object; the processor 600 is specifically further configured to perform:

[0249] The image semantic feature set is input into the attribute classification network to output a corresponding attribute classification feature set, and an attribute classification loss value is determined according to the attribute classification feature set;

[0250] The feature extraction network is trained according to the attribute classification loss value.

[0251] Based on the same inventive concept, an embodiment of the present invention also provides a target object identification device. Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0252] like Figure 7 As shown, the device includes:

[0253] An image sequence acquisition unit 700 is used to acquire an image sequence to be identified containing a target object;

[0254] An image feature extraction unit 701 is configured to input the to-be-recognized image sequence into a feature extraction network and output object features of a target object contained in the to-be-recognized image sequence; wherein a training set for the feature extraction network includes a set of image sequences corresponding to a plurality of training objects, an image sequence in the set of image sequences includes a plurality of image pairs, each of the image pairs including two images, one of which is obtained by converting the other image, and the two images contained in the image pairs are of different image types;

[0255] The recognition result determination unit 702 is configured to identify the object features of the target object contained in the to-be-recognized image sequence and determine the recognition result of the target object.

[0256] As an optional implementation manner, the image pair includes an original image and an image obtained by converting the original image; the image feature extraction unit 701 is specifically configured to determine the image pair in the following manner:

[0257] If it is determined that the original image is a visible light image, converting the visible light image into an infrared image, and determining an image pair based on the visible light image and the converted infrared image;

[0258] If it is determined that the original image is an infrared image, the infrared image is converted into a visible light image, and an image pair is determined based on the infrared image and the converted visible light image.

[0259] As an optional implementation manner, the image feature extraction unit 701 is specifically configured to:

[0260] If it is determined that the original image is a visible light image, converting the visible light image into an infrared image using a first conversion network; if it is determined that the original image is an infrared image, converting the infrared image into a visible light image using a second conversion network;

[0261] The first conversion network and the second conversion network are obtained by training based on a sample set, the sample set includes a plurality of sample pairs, and the training process of the first conversion network and the second conversion network includes:

[0262] If the image included in the sample pair is a visible light image, the sample pair is input into a first conversion network to output an infrared sample pair; the infrared sample pair is input into a second conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using reconstruction loss values ​​determined from the sample pair and the reconstructed sample pair;

[0263] If the image contained in the sample pair is an infrared image, the sample pair is input into the second conversion network to output a visible light sample pair; the visible light sample pair is input into the first conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using the reconstruction loss values ​​determined by the sample pair and the reconstructed sample pair.

[0264] As an optional implementation, the sample pair includes adjacent frame images; the optical flow represents inter-frame difference information of the adjacent frame images; and the image feature extraction unit 701 is further configured to:

[0265] If the image included in the sample pair is a visible light image, determining a first optical flow of the sample pair and a second optical flow of the infrared sample pair, wherein a first conversion network and a second conversion network are trained according to an inter-frame difference loss value determined by the first optical flow and the second optical flow;

[0266] If the image included in the sample pair is an infrared image, a third optical flow of the sample pair and a fourth optical flow of the visible light sample pair are determined, wherein the first conversion network and the second conversion network are trained according to the inter-frame difference loss value determined by the third optical flow and the fourth optical flow.

[0267] As an optional implementation manner, the image feature extraction unit 701 is further specifically configured to:

[0268] If the image included in the sample pair is a visible light image, training the first conversion network and the second conversion network according to the discriminant classification loss value determined by the infrared sample pair and the pre-acquired real infrared image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real visible light image;

[0269] If the image included in the sample pair is an infrared image, the first conversion network and the second conversion network are trained based on the discriminant classification loss value determined by the visible light sample pair and the pre-acquired real visible light image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real infrared image.

[0270] As an optional implementation, the feature extraction network comprises a backbone network, a first extraction network and a second extraction network; the image feature extraction unit 701 is specifically configured to:

[0271] input the image sequence to be recognized into the backbone network, and output corresponding image semantic features;

[0272] input the image semantic features into the first extraction network to output first features, and input the image semantic features into the second extraction network to output second features; wherein the first features represent sequence features of the image sequence to be recognized, and the second features represent gait features of the target object;

[0273] determine object features of the target object contained in the image sequence to be recognized according to the first features and the second features.

[0274] As an optional implementation, the image feature extraction unit 701 is specifically configured to:

[0275] determine the splicing features obtained by splicing the first features and the second features as the object features; or,

[0276] determine the fusion features obtained by adding the first features and the second features as the object features.

[0277] As an optional implementation, further comprising a sequence feature extraction network and a gait feature extraction network; the feature extraction network comprises a backbone network, a first extraction network and a second extraction network; the image feature extraction unit 701 is specifically configured to:

[0278] input an image sequence set into the sequence feature extraction network to output a corresponding sequence feature set; input the image sequence set into the gait feature extraction network to output a corresponding gait feature set; input the image sequence set into the backbone network of the feature extraction network to output an image semantic feature set, and input the image semantic feature set into the first extraction network and the second extraction network to obtain a first feature set and a second feature set, respectively;

[0279] determine a first distillation loss value according to the sequence feature set and the first feature set, and determine a second distillation loss value according to the gait feature set and the second feature set;

[0280] train the feature extraction network according to feature loss values corresponding to the sequence feature set, the gait feature set, the first feature set and the second feature set, and the first distillation loss value and the second distillation loss value.

[0281] As an optional implementation, the feature extraction network further includes an attribute classification network, which is used to extract local features of the target object; the image feature extraction unit 701 is further specifically used to:

[0282] Inputting the image semantic feature set into the attribute classification network, outputting a corresponding attribute classification feature set, and determining an attribute classification loss value based on the attribute classification feature set;

[0283] The feature extraction network is trained according to the attribute classification loss value.

[0284] Based on the same inventive concept, embodiments of the present disclosure provide a computer storage medium comprising computer program code. When executed on a computer, the computer performs any of the target object recognition methods discussed above. Because the principles underlying the problem solved by the computer storage medium are similar to those of the target object recognition method, the implementation of the computer storage medium can be referenced to the implementation of the method, and any repetitions will not be repeated.

[0285] In a specific implementation process, computer storage media may include: Universal Serial Bus Flash Drive (USB), mobile hard disk, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0286] Based on the same inventive concept, embodiments of the present disclosure further provide a computer program product, comprising computer program code that, when executed on a computer, causes the computer to perform any of the target object recognition methods discussed above. Because the principles underlying the problems solved by the aforementioned computer program products are similar to those of the target object recognition methods, the implementation of the aforementioned computer program products can be referenced to the implementation of the methods, and any repetitions will not be repeated.

[0287] The computer program product can employ any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0288] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0289] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that performs the functions specified in one or more boxes.

[0290] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0291] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0292] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A target object recognition method, characterized in that: The method includes: Obtain an image sequence to be identified containing the target object; The image sequence to be identified is input into a feature extraction network, and the object features of the target object contained in the image sequence to be identified are output; wherein the training set of the feature extraction network includes a plurality of image sequence sets corresponding to the training objects, and the image sequence in the image sequence set includes a plurality of image pairs, and the image pair includes two images, one of which is obtained by converting the other image, and the image types of the two images contained in the image pair are different; further comprising a sequence feature extraction network and a gait feature extraction network; the feature extraction network includes a backbone network, a first extraction network and a second extraction network; the training process of the feature extraction network includes: inputting the image sequence set into the sequence feature extraction network Taking the network and outputting the corresponding sequence feature set; inputting the image sequence set into the gait feature extraction network and outputting the corresponding gait feature set; inputting the image sequence set into the backbone network of the feature extraction network and outputting the image semantic feature set; inputting the image semantic feature set into the first extraction network and the second extraction network to obtain the first feature set and the second feature set respectively; determining a first distillation loss value based on the sequence feature set and the first feature set, and determining a second distillation loss value based on the gait feature set and the second feature set; training the feature extraction network based on the feature loss values ​​corresponding to the sequence feature set, the gait feature set, the first feature set, and the second feature set, as well as the first distillation loss value and the second distillation loss value; Identify the object features of the target object contained in the to-be-identified image sequence and determine the identification result of the target object.

2. The method according to claim 1, characterized in that The image pair includes an original image and an image converted from the original image; the image pair is determined by: If it is determined that the original image is a visible light image, converting the visible light image into an infrared image, and determining an image pair based on the visible light image and the converted infrared image; If it is determined that the original image is an infrared image, the infrared image is converted into a visible light image, and an image pair is determined based on the infrared image and the converted visible light image.

3. The method according to claim 2, characterized in that If it is determined that the original image is a visible light image, converting the visible light image into an infrared image using a first conversion network; if it is determined that the original image is an infrared image, converting the infrared image into a visible light image using a second conversion network; The first conversion network and the second conversion network are obtained by training based on a sample set, the sample set includes a plurality of sample pairs, and the training process of the first conversion network and the second conversion network includes: If the image included in the sample pair is a visible light image, the sample pair is input into a first conversion network to output an infrared sample pair; the infrared sample pair is input into a second conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using reconstruction loss values ​​determined from the sample pair and the reconstructed sample pair; If the image contained in the sample pair is an infrared image, the sample pair is input into the second conversion network to output a visible light sample pair; the visible light sample pair is input into the first conversion network to output a reconstructed sample pair; and the first conversion network and the second conversion network are trained using the reconstruction loss values ​​determined by the sample pair and the reconstructed sample pair.

4. The method according to claim 3, characterized in that The sample pair includes adjacent frame images; the optical flow represents the difference information between the adjacent frame images; and further includes: If the image included in the sample pair is a visible light image, determining a first optical flow of the sample pair and a second optical flow of the infrared sample pair, wherein a first conversion network and a second conversion network are trained according to an inter-frame difference loss value determined by the first optical flow and the second optical flow; If the image included in the sample pair is an infrared image, a third optical flow of the sample pair and a fourth optical flow of the visible light sample pair are determined, wherein the first conversion network and the second conversion network are trained according to the inter-frame difference loss value determined by the third optical flow and the fourth optical flow.

5. The method according to claim 3, characterized in that Also includes: If the image included in the sample pair is a visible light image, training the first conversion network and the second conversion network according to the discriminant classification loss value determined by the infrared sample pair and the pre-acquired real infrared image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real visible light image; If the image included in the sample pair is an infrared image, the first conversion network and the second conversion network are trained based on the discriminant classification loss value determined by the visible light sample pair and the pre-acquired real visible light image, and the discriminant classification loss value determined by the reconstructed sample pair and the pre-acquired real infrared image.

6. The method according to any one of claims 1 to 5, characterized in that: The feature extraction network includes a backbone network, a first extraction network and a second extraction network; Inputting the image sequence to be identified into a feature extraction network and outputting object features of the target object contained in the image sequence to be identified includes: Inputting the image sequence to be recognized into the backbone network and outputting corresponding image semantic features; Inputting the image semantic features into a first extraction network to output a first feature, and inputting the image semantic features into a second extraction network to output a second feature; wherein the first feature represents a sequence feature of an image sequence to be identified, and the second feature represents a gait feature of the target object; An object feature of a target object included in the sequence of images to be identified is determined according to the first feature and the second feature.

7. The method according to claim 6, characterized in that Determining, based on the first feature and the second feature, an object feature of a target object included in the sequence of images to be identified, includes: Determine a spliced ​​feature obtained by splicing the first feature and the second feature as the object feature; or A fusion feature obtained by adding the first feature and the second feature is determined as the object feature.

8. The method according to claim 1, characterized in that The feature extraction network further includes an attribute classification network, which is used to extract local features of the target object; the training process of the feature extraction network further includes: Inputting the image semantic feature set into the attribute classification network, outputting a corresponding attribute classification feature set, and determining an attribute classification loss value based on the attribute classification feature set; The feature extraction network is trained according to the attribute classification loss value.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and execute the steps of any one of the methods of claims 1 to 8.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images