Pedestrian re-identification method and related device

Abstract

The invention discloses a pedestrian re-identification method and a related device, and relates to the technical field of image processing, and the method comprises the steps: obtaining a ground monitoring image of a target pedestrian and a to-be-identified unmanned plane image set, employing a pedestrian re-identification model to extract the local features, global features and semantic features of the ground monitoring image and each unmanned plane image, and obtaining a pedestrian re-identification model; and determining the unmanned aerial vehicle image of the target pedestrian from the to-be-identified unmanned aerial vehicle image set based on the extracted features. According to the method, the pedestrian re-identification model is trained based on the multi-scale and multi-view training data, the model can capture more global and local detail features and environmental context features of the pedestrian due to multiple scales, the discrimination ability of the model to the pedestrian is enhanced, the model can provide better re-identification performance when facing problems such as shielding, and the pedestrian re-identification efficiency is improved. And meanwhile, the unmanned aerial vehicle image can collect wider environment information, so that more context information can be provided for the pedestrian re-identification process, and the re-identification performance is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a pedestrian re-identification method and related devices. Background Art

[0002] Person Re-identification (ReID) technology is an important technology in the field of computer vision. Its core is to use computer vision technology to determine whether there is a specific pedestrian in an image or video sequence. This technology is widely used in video surveillance, security, intelligent transportation and other fields, and is of great significance to improving public safety.

[0003] The current mainstream pedestrian re-identification methods mainly include: detection methods based on traditional image pedestrian features and end-to-end feature extraction and matching methods based on deep learning.

[0004] The detection method based on traditional image pedestrian features mainly uses the outer ring features, shape features, gait features and other information of pedestrians for modeling and matching. The typical approach is to extract the feature descriptors of pedestrian images, and then measure the similarity between different pedestrian descriptors through distance metrics or learned similarity functions. This method has a relatively good pedestrian re-identification effect for images with limited viewpoint changes. However, in drone scenarios, due to the large changes in viewpoint, scale and posture, it is difficult for a single feature descriptor to obtain ideal re-identification performance.

[0005] The end-to-end feature extraction and matching method based on deep learning trains the pedestrian re-identification model through large-scale training data, and automatically discovers high-level pedestrian semantic features based on the pedestrian re-identification model, thereby improving the performance and robustness of re-identification. However, the training data of the pedestrian re-identification model is often taken by surveillance cameras, which are shot at a low altitude and the lens is still, while drones often shoot pedestrians while moving at high altitudes. The data distribution in the two scenarios is quite different, which makes it difficult to directly apply the pedestrian re-identification model trained in classic scenarios to drone scenarios. Summary of the invention

[0006] In view of the above problems, this application provides a pedestrian re-identification method and related devices to achieve the purpose of combining drones with ground monitoring to accurately re-identify pedestrians. The specific solution is as follows:

[0007] The first aspect of the present application provides a pedestrian re-identification method, comprising:

[0008] Acquire a ground surveillance image of a target pedestrian and a set of unmanned aerial vehicle images to be identified, wherein the set of unmanned aerial vehicle images to be identified includes multiple unmanned aerial vehicle images;

[0009] Using a pedestrian re-identification model, respectively extract local features, global features, and semantic features of the ground surveillance image and each of the drone images, so as to determine the drone image of the target pedestrian from the set of drone images to be identified based on the extracted features;

[0010] Among them, the pedestrian re-identification model uses training ground surveillance images and training drone image sets marked with training pedestrian labels as training data. Each drone image contained in the training drone image set has perspective and scale differences with the training ground surveillance images. The global features represent the overall appearance information of the pedestrian and the information about the pedestrian and the surrounding environment. The local features represent the detailed information of different areas of the pedestrian's body. The semantic features represent the information of the invariant attributes of the pedestrian.

[0011] In a possible implementation, for any one of the ground monitoring image and each of the drone images, a process of extracting local features of the image includes:

[0012] Segmenting the image according to body parts to obtain multiple segmented sub-images;

[0013] Using a feature extraction network to extract image features of each of the plurality of sub-images;

[0014] Performing attention-based feature enhancement processing on the image features of each of the plurality of sub-images to obtain enhanced features corresponding to the plurality of sub-images;

[0015] The enhanced features respectively corresponding to the multiple sub-images are used as local features of the image.

[0016] In a possible implementation, for any one of the ground surveillance image and each of the drone images, a process of extracting a global feature of the image includes:

[0017] A plurality of coding layers connected in series are used to perform global feature extraction processing on the image to obtain the global features of the image;

[0018] The process of using any one of the coding layers to perform global feature extraction processing includes:

[0019] Obtaining a first feature map input to the encoding layer, wherein if the encoding layer is the first encoding layer, the first feature map is an initial feature map obtained by preliminarily encoding a sequence of image blocks contained in the image; if the encoding layer is not the first encoding layer, the first feature map is a second feature map output by a previous encoding layer;

[0020] Processing the first feature map based on a multi-head attention mechanism integrated with wavelet transform to obtain query vectors and key-value pairs corresponding to multiple attention heads respectively;

[0021] Performing an inverse wavelet transform based on the downsampled feature map to obtain a first reconstructed feature map and a second reconstructed feature map, wherein the downsampled feature map is a feature map obtained in the process of processing the first feature map based on the multi-head attention mechanism fused with wavelet transform;

[0022] Obtaining an attention summary graph based on the query vectors and key-value pairs respectively corresponding to the multiple attention heads, the first reconstructed feature graph, and the second reconstructed feature graph;

[0023] Obtaining a transition feature map based on the attention summary map and the first feature map;

[0024] A second feature map output by the encoding layer is obtained based on the multi-layer perceptron and the transition feature map, wherein the second feature map output by the last encoding layer is used to obtain the global feature of the image.

[0025] In a possible implementation, the multi-head attention mechanism based on wavelet transform is used to process the first feature map to obtain query vectors and key-value pairs corresponding to multiple attention heads, including:

[0026] Normalizing the first feature map to obtain a normalized third feature map;

[0027] Performing a linear transformation on the third feature map using an embedding matrix to obtain a fourth feature map;

[0028] Decomposing the fourth feature map into four wavelet sub-bands by discrete wavelet transform;

[0029] The four wavelet sub-bands are spliced ​​together along the channel dimension, and then a convolution process is performed on the splicing result to obtain the down-sampled feature map;

[0030] Perform multiple sets of linear transformations on the downsampled feature map to obtain query vectors and key-value pairs corresponding to the multiple attention heads.

[0031] In a possible implementation, the preliminary encoding process includes:

[0032] Encoding each image block in the image block sequence contained in the image into a vector of a preset length, and obtaining a vector corresponding to each image block;

[0033] Based on the position of each image block in the image, position encoding is performed on each image block to obtain a position encoding result corresponding to each image block;

[0034] The initial feature map is obtained based on the vectors and position encoding results corresponding to all the image blocks included in the image block sequence.

[0035] In a possible implementation, for any one of the ground surveillance image and each of the drone images, a process of extracting the semantic features of the image includes:

[0036] Performing convolution processing and activation processing on the global features of the image to obtain a fifth feature map corresponding to the image, wherein a convolution dimension of a convolution function used in the convolution processing includes the number of the invariant attributes, and the activation processing is used to enhance the attention to the area where the invariant attributes are located in the image;

[0037] The global features of the image are processed using a preset attention mechanism to obtain an attribute attention map corresponding to the image;

[0038] Based on the attribute attention map and the fifth feature map corresponding to the image, the semantic features of the image are obtained.

[0039] In a possible implementation, the loss function of the pedestrian re-identification model is a joint loss function composed of a triple loss function, a metric distillation loss function and a cross entropy loss function.

[0040] A second aspect of the present application provides a pedestrian re-identification device, comprising:

[0041] An image acquisition module, used to acquire a ground surveillance image of a target pedestrian and a set of unmanned aerial vehicle images to be identified, wherein the set of unmanned aerial vehicle images to be identified includes multiple unmanned aerial vehicle images;

[0042] A pedestrian re-identification module, configured to extract local features, global features, and semantic features of the ground surveillance image and each of the drone images respectively by using a pedestrian re-identification model, so as to determine the drone image of the target pedestrian from the set of drone images to be identified based on the extracted features;

[0043] Among them, the pedestrian re-identification model uses training ground surveillance images and training drone image sets marked with training pedestrian labels as training data. Each drone image contained in the training drone image set has perspective and scale differences with the training ground surveillance images. The global features represent the overall appearance information of the pedestrian and the information about the pedestrian and the surrounding environment. The local features represent the detailed information of different areas of the pedestrian's body. The semantic features represent the information of the invariant attributes of the pedestrian.

[0044] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the pedestrian re-identification method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0045] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0046] The memory is used to store computer programs;

[0047] The processor is used to execute the computer program so that the electronic device can implement the pedestrian re-identification method of the first aspect or any implementation manner of the first aspect.

[0048] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the pedestrian re-identification method of the above-mentioned first aspect or any implementation method of the first aspect.

[0049] By means of the above technical solution, the pedestrian re-identification method provided by the present application obtains the ground surveillance image of the target pedestrian and the set of drone images to be identified, and uses the pedestrian re-identification model to respectively extract the local features, global features and semantic features of the ground surveillance image and each drone image, so as to determine the drone image of the target pedestrian from the set of drone images to be identified based on the extracted features. The present application trains the pedestrian re-identification model based on multi-scale and multi-perspective training data. The multi-scale training data enables the pedestrian re-identification model to capture more global and local detail features and environmental context features of pedestrians, thereby enhancing the model's ability to discriminate pedestrians. The multi-perspective training data enables the pedestrian re-identification model to provide better re-identification performance when facing problems such as occlusion. At the same time, drone images can collect broader environmental information, thereby providing more contextual information for the pedestrian re-identification process, thereby improving the re-identification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0051] Figure 1 A schematic diagram of a system architecture provided for this application;

[0052] Figure 2 A schematic diagram of an optional hardware structure of the terminal 100 provided in this application;

[0053] Figure 3 A schematic diagram of the structure of a server 200 provided in this application;

[0054] Figure 4 A flowchart of a pedestrian re-identification method provided in this application;

[0055] Figure 5 is a structural schematic diagram of a local feature extraction module for extracting local features of an image;

[0056] Figure 6 This is a schematic diagram of the structure of the ResNet50 feature extraction network;

[0057] Figure 7 This is a schematic diagram of the structure of the first bottleneck block in ResNet50;

[0058] Figure 8 This is a schematic diagram of the structure of the second bottleneck block in ResNet50;

[0059] Fig. 9 is a schematic diagram of the structure of a coding layer;

[0060] Fig.10 It is a schematic diagram of the structure of a multi-head self-attention module;

[0061] Fig.11 It is a structural diagram of an attribute parsing module;

[0062] Fig.12 A schematic diagram of the structure of a pedestrian re-identification device provided in this application;

[0063] Fig.13 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0064] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0065] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0066] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0067] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0068] Among them, an application can be installed on the terminal 100, and the above application and web page can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the interface and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters and return the processing results to the terminal 100.

[0069] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0070] Next describe Figure 1 The product form of the mid-terminal 100;

[0071] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0072] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0073] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0074] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0075] Among them, the input device 132 can receive input data and the like.

[0076] The display unit 140 may be used to display information input by a user or provided to a user, various menus of the terminal 100, an interactive interface, file display, and / or playback of any multimedia file.

[0077] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0078] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0079] Among them, the memory 120 can be used to store software codes related to the pedestrian re-identification method, the processor 170 can execute the steps of the pedestrian re-identification method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0080] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (Global System of Mobile Communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0081] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0082] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0083] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0084] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0085] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0086] Next describe Figure 1 The product form of the server 200;

[0087] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0088] The bus 201 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0089] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0090] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0091] The memory 204 may be used to store software codes related to the pedestrian re-identification method, and the processor 202 may execute the steps of the pedestrian re-identification method of the chip, and may also schedule other units to implement corresponding functions.

[0092] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, DSPs, microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0093] The present application provides a pedestrian re-identification method. The pedestrian re-identification method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0094] Reference Figure 4 , Figure 4 A flowchart of a pedestrian re-identification method provided in an embodiment of the present application, the method may include:

[0095] Step S401: Acquire a ground surveillance image of a target pedestrian and a set of unmanned aerial vehicle images to be identified, wherein the set of unmanned aerial vehicle images to be identified includes multiple unmanned aerial vehicle images.

[0096] In this embodiment, the required area can be monitored by the ground monitoring equipment, and when the target pedestrian is in the monitoring area, the ground monitoring equipment can collect the ground monitoring image of the target pedestrian.

[0097] At the same time, the present application can also obtain a set of drone images to be identified by taking overhead images through drones.

[0098] Optionally, the ground monitoring images and the multiple drone images in the set of drone images to be identified can be images collected synchronously or asynchronously; the ground monitoring images and the multiple drone images in the set of drone images to be identified can have perspective and / or scale differences or have no perspective and scale differences; the monitoring areas of the ground monitoring equipment and the drone can be the same or different; this application does not make specific limitations.

[0099] Optionally, the above-mentioned ground surveillance image is an image containing only the target pedestrian; optionally, the target pedestrian may include one pedestrian or multiple pedestrians. If the target pedestrian includes multiple pedestrians, the ground surveillance image in this step includes the ground surveillance images of the multiple pedestrians respectively.

[0100] Optionally, the drone image may contain only one pedestrian; it is understandable that a drone may capture multiple pedestrians in one frame of image. In order to facilitate subsequent processing, the image captured by the drone may be segmented according to pedestrians to obtain the drone image containing only one pedestrian. Optionally, the segmentation process may be implemented by a YOLO (You Only Look Once) model.

[0101] Step S402: using the pedestrian re-identification model, respectively extracting local features, global features and semantic features of the ground surveillance image and each drone image, so as to determine the drone image of the target pedestrian from the set of drone images to be identified based on the extracted features.

[0102] This embodiment can pre-train a pedestrian re-identification model. Specifically, a panoramic scene can be captured by taking a bird's-eye view image through a drone, and detailed images can be taken synchronously or asynchronously through multiple ground monitoring devices (such as ground monitoring cameras) with different viewing angles. All images are calibrated in time and space, and pedestrian positions and identifications are manually marked. Finally, the paired ground-air multi-view image pairs are organized into training, verification and test data sets in a specific format. Among them, the above-mentioned time and space position calibration refers to determining the time and location of image shooting to facilitate data archiving.

[0103] Then, the training data of the pedestrian re-identification model provided in this embodiment are training ground surveillance images and training drone image sets marked with training pedestrian labels, that is, the training process can input the training ground surveillance images and training drone image sets of training pedestrians into the pre-built neural network model to obtain the model prediction output, and then compare the model prediction output with the training drone image set marked with training pedestrian labels (that is, the label can be used to determine whether the pedestrian in the drone image is the training pedestrian) and calculate the loss, and train the parameters of the neural network model based on the loss to obtain the pedestrian re-identification model.

[0104] In order to make the pedestrian re-identification model have higher recognition accuracy, it is optional to make each drone image contained in the training drone image set have perspective and scale differences with the training ground monitoring images. The perspective difference refers to the difference between the drone's top-down angle and the ground monitoring angle to reflect the difference characteristics of pedestrian targets from different perspectives; the scale difference means that the scale of pedestrian targets in drone images should be significantly smaller than that of ground monitoring images to simulate large-scale changes in actual situations.

[0105] Optionally, the person re-identification model is trained using the Adam (Adaptive Moment Estimation) optimizer with a learning rate of 1e-4.

[0106] This embodiment can improve the robustness of the pedestrian re-identification model by using the perspective difference and scale difference between the image pairs.

[0107] In this embodiment, the pedestrian re-identification model can be used to extract the local features, global features and semantic features of the ground surveillance image and each drone image respectively, that is, to extract the local features, global features and semantic features of the ground surveillance image, and at the same time, it is also necessary to extract the local features, global features and semantic features of each drone image.

[0108] Here, global features represent the overall appearance information of pedestrians and the information about pedestrians and their surroundings; local features represent detailed information of different areas of the pedestrian's body, such as head features, clothing textures, shoe styles and other detailed features; semantic features represent information about invariant attributes of pedestrians, such as whether pedestrians are wearing hats or carrying backpacks. Due to the difference in height between the drone's perspective and the ground monitoring perspective, even different images of the same pedestrian will have greatly changed body postures, but attributes such as whether or not they are wearing hats will not change significantly. Therefore, in the process of pedestrian re-identification, the semantic features of pedestrian images, that is, attribute features, play an important role.

[0109] Furthermore, this embodiment can determine the drone image of the target pedestrian from the set of drone images to be identified based on the local features, global features, and semantic features of the extracted ground surveillance image and each drone image.

[0110] For example, optionally, the fusion features of the target pedestrian can be obtained based on the local features, global features and semantic features of the ground monitoring image; the fusion features of the pedestrian to be identified contained in each drone image can be obtained based on the local features, global features and semantic features of each drone image. Then, the fusion features of the target pedestrian are compared with the fusion features of the pedestrian to be identified contained in each drone image. The comparison process can be achieved by calculating the distance, calculating the similarity, etc., so as to determine whether the pedestrian to be identified contained in each drone image is the target pedestrian through the comparison process, and then obtain the drone image of the target pedestrian.

[0111] The pedestrian re-identification method provided in the present application obtains a ground surveillance image of a target pedestrian and a set of drone images to be identified, and uses a pedestrian re-identification model to extract local features, global features, and semantic features of the ground surveillance image and each drone image, respectively, so as to determine the drone image of the target pedestrian from the set of drone images to be identified based on the extracted features. The present application trains a pedestrian re-identification model based on multi-scale and multi-perspective training data. The multi-scale training data enables the pedestrian re-identification model to capture more global and local detail features and environmental context features of pedestrians, thereby enhancing the model's ability to discriminate pedestrians. The multi-perspective training data enables the pedestrian re-identification model to provide better re-identification performance when facing problems such as occlusion. At the same time, drone images can collect broader environmental information, thereby providing more contextual information for the pedestrian re-identification process, thereby improving the re-identification performance.

[0112] In some embodiments of the present application, the process of step S402 "using a pedestrian re-identification model to respectively extract local features, global features, and semantic features of the ground surveillance image and each drone image" is introduced.

[0113] In this embodiment, for the ground surveillance image and each drone image, the process of extracting local features, global features and semantic features is the same. For the sake of ease of introduction, the ground surveillance image and any one of the drone images are taken as an example to introduce the process of extracting local features, global features and semantic features of the image.

[0114] First, the process of extracting local features of the image includes: segmenting the image according to body parts to obtain multiple segmented sub-images, using a feature extraction network to extract image features of each of the multiple sub-images, performing attention-based feature enhancement processing on the image features of each of the multiple sub-images to obtain enhanced features corresponding to the multiple sub-images, and using the enhanced features corresponding to the multiple sub-images as local features of the image.

[0115] Optionally, the image may be segmented into a head sub-image, an upper body sub-image, and a lower body sub-image.

[0116] See also Figure 5 , shows a schematic diagram of the structure of a local feature extraction module for extracting local features of an image, such as Figure 5 ,The local feature extraction module is composed of the image segmentation module, ,feature extraction module and attention module.

[0117] Among them, the image segmentation module segments the image according to body parts to obtain multiple segmented sub-images, the feature extraction module uses a feature extraction network to extract the image features of each of the multiple sub-images, and the attention module performs attention-based feature enhancement processing on the image features of each of the multiple sub-images to obtain enhanced features corresponding to the multiple sub-images, and uses the enhanced features corresponding to the multiple sub-images as local features of the image.

[0118] Optionally, the image segmentation module is implemented by a positioning layer, which segments different regions by performing spatial operations such as scaling, moving, and cropping. The principle is as follows:

[0119] Formula (1);

[0120] in, and They are and The scaling parameter of the direction, and They are and The translation parameter of the direction, is the original coordinate of the pixel in the image, is the transformed coordinates of the pixel in the image.

[0121] Then, the position of each pixel in the segmented sub-image From its original coordinates and the transformed coordinates Sure.

[0122] Optionally, the feature extraction module can use a ResNet50 feature extraction network.

[0123] See also Figure 6 , which shows the structure diagram of the ResNet50 feature extraction network, see Figure 7 and Figure 8 , respectively, are structural diagrams of the first bottleneck block and the second bottleneck block in ResNet50.

[0124] like Figure 6 As shown in the figure, the network structure of the ResNet50 feature extraction network can be divided into five stages. The first stage consists of the initial convolution layer 1 and the maximum pooling layer. The second stage consists of a first bottleneck block and two second bottleneck blocks. The third stage consists of a first bottleneck block and three second bottleneck blocks. The fourth stage consists of a first bottleneck block and five second bottleneck blocks. The fifth stage consists of a first bottleneck block and two second bottleneck blocks. Figure 7As shown in , the first bottleneck block consists of three convolutional layers (Convolutional Layer 2, Convolutional Layer 3, and Convolutional Layer 4) connected in series, another convolutional layer (Convolutional Layer 5), and a distilled linear unit (RELU). Figure 8 As shown, the second bottleneck block consists of three convolutional layers (Convolutional Layer 6, Convolutional Layer 7, and Convolutional Layer 8) connected in series and RELU.

[0125] The convolution parameters in the above convolution layers 1 to 8 may be the same or different, and this application does not limit this.

[0126] Figure 6~Figure 8 The ResNet50 feature extraction network shown has a deeper network structure and can learn complex and abstract feature representations. At the same time, due to the existence of residual connections, features can be effectively reused between different layers of the network. This layer-by-layer feature reuse mechanism enables the network to better capture image features at different levels, thereby improving the richness and diversity of features.

[0127] After the feature extraction module extracts the image features of each of the multiple sub-images, the image features can be passed to the attention module for processing. In the attention module, the attention score of each sub-image can be determined based on the image features of each sub-image, and then the enhanced features corresponding to each sub-image can be obtained based on the attention score and image features of each sub-image, and then the enhanced features corresponding to the multiple sub-images can be used as the local features of the image.

[0128] Optionally, the calculation process of the attention score of each sub-image can use the following formula (2), and the calculation process of the enhanced feature corresponding to each sub-image can use the following formula (3).

[0129] Formula (2);

[0130] Formula (3);

[0131] in, The parameters of the person re-identification model are used to increase the dimension; The parameters of the person re-identification model are used to reduce the dimension; It is a distilled linear unit, used to introduce nonlinear representation; is the sigmoid activation function, represents the image features of the i-th sub-image, represents the attention score of the i-th sub-image, Represents the enhanced features corresponding to the i-th sub-image.

[0132] In summary, the embodiments of the present application provide a method for extracting local features. By first segmenting the pedestrian image and then combining the attention mechanism to extract the local features of the segmented sub-image, more local details of the sub-image can be extracted. Based on these local details, different pedestrians can be better distinguished, effectively improving the pedestrian re-identification effect.

[0133] Second, the process of extracting the global features of the image may include: performing global feature extraction processing on the image using a plurality of encoding layers connected in series to obtain the global features of the image.

[0134] Optionally, the encoding layer is a transformer encoder integrated with wavelet transform.

[0135] See also Fig. 9 , shows a schematic diagram of the structure of a coding layer. Fig. 9 The encoding layer consists of a multi-head self-attention module and a multi-layer perceptron module. Layer normalization (i.e., normalization module 1 and normalization module 2) is applied before each multi-head self-attention module and multi-layer perceptron module, and residual connection is applied after each multi-head self-attention module and multi-layer perceptron module.

[0136] Optional, such as Fig. 9 , the process of using any coding layer to perform global feature extraction processing may include: obtaining a first feature map input to the coding layer, wherein, if the coding layer is the first coding layer, the first feature map is an initial feature map obtained by preliminarily encoding a sequence of image blocks contained in the image, and if the coding layer is not the first coding layer, the first feature map is a second feature map output by the previous coding layer; processing the first feature map based on a multi-head attention mechanism fused with wavelet transform to obtain query vectors and key-value pairs corresponding to multiple attention heads, performing inverse wavelet transform based on the downsampled feature map to obtain a first reconstructed feature map and a second reconstructed feature map, wherein the downsampled feature map is a feature map obtained in the process of processing the first feature map based on a multi-head attention mechanism fused with wavelet transform, obtaining an attention summary map based on the query vectors and key-value pairs corresponding to the multiple attention heads, the first reconstructed feature map and the second reconstructed feature map, and obtaining a transition feature map (corresponding to) based on the attention summary map and the first feature map. Fig. 9 The normalization module 1, the multi-head self-attention module and the first residual connection of the encoding layer are used to obtain the second feature map output by the encoding layer based on the multi-layer perceptron and the transition feature map, wherein the second feature map output by the last encoding layer is used to obtain the global feature of the image (corresponding to Fig. 9 Normalization module 2, multi-layer perceptron module and the second residual connection).

[0137] Optionally, the above-mentioned process of performing preliminary encoding on the image block sequence contained in the image to obtain the initial feature map may include: encoding each image block in the image block sequence contained in the image into a vector of a preset length to obtain a vector corresponding to each image block; performing position encoding on each image block based on the position of each image block in the image to obtain a position encoding result corresponding to each image block; and obtaining the initial feature map based on the vectors and position encoding results corresponding to all the image blocks contained in the image block sequence.

[0138] Optionally, in this embodiment, the sizes of the image blocks in the image block sequence may be the same, that is, in this embodiment, the image may be divided into a plurality of image blocks of fixed sizes to form an image block sequence.

[0139] Optionally, a projection method may be adopted to project each image block in the image block sequence into a vector of a preset length.

[0140] Optionally, the calculation formula of the initial feature map is as follows: Formula (4).

[0141] Formula (4);

[0142] in, represents the initial feature map, Represents the vector corresponding to the i-th image block contained in the image, Indicates the number of image blocks contained in the image. Indicates the position encoding result of all image blocks contained in the image. Here, Included elements, where the i-th element represents the position encoding result corresponding to the i-th image block.

[0143] As described above, in this embodiment, the initial feature map As the first feature map input to the first encoding layer, the second feature map output by the first encoding layer is obtained ; Then the second feature map output by the first encoding layer As the first feature map, it is input to the second encoding layer to obtain the second feature map output by the second encoding layer. ; Similarly, The second feature map output by the encoding layer As the first feature map, input to the The second feature map of the encoding layer is obtained ; And so on, until the last encoding layer outputs the second feature map Finally, based on the second feature map output by the last encoding layer Get the global features of the image .in, , Indicates the number of coding layers.

[0144] With " The second feature map output by the encoding layer As the first feature map, input to the The second feature map of the encoding layer is obtained " process as an example, the process can be obtained using the following formula (5) and formula (6): .

[0145] Formula (5);

[0146] Formula (6);

[0147] in, represents the transition feature map, It represents the attention summary graph obtained by processing the multi-head self-attention module MA. represents normalization processing, Represents a multi-layer perceptron.

[0148] Optionally, this embodiment may adopt the following formula (7) based on the second feature map output by the last encoding layer: Get the global features of the image .

[0149] Formula (7).

[0150] In a possible implementation, the above-mentioned process of "processing the first feature map based on a multi-head attention mechanism integrated with wavelet transform to obtain query vectors and key-value pairs corresponding to multiple attention heads" may include: normalizing the first feature map to obtain a normalized third feature map; linearly transforming the third feature map using an embedding matrix to obtain a fourth feature map; decomposing the fourth feature map into four wavelet sub-bands through discrete wavelet transform; splicing the four wavelet sub-bands along the channel dimension, and then convolving the spliced ​​result to obtain a downsampled feature map; performing multiple sets of linear transformations on the downsampled feature map to obtain query vectors and key-value pairs corresponding to multiple attention heads.

[0151] The above process of "normalizing the first feature map" can be achieved by Fig. 9 The normalization module 1 is implemented.

[0152] See also Fig.10 , shows a schematic diagram of the structure of a multi-head self-attention module. Fig.10The multi-head self-attention module consists of linear layer 1~linear layer 7, wavelet transform layer 1, wavelet transform layer 2, convolution layer 9, convolution layer 10, inverse wavelet transform layer 1, inverse wavelet transform layer 2 and normalization layer 1.

[0153] The above embedding matrix is Fig.10 The parameters in the linear layer 1 of , the above process of "using the embedding matrix to linearly transform the third feature map to obtain the fourth feature map" can be used using the following formula (8).

[0154] Formula (8);

[0155] in, is the parameter in the linear layer 1 of the pedestrian re-identification model, representing the embedding matrix; represents the third feature map, i.e. Fig.10 Input; represents the fourth feature map.

[0156] The above discrete wavelet transform can be Fig.10 The wavelet transform layer 1 and the wavelet transform layer 2 are implemented. Optionally, the process of "decomposing the fourth feature map into four wavelet sub-bands by discrete wavelet transform" may include: decomposing the fourth feature map into four wavelet sub-bands by redundant wavelet transform, wherein the wavelet basis function and the transform formula are respectively the following formula (9) and formula (10).

[0157] Formula (9);

[0158] Formula (10);

[0159] in, is the value obtained by wavelet transform, is the discrete scale factor, is the discrete displacement factor, ; To sum the variables, It is an oscillating and rapidly decaying wave, which mathematically satisfies the condition of integrating to zero; Feature map that is input to the wavelet transform block.

[0160] The above-mentioned "convolution processing of the splicing result" can be achieved by Fig.10 The convolutional layer 9 and convolutional layer 10 are implemented.

[0161] Fig.10 The linear layer 3, linear layer 4 and linear layer 5 each contain Linear layers, representing Attention heads are used to perform multiple sets of linear transformations on the downsampled feature maps to obtain query vectors and key-value pairs corresponding to multiple attention heads.

[0162] Optionally, a set of linear transformations are performed on the downsampled feature map to obtain the query vector and key-value pair corresponding to the j-th attention head using the following formulas (11) to (13).

[0163] Formula (11);

[0164] Formula (12);

[0165] Formula (13);

[0166] in, , and are all parameters in the person re-identification model (i.e., parameters in linear layer 3, linear layer 4, and linear layer 5), representing the transformation matrix corresponding to the j-th attention head; represents the downsampled feature map, represents the query vector corresponding to the jth attention head, represents the key corresponding to the jth attention head, represents the value corresponding to the jth attention head, , Represents the number of attention heads.

[0167] In a possible implementation, the process of “performing an inverse wavelet transform based on the downsampled feature map to obtain a first reconstructed feature map and a second reconstructed feature map” may include: performing two sets of different linear transforms on the downsampled feature map (by Fig.10 ), obtain two different feature maps, and then perform inverse wavelet transform on the two different feature maps respectively (through Fig.10 The inverse wavelet transform layer 1 and the inverse wavelet transform layer 2 are implemented), and the first reconstructed feature map is obtained. and the second reconstructed feature map , thereby increasing the receptive field of feature extraction.

[0168] Optionally, the above process of "obtaining an attention summary graph based on the query vectors and key-value pairs corresponding to the multiple attention heads, the first reconstructed feature graph, and the second reconstructed feature graph" can be performed by Fig.10 The multiplication unit , normalization layer 1 and linear layer 7) can be calculated according to the following formula (14).

[0169] Formula (14);

[0170] in, represents the feature attention map corresponding to the jth attention head, and its calculation formula is as follows (15); , and They are all parameters in the person re-identification model, representing different transformation matrices; Represents a join operation.

[0171] Formula (15);

[0172] in, express Length, The calculation formula is as follows: Formula (16).

[0173] Formula (16);

[0174] in, Representation vector Medium Element of Normalized value, For vector The cth element in .

[0175] In summary, this embodiment introduces wavelet transform in the multi-head self-attention module for reversible downsampling, which can avoid the loss of high-frequency information such as texture details in the sampled samples, and effectively reduce the computational cost. At the same time, the inverse wavelet transform can enhance the self-attention output by aggregating local context information, further improving the accuracy of global features.

[0176] After being processed by the multi-head self-attention mechanism and the multi-layer perceptron, the feature at each position in the global feature can represent the overall relationship between a local feature and other areas in the image. Based on the overall relationship, the global features of the image can be captured more accurately as a whole, thereby improving the performance of pedestrian re-identification.

[0177] Third, the process of extracting the semantic features of the image may include: performing convolution processing and activation processing on the global features of the image to obtain a fifth feature map corresponding to the image, wherein the convolution dimension of a convolution function used in the convolution processing includes the number of invariant attributes, and the activation processing is used to enhance the attention to the area where the invariant attributes are located in the image; using a preset attention mechanism to process the global features of the image to obtain an attribute attention map corresponding to the image; based on the attribute attention map corresponding to the image and the fifth feature map, the semantic features of the image are obtained.

[0178] In this embodiment, the process of extracting the semantic features of the image can be implemented by an attribute parsing module. Optionally, a structure of an attribute parsing module is as follows: Fig.11 As shown, the attribute parsing module consists of convolutional layers 11 to 13, an activation function layer, a linear layer 8 to 11, a normalization layer 2 and a pooling layer.

[0179] In this embodiment, the attribute parsing module is located after the last encoding layer mentioned above.

[0180] The above-mentioned “convolution processing and activation processing of the global features of the image” can be achieved through convolution layers 11 to 13 and an activation function layer.

[0181] Optionally, the activation function in the activation function layer is as follows:

[0182] Formula (17);

[0183] in, , and is a scaling factor between 0 and 1, represents the output of the previous convolutional layer of the activation function in the attribute parsing module, Represents the activation value, which can be used to obtain the fifth feature map corresponding to the image.

[0184] In this embodiment, the attribute parsing module can also use a preset attention mechanism (i.e., linear layer 8 to linear layer 11 and normalization layer 2) to process the global features of the image to obtain an attribute attention map corresponding to the image. Optionally, the process can use the following formula (18).

[0185] Formula (18);

[0186] in, Represents the attribute attention map corresponding to the image, represents the preset attention mechanism, They respectively represent a set of query vectors, keys, and values ​​obtained based on the global features of the image. The query vectors, keys, and values ​​here are different from the previous ones. The keys are obtained through the linear layer 9, the values ​​are obtained through the linear layer 8, and the query vectors are obtained through the linear layer 10.

[0187] The attribute attention map corresponding to the above image The dimension is M×w×h, where M represents the number of invariant attributes, and w and h are the attribute attention maps corresponding to the image. width and height.

[0188] The above process of "obtaining the semantic features of the image based on the attribute attention map and the fifth feature map corresponding to the image" can be achieved by Fig.11 The multiplication unit and pooling layer, optionally, the process may include: obtaining the attention feature corresponding to each element based on each element and the fifth feature map contained in the attribute attention map corresponding to the image, and obtaining the semantic feature of the image based on the attention features corresponding to each element .

[0189] Optionally, the above process of "obtaining the attention feature corresponding to each element based on each element and the fifth feature map contained in the attribute attention map corresponding to the image" (through Fig.11 The multiplication unit can be calculated according to the following formula (19).

[0190] Formula (19);

[0191] in, Represents the attribute attention map corresponding to the image The attention feature corresponding to the y-th element contained in represents the fifth feature map, Represents the attribute attention map corresponding to the image The y-th element contained in .

[0192] Optionally, the above process of "obtaining the semantic features of the image based on the attention features corresponding to each element" (through Fig.11 The pooling layer implementation of the image may include: performing generalized mean pooling processing on the attention features corresponding to each element to obtain the semantic features of the image.

[0193] In summary, this embodiment uses the activation function provided by the attribute parsing module to enhance the pedestrian re-identification model's attention to the area where the relevant invariant attributes are located, while reducing potential biases, making the obtained fifth feature map more accurate, thereby improving the accuracy of the semantic features of the image.

[0194] In other embodiments of the present application, the process of fusing the local features, global features and semantic features of any image in step S402 is introduced.

[0195] Optionally, this embodiment may pre-configure weights for the local features, global features, and semantic features of the image, respectively, and then the local features, global features, and semantic features of the image may be weightedly fused using the pre-configured weights.

[0196] Optionally, the pre-configured weights may be configured based on the importance of the local features, global features, and semantic features. Including head features , Upper body features and lower body features For example, due to the differences in height, pitch angle, etc. between the drone's perspective and the ground monitoring perspective, the same body parts of pedestrians from different perspectives may have different importance. For example, the drone flies at a higher altitude, so the head of the pedestrian image it takes will have more obvious features than the lower body, and a larger weight should be pre-configured.

[0197] Optionally, the weighted fusion process may be performed using the following formula (20).

[0198] Formula (20);

[0199] in, represents the fusion feature, , , , and Represented as global features , head features , Upper body features , Lower body features and semantic features The respective pre-configured weights.

[0200] In this embodiment, It contains a comprehensive and detailed representation of pedestrians, providing a reliable distance calculation basis for cross-view pedestrian re-identification based on the UAV perspective and ground monitoring perspective. Pedestrian re-identification can be performed more accurately based on this fusion feature.

[0201] The following embodiment introduces the loss function of the above-mentioned person re-identification model.

[0202] Optionally, the loss function of the person re-identification model is a joint loss function consisting of a triplet loss function, a metric distillation loss function, and a cross entropy loss function.

[0203] The following formula (21) is the calculation formula for measuring the distillation loss function.

[0204] Formula (21);

[0205] in, Represents the image samples contained in the training data and image samples The corresponding metric distillation loss, M is the number of invariant attributes, Represents an image sample (i.e. training ground surveillance images) and image samples The total distance (i.e. training drone images); Represents the image sample under the influence of the yth invariant attribute and image samples The distance between the two image samples is determined by their respective attention features Calculated.

[0206] The following formula (22) is the calculation formula for the triplet loss function.

[0207] Formula (22);

[0208] in, Represents the image samples contained in the training data The corresponding triplet loss is, Indicates a predefined distance.

[0209] The triplet loss function aims to ensure that for any given image sample , which to negative samples Distance Greater than the positive sample Distance Add predefined distance .

[0210] The following formula (23) is the calculation formula for the cross entropy loss function.

[0211] Formula (23);

[0212] in, Indicates the number of image samples contained in the training data. represents the predicted probability of the pedestrian re-identification model for the i-th image sample, represents the true label of the one-hot encoding corresponding to the i-th image sample, Represents the cross entropy loss corresponding to the i-th image sample contained in the training data.

[0213] Optionally, this embodiment may use the following formula (24) to calculate the total model loss.

[0214] Formula (24);

[0215] in, and are the parameters of the pedestrian re-identification model. It represents the sum of the metric distillation losses corresponding to all image pairs contained in the training data (an image pair contains two image samples in the training data), Represents the sum of the triplet losses corresponding to all image samples contained in the training data. Represents the sum of the cross entropy losses corresponding to all image samples contained in the training data. Represents the total loss of the model.

[0216] In summary, this embodiment combines the metric distillation loss, triplet loss and cross entropy loss, so that the pedestrian re-identification model can be optimized with the goal of making the fusion feature distance of the same pedestrian under different perspectives as equal as possible, effectively improving the performance of the pedestrian re-identification model for pedestrian re-identification.

[0217] A pedestrian re-identification method provided in an embodiment of the present application is introduced above, and a device for executing the above-mentioned pedestrian re-identification method will be introduced below.

[0218] See also Fig.12 , Fig.12 This is a schematic diagram of the structure of a pedestrian re-identification device provided in an embodiment of the present application. Fig.12 As shown, the device may include:

[0219] An image acquisition module 501 is used to acquire a ground surveillance image of a target pedestrian and a set of unmanned aerial vehicle images to be identified, wherein the set of unmanned aerial vehicle images to be identified includes multiple unmanned aerial vehicle images;

[0220] A pedestrian re-identification module 502 is used to extract local features, global features and semantic features of the ground monitoring image and each drone image respectively by using a pedestrian re-identification model, so as to determine the drone image of the target pedestrian from the set of drone images to be identified based on the extracted features;

[0221] Among them, the pedestrian re-identification model uses training ground surveillance images and training drone image sets with labeled training pedestrians as training data. Each drone image contained in the training drone image set has perspective and scale differences from the training ground surveillance images. The global features represent the overall appearance information of the pedestrian and the information about the pedestrian and the surrounding environment. The local features represent the detailed information of different areas of the pedestrian's body. The semantic features represent the information of the invariant attributes of the pedestrian.

[0222] In a possible implementation, when extracting local features of any one of the ground surveillance image and each drone image, the pedestrian re-identification module may be used to:

[0223] Segmenting the image according to body parts to obtain multiple segmented sub-images;

[0224] A feature extraction network is used to extract the image features of each of the multiple sub-images;

[0225] Performing attention-based feature enhancement processing on the image features of each of the multiple sub-images to obtain enhanced features corresponding to the multiple sub-images;

[0226] The enhanced features corresponding to the multiple sub-images are used as local features of the image.

[0227] In a possible implementation, when extracting the global features of any one of the ground surveillance image and each drone image, the pedestrian re-identification module may be used to:

[0228] A plurality of coding layers connected in series are used to perform global feature extraction processing on the image to obtain the global features of the image;

[0229] The process of using any coding layer to perform global feature extraction processing includes:

[0230] Obtaining a first feature map input to the encoding layer, wherein if the encoding layer is the first encoding layer, the first feature map is an initial feature map obtained by preliminarily encoding a sequence of image blocks contained in the image; if the encoding layer is not the first encoding layer, the first feature map is a second feature map output by a previous encoding layer;

[0231] The first feature map is processed based on a multi-head attention mechanism integrated with wavelet transform to obtain query vectors and key-value pairs corresponding to multiple attention heads respectively;

[0232] Performing an inverse wavelet transform based on the downsampled feature map to obtain a first reconstructed feature map and a second reconstructed feature map, wherein the downsampled feature map is a feature map obtained in the process of processing the first feature map based on a multi-head attention mechanism fused with wavelet transform;

[0233] Based on the query vectors and key-value pairs corresponding to the multiple attention heads, the first reconstructed feature map, and the second reconstructed feature map, an attention summary map is obtained;

[0234] Based on the attention summary map and the first feature map, a transition feature map is obtained;

[0235] A second feature map output by the encoding layer is obtained based on the multi-layer perceptron and the transition feature map, wherein the second feature map output by the last encoding layer is used to obtain the global feature of the image.

[0236] In a possible implementation, when the above-mentioned person re-identification module processes the first feature map based on the multi-head attention mechanism integrated with wavelet transform to obtain query vectors and key-value pairs corresponding to multiple attention heads, the following can be used:

[0237] Normalizing the first feature map to obtain a normalized third feature map;

[0238] The third feature map is linearly transformed using an embedding matrix to obtain a fourth feature map;

[0239] Decomposing the fourth feature map into four wavelet subbands by discrete wavelet transform;

[0240] The four wavelet subbands are spliced ​​together along the channel dimension, and then the spliced ​​results are convolved to obtain the downsampled feature map;

[0241] Multiple sets of linear transformations are performed on the downsampled feature maps to obtain query vectors and key-value pairs corresponding to multiple attention heads.

[0242] In a possible implementation, the pedestrian re-identification module can be used to:

[0243] Encoding each image block in the image block sequence contained in the image into a vector of a preset length, and obtaining a vector corresponding to each image block;

[0244] Based on the position of each image block in the image, position encoding is performed on each image block to obtain a position encoding result corresponding to each image block;

[0245] Based on the vectors and position encoding results corresponding to all the image blocks contained in the image block sequence, an initial feature map is obtained.

[0246] In a possible implementation, when extracting the semantic features of any one of the ground surveillance image and each drone image, the pedestrian re-identification module may be used to:

[0247] Performing convolution processing and activation processing on the global features of the image to obtain a fifth feature map corresponding to the image, wherein a convolution dimension of a convolution function used in the convolution processing includes the number of invariant attributes, and the activation processing is used to enhance the attention to the region where the invariant attributes are located in the image;

[0248] The global features of the image are processed using a preset attention mechanism to obtain an attribute attention map corresponding to the image;

[0249] Based on the attribute attention map and the fifth feature map corresponding to the image, the semantic features of the image are obtained.

[0250] In a possible implementation, the loss function of the pedestrian re-identification model in the above-mentioned pedestrian re-identification module is a joint loss function composed of a triple loss function, a metric distillation loss function and a cross entropy loss function.

[0251] The pedestrian re-identification device provided in this application corresponds to the pedestrian re-identification method mentioned above. For details, please refer to the above introduction and will not be repeated here.

[0252] The present application also provides an electronic device in an embodiment. Fig.13 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Fig.13 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0253] like Fig.13 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0254] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig.13 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0255] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any pedestrian re-identification method provided in the embodiment of the present application.

[0256] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any pedestrian re-identification method provided in the embodiment of the present application.

[0257] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0258] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0259] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0260] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A pedestrian re-identification method, characterized in that: include: Acquire a ground surveillance image of a target pedestrian and a set of unmanned aerial vehicle images to be identified, wherein the set of unmanned aerial vehicle images to be identified includes multiple unmanned aerial vehicle images; Using a pedestrian re-identification model, respectively extract local features, global features, and semantic features of the ground surveillance image and each of the drone images, so as to determine the drone image of the target pedestrian from the set of drone images to be identified based on the extracted features; Among them, the pedestrian re-identification model uses training ground surveillance images and training drone image sets marked with training pedestrian labels as training data. Each drone image contained in the training drone image set has perspective and scale differences with the training ground surveillance images. The global features represent the overall appearance information of the pedestrian and the information about the pedestrian and the surrounding environment. The local features represent the detailed information of different areas of the pedestrian's body. The semantic features represent the information of the invariant attributes of the pedestrian.

2. The pedestrian re-identification method according to claim 1, characterized in that: For any one of the ground monitoring image and each of the drone images, a process of extracting local features of the image includes: Segmenting the image according to body parts to obtain multiple segmented sub-images; Using a feature extraction network to extract image features of each of the plurality of sub-images; Performing attention-based feature enhancement processing on the image features of each of the plurality of sub-images to obtain enhanced features corresponding to the plurality of sub-images; The enhanced features respectively corresponding to the multiple sub-images are used as local features of the image.

3. The pedestrian re-identification method according to claim 1, characterized in that: For any one of the ground monitoring image and each of the drone images, a process of extracting the global features of the image includes: A plurality of coding layers connected in series are used to perform global feature extraction processing on the image to obtain the global features of the image; The process of using any one of the coding layers to perform global feature extraction processing includes: Obtaining a first feature map input to the encoding layer, wherein if the encoding layer is the first encoding layer, the first feature map is an initial feature map obtained by preliminarily encoding a sequence of image blocks contained in the image; if the encoding layer is not the first encoding layer, the first feature map is a second feature map output by a previous encoding layer; Processing the first feature map based on a multi-head attention mechanism integrated with wavelet transform to obtain query vectors and key-value pairs corresponding to multiple attention heads respectively; Performing an inverse wavelet transform based on the downsampled feature map to obtain a first reconstructed feature map and a second reconstructed feature map, wherein the downsampled feature map is a feature map obtained in the process of processing the first feature map based on the multi-head attention mechanism fused with wavelet transform; Obtaining an attention summary graph based on the query vectors and key-value pairs respectively corresponding to the multiple attention heads, the first reconstructed feature graph, and the second reconstructed feature graph; Obtaining a transition feature map based on the attention summary map and the first feature map; A second feature map output by the encoding layer is obtained based on the multi-layer perceptron and the transition feature map, wherein the second feature map output by the last encoding layer is used to obtain the global feature of the image.

4. The pedestrian re-identification method according to claim 3, characterized in that: The multi-head attention mechanism based on wavelet transform is used to process the first feature map to obtain query vectors and key-value pairs corresponding to multiple attention heads, including: Normalizing the first feature map to obtain a normalized third feature map; Performing a linear transformation on the third feature map using an embedding matrix to obtain a fourth feature map; Decomposing the fourth feature map into four wavelet sub-bands by discrete wavelet transform; The four wavelet sub-bands are spliced ​​together along the channel dimension, and then a convolution process is performed on the splicing result to obtain the down-sampled feature map; Perform multiple sets of linear transformations on the downsampled feature map to obtain query vectors and key-value pairs corresponding to the multiple attention heads.

5. The pedestrian re-identification method according to claim 3, characterized in that: The preliminary coding process includes: Encoding each image block in the image block sequence contained in the image into a vector of a preset length, and obtaining a vector corresponding to each image block; Based on the position of each image block in the image, position encoding is performed on each image block to obtain a position encoding result corresponding to each image block; The initial feature map is obtained based on the vectors and position encoding results corresponding to all the image blocks included in the image block sequence.

6. The pedestrian re-identification method according to claim 3, characterized in that: For any one of the ground monitoring image and each of the drone images, a process of extracting the semantic features of the image includes: Performing convolution processing and activation processing on the global features of the image to obtain a fifth feature map corresponding to the image, wherein a convolution dimension of a convolution function used in the convolution processing includes the number of the invariant attributes, and the activation processing is used to enhance the attention to the area where the invariant attributes are located in the image; The global features of the image are processed using a preset attention mechanism to obtain an attribute attention map corresponding to the image; Based on the attribute attention map and the fifth feature map corresponding to the image, the semantic features of the image are obtained.

7. The pedestrian re-identification method according to any one of claims 1 to 6, characterized in that: The loss function of the pedestrian re-identification model is a joint loss function composed of a triplet loss function, a metric distillation loss function and a cross entropy loss function.

8. A pedestrian re-identification device, characterized in that: include: An image acquisition module, used to acquire a ground surveillance image of a target pedestrian and a set of unmanned aerial vehicle images to be identified, wherein the set of unmanned aerial vehicle images to be identified includes multiple unmanned aerial vehicle images; A pedestrian re-identification module, configured to extract local features, global features, and semantic features of the ground surveillance image and each of the drone images respectively by using a pedestrian re-identification model, so as to determine the drone image of the target pedestrian from the set of drone images to be identified based on the extracted features; Among them, the pedestrian re-identification model uses training ground surveillance images and training drone image sets marked with training pedestrian labels as training data. Each drone image contained in the training drone image set has perspective and scale differences with the training ground surveillance images. The global features represent the overall appearance information of the pedestrian and the information about the pedestrian and the surrounding environment. The local features represent the detailed information of different areas of the pedestrian's body. The semantic features represent the information of the invariant attributes of the pedestrian.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the pedestrian re-identification method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the pedestrian re-identification method as described in any one of claims 1 to 7.

Images