Pedestrian re-identification method, device and electronic equipment

Abstract

The present invention provides a person re-identification method, device, and electronic device. This relates to the field of artificial intelligence technology. The method includes: generating training samples through an adaptive learning method; training a special neural network based on the training samples to obtain a trained neural network, wherein the neural network includes scalable convolutions determined based on a convolution kernel and a convolution kernel offset; and performing person re-identification based on the trained neural network. This method can better extract the unique characteristics of a specific target and improve recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a pedestrian re-identification method, device and electronic equipment. Background Art

[0002] In surveillance videos, high-quality facial images are often difficult to obtain due to limitations in camera resolution and shooting angle. When face recognition fails, ReID becomes a crucial alternative. A crucial feature of ReID is its ability to span cameras, making it crucial to design a method for accurately retrieving objects across multiple cameras. Summary of the Invention

[0003] The purpose of the present invention is to provide a pedestrian re-identification method, device and electronic equipment to alleviate the technical problem of low recognition accuracy existing in the prior art.

[0004] In a first aspect, a person re-identification method is provided. The method comprises:

[0005] Generate training samples through adaptive learning methods;

[0006] Training a special neural network based on the training sample to obtain a trained neural network, wherein the neural network includes an expandable convolution, and the expandable convolution is determined based on a convolution kernel and a convolution kernel offset;

[0007] Pedestrian re-identification is performed based on the trained neural network.

[0008] In some optional implementations, generating training samples by an adaptive learning method includes:

[0009] Generating initial training samples based on an adversarial neural network; wherein the adversarial neural network includes an appearance encoder, a structure encoder, a decoder, and a discriminator to distinguish between real samples and generated samples;

[0010] The initial training samples are subjected to secondary optimization filtering to obtain final training samples, and the final training samples are used to train the special neural network.

[0011] In some optional implementations, the feature extraction process based on the expandable convolution is determined based on the following formula:

[0012]

[0013] Among them, R is the eigenvalue after convolution, J is the convolution kernel, (m, n) is the convolution size, i is the matrix to be convolved, and Δp is the convolution kernel offset.

[0014] In some optional implementations, the feature extraction process based on the expandable convolution is determined based on the following formula:

[0015]

[0016] Where R is the eigenvalue after convolution, J is the convolution kernel, (m, n) is the convolution size, i is the matrix to be convolved, Δp is the convolution kernel offset, and F is the bilinear interpolation method to make the obtained eigenvalue become an integer.

[0017] In some optional implementations, a special neural network is trained based on a combined loss function and training samples, where the combined loss function is determined based on the following formula:

[0018]

[0019] where z i is the output value of the i-th node, C is the number of output nodes, that is, the number of classification categories, a is the anchor (anchor example), p is positive, and it is a sample of the same category as a; n is negative, and it is a sample of a different category; margin is a constant greater than 0.

[0020] In some optional implementations, performing person re-identification based on the trained neural network includes:

[0021] Based on the trained neural network, feature extraction is performed on the test library and retrieval images to generate feature values ​​corresponding to the feature library and retrieval images;

[0022] A feature comparison is performed on the feature values ​​corresponding to the feature library and the search image, and a special sorting method is used to output the result. The special sorting method includes feature comparison and sorting of the search target, weighted averaging of the first n feature values ​​of the sorted result to obtain the first feature value, performing a secondary sorting on the first feature value to obtain the sorted result, and performing pedestrian re-identification based on the sorted result.

[0023] In a second aspect, a pedestrian re-identification device is provided. The device includes:

[0024] A generation module is used to generate training samples through an adaptive learning method;

[0025] A training module, configured to train a special neural network based on the training samples to obtain a trained neural network, wherein the neural network includes an expandable convolution, and the expandable convolution is determined based on a convolution kernel and a convolution kernel offset;

[0026] The recognition module is used to perform pedestrian re-identification based on the trained neural network.

[0027] In some optional implementations, the generating module is specifically configured to:

[0028] Generating initial training samples based on an adversarial neural network; wherein the adversarial neural network includes an appearance encoder, a structure encoder, a decoder, and a discriminator to distinguish between real samples and generated samples;

[0029] The initial training samples are subjected to secondary optimization filtering to obtain final training samples, and the final training samples are used to train the special neural network.

[0030] In some optional implementations, the identification module is specifically configured to:

[0031] Based on the trained neural network, feature extraction is performed on the test library and retrieval images to generate feature values ​​corresponding to the feature library and retrieval images;

[0032] A feature comparison is performed on the feature values ​​corresponding to the feature library and the search image, and a special sorting method is used to output the result. The special sorting method includes feature comparison and sorting of the search target, weighted averaging of the first n feature values ​​of the sorted result to obtain the first feature value, performing a secondary sorting on the first feature value to obtain the sorted result, and performing pedestrian re-identification based on the sorted result.

[0033] The present invention provides a person re-identification method, device, and electronic device. Training samples are generated through an adaptive learning method. A specialized neural network is trained based on the training samples to obtain a trained neural network. The neural network includes scalable convolutions determined based on a convolution kernel and a convolution kernel offset. Person re-identification is then performed based on the trained neural network. This allows for better extraction of unique characteristics of specific targets, improving recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0036] Figure 2 A schematic flow chart of a pedestrian re-identification method provided by an embodiment of the present invention;

[0037] Figure 3This is an example of generating training samples in a pedestrian re-identification method provided by an embodiment of the present invention;

[0038] Figure 4 An example of a special neural network structure in a pedestrian re-identification method provided by an embodiment of the present invention;

[0039] Figure 5 A schematic structural diagram of a pedestrian re-identification device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0041] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0042] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0043] First, refer to Figure 1 An example electronic device 100 for implementing the person re-identification method according to an embodiment of the present invention is described.

[0044] like Figure 1 As shown, the electronic device 100 includes one or more processing devices 102, one or more storage devices 104, an input device 106, and an output device 108, which are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as needed.

[0045] The processing device 102 may be a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0046] The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processing device 102 may execute the program instructions to implement the client functions and / or other desired functions in the embodiments of the present invention described below (implemented by the processor). Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0047] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0048] The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0049] According to an embodiment of the present invention, an embodiment of a pedestrian re-identification method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] Figure 2 A flowchart of a pedestrian re-identification method provided by an embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0051] S210, generating training samples through an adaptive learning method.

[0052] Initial training samples can be generated based on adversarial neural networks. For example, Figure 3As shown in the figure, the adversarial neural network mainly consists of two parts: a generation unit and a discrimination unit. The generation unit is used to generate samples based on random noise, and use the generated samples to deceive the discrimination unit. If the discrimination unit cannot distinguish the authenticity, it means that the generation unit is capable enough. Among them, the adversarial neural network includes an appearance encoder, a structure encoder, a decoder, and a discriminator to distinguish between real samples and generated samples. In addition, the discrimination unit can be embedded in the generation unit by sharing the appearance generator. It is the main structure of pedestrian re-identification. To put it bluntly, it is embedded in the generation module. The pedestrian re-identification module can better identify the authenticity of the generated samples.

[0053] In order to make the generated samples more in line with actual scenarios and more generalizable, the initial training samples can be subjected to secondary optimization filtering to obtain the final training samples, which are used to train a special neural network. This method has a fast processing speed and accurate processing results. It can be completed by computer, greatly reducing the amount of manual labeling. This method mainly combines different model algorithms for different targets to perform secondary filtering on samples. Specifically: First, the model algorithm corresponding to the target is used to extract features, sort, and cluster the samples. Only one of the two particularly similar pictures is retained, and two pictures with relatively large differences are divided into two groups. In order to make the secondary filtering more accurate, this method can prevent the model from learning some useless information and also avoid learning some bad information. Improve the accuracy of the model and the speed of training.

[0054] S220: Training a special neural network based on the training sample to obtain a trained neural network, where the neural network includes an expandable convolution, and the expandable convolution is determined based on a convolution kernel and a convolution kernel offset.

[0055] Among them, the special neural network can be an improved RepVGG. RepVGG has the following main features: (1) no branch structure; (2) only uses 3×3 convolution; (3) only uses ReLU as the activation function. In short, the model structure is to stack more than 20 3x3 convolution layers, divided into 5 stages, the first layer of each stage is a stride=2 downsampling, and each convolution layer uses ReLU as the activation function. Why choose RepVGG as the baseline? Because it has the following advantages: (1) 3x3 convolution is very fast. On the GPU, the computational density of 3x3 convolution (theoretical computational volume divided by the time taken) can reach four times that of 1x1 and 5x5 convolution. (2) The single-channel architecture is very fast because of its high parallelism. For the same amount of computation, the efficiency of "big and complete" operations far exceeds that of "small and fragmented" operations. (3) The single-channel architecture saves memory. For example, although the shortcut of ResNet does not account for the computational volume, it doubles the memory usage. (4) The single-channel architecture is more flexible and can easily change the width of each layer (such as pruning). (5) The main part of RepVGG has only one operator: 3x3 convolution followed by ReLU. When designing a dedicated chip, given the chip size or cost, we can integrate a large number of 3x3 convolution-ReLU computing units to achieve high efficiency. The improved RepVGG network structure is as follows Figure 4 shown.

[0056] The previous RepVGG algorithm for pedestrian feature extraction may not accurately extract detailed features of pedestrians, but may only extract obvious features such as the color of pedestrians' clothing. In order to extract deeper and broader features that are unique to the target identity, we changed the convolutional layers in RepVGG to expandable convolutions.

[0057] Ordinary convolution mainly consists of two steps. First, a fixed-size convolution kernel is used to perform sliding sampling on the feature map. Then, the sampling points are multiplied by weights and summed. The traditional convolution process is as follows:

[0058]

[0059] Where R is the eigenvalue after convolution, J is the convolution kernel, m, n are the convolution sizes, and I is the matrix to be convolved. In order to better extract detailed features, the present invention expands the convolution kernel on the original basis and adds an offset to the convolution kernel to improve the feature extraction capability. The specific formula is as follows:

[0060]

[0061] Among them, R is the eigenvalue after convolution, J is the convolution kernel, m, n is the convolution size, and I is the matrix to be convolved. Δp is the convolution kernel offset, and ΔP in the above formula is nSince it is not an integer, bilinear interpolation is used (find the 4 pixels with the closest coordinates and calculate the value of the coordinates). The specific formula is as follows:

[0062] R(i, j)=F*(∑ m ∑ n I(i+m,j+n)J(m+Δp,n+Δp)) Formula (3)

[0063] Where F is a bilinear interpolation method, which makes the obtained eigenvalues ​​become integers.

[0064] It can be known from the comparative documents that the deformable process formula is also known.

[0065] At the same time, in order to make the model better distinguish between information between groups and within groups, a combined loss function is used during training. The specific formula is as follows:

[0066]

[0067] where z i is the output value of the i-th node, C is the number of output nodes (i.e., the number of categories), a is the anchor, p is a positive sample belonging to the same category as a, n is a negative sample belonging to a different category, and margin is a constant greater than 0. The ultimate optimization goal is to reduce the distance between a and p and increase the distance between a and n. λ and ω are parameters that control the weight of the loss.

[0068] This loss function can be used to better adapt the loss function according to the amount of data and sample difficulty, so that the training process can converge better, and the final result can better narrow the distance within the same group and widen the distance between different groups, thereby improving the accuracy of subsequent retrieval and sorting.

[0069] S230: Perform person re-identification based on the trained neural network.

[0070] Based on the trained neural network, feature extraction can be performed on the test library and the retrieval image to generate feature values ​​corresponding to the feature library and the retrieval image; then, feature comparison is performed on the feature values ​​corresponding to the feature library and the retrieval image, and the results are output using a special sorting method. The special sorting method includes feature comparison and sorting of the retrieval target, weighted averaging of the first n feature values ​​of the sorting result to obtain the first feature value, secondary sorting of the first feature value to obtain the sorting result, and pedestrian re-identification based on the sorting result.

[0071] In an embodiment of the present invention, a training sample can be generated by an adaptive learning method; the training sample is optimized by a special secondary training sample optimization method; the trained training sample is placed in a special neural network for training, and the model weight is learned and adjusted; the test library (Base) and the retrieval image (Query) are sent to the model weight for feature extraction, and the feature values ​​corresponding to the feature library and the retrieval image are generated; the feature values ​​corresponding to the feature library and the retrieval image are compared, and the results are output using a special sorting method. In this way, by generating samples, secondary optimization filtering of samples, model training, feature extraction, model comparison, and re-sorting output results, the unique characteristics of a specific target can be better extracted, making the sorting results more accurate.

[0072] Figure 5 This is a schematic diagram of the structure of a pedestrian re-identification device provided by an embodiment of the present invention. Figure 5 As shown, the device includes:

[0073] A generating module 501 is used to generate training samples by an adaptive learning method;

[0074] A training module 502 is configured to train a special neural network based on the training samples to obtain a trained neural network, wherein the neural network includes a scalable convolution, and the scalable convolution is determined based on a convolution kernel and a convolution kernel offset;

[0075] The recognition module 503 is used to perform pedestrian re-identification based on the trained neural network.

[0076] In some embodiments, the generating module 501 is specifically configured to:

[0077] Generate initial training samples based on an adversarial neural network; the adversarial neural network includes an appearance encoder, a structure encoder, a decoder, and a discriminator to distinguish between real samples and generated samples;

[0078] The initial training samples are subjected to secondary optimization filtering to obtain the final training samples, which are used to train a special neural network.

[0079] In some embodiments, the feature extraction process based on expandable convolution is determined based on the above formula (II).

[0080] In some embodiments, the feature extraction process based on expandable convolution is determined based on the above formula (III).

[0081] In some embodiments, a special neural network is trained based on a combined loss function and training samples, and the combined loss function is determined based on the above formula (IV).

[0082] In some embodiments, the identification module 503 is specifically configured to:

[0083] Based on the trained neural network, feature extraction is performed on the test library and retrieval images to generate feature values ​​corresponding to the feature library and retrieval images;

[0084] Perform feature comparison on the feature values ​​corresponding to the feature library and the search image, and use a special sorting method to output the results. The special sorting method includes feature comparison and sorting of the search target, weighted averaging of the first n feature values ​​of the sorted results to obtain the first feature value, performing secondary sorting on the first feature value to obtain the sorted result, and performing pedestrian re-identification based on the sorted result.

[0085] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0086] Furthermore, this embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method provided in the aforementioned method embodiment are executed.

[0087] The computer program products of the pedestrian re-identification method, device and system provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.

[0088] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0090] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0091] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0092] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0093] Finally, it should be noted that the above embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field should understand that any technician familiar with this technical field can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention.

Claims

1. A pedestrian re-identification method, characterized in that: include: Generate training samples through adaptive learning methods; Training a special neural network based on the training samples to obtain a trained neural network, wherein the neural network includes multiple layers of scalable convolutions, wherein the scalable convolutions are determined based on a convolution kernel and a convolution kernel offset; the special neural network is an improved RepVGG; Perform person re-identification based on the trained neural network; The feature extraction process based on expandable convolution is determined based on the following formula: Where R is the eigenvalue after convolution, J is the convolution kernel, (m, n) is the convolution size, I is the matrix to be convolved, and Δp is the convolution kernel offset; The special neural network is trained based on a combined loss function and training samples, and the combined loss function is determined based on the following formula: where z i is the output value of the i-th node, C is the number of output nodes, that is, the number of classification categories, a is the anchor example, p is positive, and it is a sample of the same category as a; n is negative, and it is a sample of a different category; margin is a constant greater than 0.

2. The method according to claim 1, characterized in that The method of generating training samples by adaptive learning includes: Generating initial training samples based on an adversarial neural network; wherein the adversarial neural network includes an appearance encoder, a structure encoder, a decoder, and a discriminator to distinguish between real samples and generated samples; The initial training samples are subjected to secondary optimization filtering to obtain final training samples, and the final training samples are used to train the special neural network.

3. The method according to claim 1, characterized in that The feature extraction process based on expandable convolution is determined based on the following formula: Where R is the eigenvalue after convolution, J is the convolution kernel, (m, n) is the convolution size, I is the matrix to be convolved, Δp is the convolution kernel offset, and F is the bilinear interpolation method to make the obtained eigenvalue become an integer.

4. The method according to claim 1, wherein The pedestrian re-identification based on the trained neural network includes: Based on the trained neural network, feature extraction is performed on the test library and retrieval images to generate feature values ​​corresponding to the feature library and retrieval images; A feature comparison is performed on the feature values ​​corresponding to the feature library and the search image, and a special sorting method is used to output the result. The special sorting method includes feature comparison and sorting of the search target, weighted averaging of the first n feature values ​​of the sorted result to obtain the first feature value, performing a secondary sorting on the first feature value to obtain the sorted result, and performing pedestrian re-identification based on the sorted result.

5. A pedestrian re-identification device, characterized in that: include: A generation module is used to generate training samples through an adaptive learning method; A training module is configured to train a special neural network based on the training samples to obtain a trained neural network, wherein the neural network includes multiple layers of scalable convolutions, wherein the scalable convolutions are determined based on convolution kernels and convolution kernel offsets; the special neural network is a modified RepVGG. Recognition module, used for pedestrian re-identification based on the trained neural network; The feature extraction process based on expandable convolution is determined based on the following formula: Where R is the eigenvalue after convolution, J is the convolution kernel, (m, n) is the convolution size, I is the matrix to be convolved, and Δp is the convolution kernel offset; The special neural network is trained based on a combined loss function and training samples, and the combined loss function is determined based on the following formula: where z i is the output value of the i-th node, C is the number of output nodes, that is, the number of classification categories, a is the anchor example, p is positive, and it is a sample of the same category as a; n is negative, and it is a sample of a different category; margin is a constant greater than 0.

6. The device according to claim 5, characterized in that The generation module is specifically used for: Generating initial training samples based on an adversarial neural network; wherein the adversarial neural network includes an appearance encoder, a structure encoder, a decoder, and a discriminator to distinguish between real samples and generated samples; The initial training samples are subjected to secondary optimization filtering to obtain final training samples, and the final training samples are used to train the special neural network.

7. An electronic device, characterized in that: include: processing equipment and storage devices; The storage device stores a computer program, which, when executed by the processing device, executes the method according to any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are executed.

Images