Pedestrian search methods, devices, media, and computing equipment based on computer vision

Abstract

This invention provides a method, apparatus, medium, and computing device for pedestrian search based on computer vision. The method includes: determining a spatiotemporal pipeline from multiple consecutive frames of acquired images; wherein the spatiotemporal pipeline contains pedestrians to be identified in the consecutive frames; inputting the spatiotemporal pipeline into a pedestrian search model to obtain a pedestrian feature map of the pedestrians to be identified; and determining target pedestrian information corresponding to the pedestrian feature map. This invention can accurately identify specific pedestrians from images with missing features, improving the accuracy of pedestrian recognition.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer vision technology, and more specifically, the embodiments of the present invention relate to a pedestrian search method, apparatus, medium and computing device based on computer vision. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention as set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Currently, surveillance cameras are installed in densely populated public places, government departments, enterprises, residential communities, and even many residents' homes, providing reliable video surveillance resources for maintaining social order and protecting people's lives and property. However, in video surveillance, due to significant variations in camera parameters such as resolution and shooting angle, it is difficult to achieve stable acquisition of high-quality facial images, resulting in poor stability of target tracking based on facial recognition technology. In contrast, pedestrian search algorithms can provide a more robust target tracking solution for video surveillance.

[0004] Pedestrian search algorithms typically refer to detecting and identifying a pedestrian from an image database given an image containing that pedestrian. However, in practice, it has been found that current pedestrian search algorithms are primarily image-based. If a pedestrian in an image is partially occluded, then the pedestrian's feature information is missing, and the algorithm cannot accurately identify the specific pedestrian from this image with missing features, thus reducing the accuracy of pedestrian recognition. Summary of the Invention

[0005] In this context, embodiments of the present invention aim to provide a pedestrian search method, apparatus, medium, and computing device based on computer vision.

[0006] In a first aspect of the present invention, a pedestrian search method based on computer vision is provided, comprising:

[0007] A spatiotemporal pipeline is determined from multiple consecutive frames of acquired images; wherein, the spatiotemporal pipeline includes the pedestrians to be identified in the consecutive frames of images;

[0008] The spatiotemporal pipeline is input into the pedestrian search model to obtain the pedestrian feature map of the pedestrian to be identified;

[0009] Determine the target pedestrian information corresponding to the pedestrian feature map.

[0010] In one embodiment of this implementation, the step of inputting the spatiotemporal pipeline into the pedestrian search model to obtain the pedestrian feature map of the pedestrian to be identified includes:

[0011] The spatiotemporal pipeline is input into the pedestrian search model for feature extraction to obtain pedestrian features;

[0012] The pedestrian features are fused to obtain an initial feature map;

[0013] The initial feature map is subjected to classification and regression prediction to obtain the pedestrian feature map.

[0014] In one embodiment of this implementation, the step of inputting the spatiotemporal pipeline into a pedestrian search model for feature extraction to obtain pedestrian features includes:

[0015] The spatiotemporal pipeline is input into the pedestrian search model for convolution to obtain convolutional features;

[0016] The convolutional features are pooled to obtain pooled features;

[0017] The pooling features are processed by multiple residual block groups to obtain pedestrian features; wherein each residual block group contains multiple residual blocks, and the pedestrian features include a first pedestrian feature map, a second pedestrian feature map, and a third pedestrian feature map, wherein any two pedestrian feature maps in the first pedestrian feature map, the second pedestrian feature map, and the third pedestrian feature map have different dimensions.

[0018] In one embodiment of this implementation, the plurality of residual block groups includes at least a first residual block group, a second residual block group, a third residual block group, and a fourth residual block group. The process of processing the pooling features using the plurality of residual block groups to obtain pedestrian features includes:

[0019] The pooling features are processed by the first residual block group to obtain residual features;

[0020] The residual features are processed by the second residual block group to obtain the first pedestrian feature map;

[0021] The first pedestrian feature map is processed by the third residual block group to obtain the second pedestrian feature map;

[0022] The residual features are processed by the fourth residual block group to obtain the third pedestrian feature map;

[0023] The first pedestrian feature map, the second pedestrian feature map, and the third pedestrian feature map are identified as pedestrian features.

[0024] In one embodiment of this implementation, fusing the pedestrian features to obtain an initial feature map includes:

[0025] The second pedestrian feature map and the third pedestrian feature map are fused to obtain a first fused feature map;

[0026] The first pedestrian feature map and the first fused feature map are fused to obtain the second fused feature map;

[0027] The second fused feature map is convolved by a variable 3D convolutional layer to obtain an initial feature map.

[0028] In one embodiment of this implementation, fusing the second pedestrian feature map and the third pedestrian feature map to obtain a first fused feature map includes:

[0029] Perform a variable 3D convolution on the second pedestrian feature map to obtain the first sub-feature map;

[0030] The third pedestrian feature map is upsampled to obtain the second sub-feature map;

[0031] The first sub-feature map and the second sub-feature map are fused to obtain the first fused feature map;

[0032] And, the step of fusing the first pedestrian feature map and the first fused feature map to obtain the second fused feature map includes:

[0033] Perform variable 3D convolution on the first pedestrian feature map to obtain the third sub-feature map;

[0034] Upsample the first fused feature map to obtain the fourth sub-feature map;

[0035] The third sub-feature map and the fourth sub-feature map are fused to obtain the second fused feature map.

[0036] In one embodiment of this implementation, the step of performing classification regression prediction on the initial feature map to obtain a pedestrian feature map includes:

[0037] Pedestrian detection is performed on the initial feature map to obtain positive pedestrian sample features;

[0038] Pedestrian re-identification is performed on the positive pedestrian sample features to obtain a pedestrian feature map.

[0039] In a second aspect of the present invention, a pedestrian search device based on computer vision is provided, comprising:

[0040] The first determining unit is configured to determine a spatiotemporal pipeline from multiple consecutive frame images acquired; wherein the spatiotemporal pipeline includes the pedestrian to be identified in the consecutive frame images;

[0041] The input unit is used to input the spatiotemporal pipeline into the pedestrian search model to obtain the pedestrian feature map of the pedestrian to be identified.

[0042] The second determining unit is used to determine the target pedestrian information corresponding to the pedestrian feature map.

[0043] In a third aspect of the present invention, a computing device is provided, the computing device comprising: at least one processor, a memory, and an input / output unit; wherein the memory is used to store a computer program, and the processor is used to invoke the computer program stored in the memory to execute the method described in any one aspect.

[0044] In a fourth aspect of the present invention, a computer-readable storage medium is provided, comprising instructions which, when executed on a computer, cause the computer to perform the method described in any one of the first aspects.

[0045] According to embodiments of the present invention, a computer vision-based pedestrian search method, apparatus, medium, and computing device can determine a spatiotemporal pipeline from multiple consecutive frames of acquired images; wherein the spatiotemporal pipeline includes pedestrians to be identified in the consecutive frames of images; the spatiotemporal pipeline is input into a pedestrian search model to obtain a pedestrian feature map of the pedestrians to be identified; and the target pedestrian information corresponding to the pedestrian feature map is determined, thereby enabling accurate identification of specific pedestrians through images with missing features, and improving the accuracy of pedestrian identification. Attached Figure Description

[0046] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0047] Figure 1 This is a flowchart illustrating a pedestrian search method based on computer vision provided in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the structure of a pedestrian search device based on computer vision according to an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the structure of a medium according to an embodiment of the present invention is shown.

[0050] Figure 4 A schematic diagram of the structure of a computing device according to an embodiment of the present invention is shown.

[0051] Figure 5a A schematic diagram of a target detection box according to an embodiment of the present invention is shown;

[0052] Figure 5b A schematic diagram of a spatiotemporal pipeline according to an embodiment of the present invention is shown.

[0053] Figure 5c The diagram illustrates a variable 3D convolutional fusion of the spatiotemporal features of a moving target according to an embodiment of the present invention.

[0054] Figure 6 A schematic diagram of another spatiotemporal pipeline according to an embodiment of the present invention is shown.

[0055] Figure 7 A schematic diagram of the structure of a pedestrian search model according to an embodiment of the present invention is shown.

[0056] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0057] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0058] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0059] According to embodiments of the present invention, a pedestrian search method, apparatus, medium, and computing device based on computer vision are proposed.

[0060] It should be noted that the number of any elements in the accompanying drawings is for illustrative purposes only and not as a limitation, and any naming is for distinction only and has no limiting meaning.

[0061] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0062] Exemplary methods

[0063] The following is for reference. Figure 1 , Figure 1 This is a flowchart illustrating a computer vision-based pedestrian search method according to an embodiment of the present invention. It should be noted that the embodiments of the present invention can be applied to any applicable scenario.

[0064] Figure 1The flowchart of a pedestrian search method based on computer vision provided in an embodiment of the present invention, shown below, includes:

[0065] Step S101: Determine the spatiotemporal pipeline from the acquired multi-frame continuous image.

[0066] In this embodiment of the invention, the spatiotemporal pipeline includes pedestrians to be identified in the consecutive frame images.

[0067] For example, the method for determining a spacetime pipeline can be: Figure 5a , Figure 5b as well as Figure 5c Each of the examples lists consecutive frames of the same pedestrian during their movement, namely t-1, t, and t+1.

[0068] Figure 5a The image represents a traditional object detection bounding box (BBox). It can be seen that occlusion causes the loss of local pedestrian features within the detection box.

[0069] Figure 5b The proposed spatiotemporal pipeline STtube is used to detect motion pipelines formed by moving pedestrians. It captures the spatiotemporal features of pedestrians and avoids the difficulties in pedestrian detection and feature extraction caused by occlusion in single-frame cases.

[0070] Figure 5c This indicates that the spatiotemporal features of a moving target are fused using variable 3D convolution. It can be seen that this patented technology ignores features lost due to local occlusion within the pipe, instead focusing more on unoccluded features and performing spatiotemporal fusion.

[0071] For bounding boxes (BBoxes), if a pedestrian is occluded, the model can only extract detection features and ReID features from the unoccluded portion to locate and identify the pedestrian. This often results in poor performance for occluded pedestrians.

[0072] However, STtube can utilize information from all consecutive frames of the sliding window to compensate for the occlusion of pedestrians. This allows for a more comprehensive extraction of detection and ReID features, significantly improving the model's performance in detecting occluded pedestrians. The components of STtube will be introduced below.

[0073] Please refer to the following for details. Figure 6An STtube is a decahedron composed of three bounding boxes (BBoxes) from different video frames: Bs, Bm, and Be, defined using 12 coordinates. The other three values ​​indicate their temporal positions. By interpolating between (Bs, Bm) and (Bm, Be), all bounding boxes {Bs, Bs+1, ..., Bm, ..., Be-1, Be} that constitute the STtube can be reconstructed. STtubes capture spatiotemporal information by breaking down video into short slices that capture simple linear trajectories. We treat each BBox on the trajectory as a Bm of an STtube and then look for its corresponding Bs and Be. A longer STtube can capture more spatiotemporal information, but it cannot handle complex motion trajectories. To balance this, we set a condition that constrains the length of the STtube: the average intersection-over-union (IoU) between the interpolated box B* and the ground truth box B* is not less than a threshold η.

[0074] max es

[0075] st

[0076] i∈{s,s+1,...,m,...,e}

[0077] This condition will result in STtubes of varying lengths; longer STtubes can capture more spatiotemporal information, while shorter STtubes are suitable for short-term motion changes. In this model, a pipe is generated for each grid point, so the pipe can be mapped back to the original sliding window to obtain the corresponding spatiotemporal coordinates based on that grid point. For example... Figure 6 As shown, the pipeline only needs to predict u again. m b m l m 、r m d s 、u s b s l s 、r s d e 、u e b e l e 、r e These fourteen parameters.

[0078] Step S102: Input the spatiotemporal pipeline into the pedestrian search model to obtain the pedestrian feature map of the pedestrian to be identified.

[0079] Please refer to the following: Figure 7 , Figure 7 This is a schematic diagram of the pedestrian search model, which can be divided into three parts: Backbone, Spatiotemporal Feature Alignment and Aggregation (SFAA), and Head. Detailed descriptions are as follows:

[0080] Backbone uses 3DResNet50, which is mainly used to extract pedestrian features;

[0081] The SFAA structure is mainly used for the alignment and fusion of spatiotemporal features at different levels to obtain an initial feature map;

[0082] Heads are divided into Detection Heads and ReID Heads.

[0083] The Detection Head uses four convolutional layers to perform classification and regression prediction on the extracted features, which is used to predict pipeline generation.

[0084] ReID Head uses TOIM Loss to directly constrain the network.

[0085] As an optional implementation, step S102, which inputs the spatiotemporal pipeline into the pedestrian search model to obtain the pedestrian feature map of the pedestrian to be identified, can specifically be as follows:

[0086] The spatiotemporal pipeline is input into the pedestrian search model for feature extraction to obtain pedestrian features;

[0087] The pedestrian features are fused to obtain an initial feature map;

[0088] The initial feature map is subjected to classification and regression prediction to obtain the pedestrian feature map.

[0089] In this embodiment of the invention, the spatiotemporal pipeline is input into the pedestrian search model for feature extraction, and the specific method for obtaining pedestrian features can be as follows:

[0090] The spatiotemporal pipeline is input into the pedestrian search model for convolution to obtain convolutional features;

[0091] The convolutional features are pooled to obtain pooled features;

[0092] The pooling features are processed by multiple residual block groups to obtain pedestrian features; wherein each residual block group contains multiple residual blocks, and the pedestrian features include a first pedestrian feature map, a second pedestrian feature map, and a third pedestrian feature map, wherein any two pedestrian feature maps in the first pedestrian feature map, the second pedestrian feature map, and the third pedestrian feature map have different dimensions.

[0093] Optionally, the multiple residual block groups include at least a first residual block group, a second residual block group, a third residual block group, and a fourth residual block group. The process of processing the pooling features using multiple residual block groups to obtain pedestrian features includes:

[0094] The pooling features are processed by the first residual block group to obtain residual features;

[0095] The residual features are processed by the second residual block group to obtain the first pedestrian feature map;

[0096] The first pedestrian feature map is processed by the third residual block group to obtain the second pedestrian feature map;

[0097] The residual features are processed by the fourth residual block group to obtain the third pedestrian feature map;

[0098] The first pedestrian feature map, the second pedestrian feature map, and the third pedestrian feature map are identified as pedestrian features.

[0099] In this embodiment of the invention, the Backbone uses 3DResNet50 as the base network, which includes a 7*7*7 kernel convolutional layer (Conv1) and four modules named res2, res3, res4, and res5. Each of these four modules contains 3, 4, 6, and 3 residual blocks, respectively. The network structure is shown in Table 1. Our input samples have the following dimensional shape: N*3*8*800*1000, where N represents the batch size, 8 represents the frame sequence length, 3 represents the r, g, and b channels of the color image, and 800*1000 represents the input image size. After the samples pass through the Backbone, the output Res3, Res4, and Res5 feature maps have the following dimensions: N*512*4*100*125, N*1024*2*50*62, and N*2048*1*25*31, respectively.

[0100] Table 1. Network Structure of Feature Extraction Unit

[0101]

[0102]

[0103] As an optional implementation, the method for fusing the pedestrian features to obtain the initial feature map can be as follows:

[0104] The second pedestrian feature map and the third pedestrian feature map are fused to obtain the first fused feature map;

[0105] The first pedestrian feature map and the first fused feature map are fused to obtain the second fused feature map;

[0106] The second fused feature map is convolved by a variable 3D convolutional layer to obtain an initial feature map.

[0107] Optionally, the method for fusing the second pedestrian feature map and the third pedestrian feature map to obtain the first fused feature map can be as follows:

[0108] Perform a variable 3D convolution on the second pedestrian feature map to obtain the first sub-feature map;

[0109] The third pedestrian feature map is upsampled to obtain the second sub-feature map;

[0110] The first sub-feature map and the second sub-feature map are fused to obtain the first fused feature map;

[0111] Furthermore, the method of fusing the first pedestrian feature map and the first fused feature map to obtain the second fused feature map can specifically be as follows:

[0112] Perform variable 3D convolution on the first pedestrian feature map to obtain the third sub-feature map;

[0113] Upsample the first fused feature map to obtain the fourth sub-feature map;

[0114] The third sub-feature map and the fourth sub-feature map are fused to obtain the second fused feature map.

[0115] In this embodiment of the invention, SFAA is a spatiotemporal feature alignment and fusion module. It aligns the upper-layer feature map with regions and upsamples it through variable convolution, and then fuses it with the current-layer feature map through a concatenation operation to obtain richer feature representation.

[0116] This module performs spatiotemporal feature alignment and fusion on the feature maps D3, D4, and D5 output from Res3, Res4, and Res5 in Backbone. Specifically, feature map D5 and feature map D4 are fused to obtain feature map S4. Then, feature map S4 is fused with feature map D3 to obtain feature map S3. After that, a variable 3D convolution is performed to reduce the channel dimension of feature map S3 from 768 back to 256 to obtain the output feature. The output feature is then used as the output of the SFAA module.

[0117] Fusion specifically includes upsampling, 3D deformable convolution, and concat operations. For details on the connection structure, please refer to [link / reference needed]. Figure 7The fusion operation in [the context of the text].

[0118] Ultimately, the detection and ReID tasks will be trained based on this feature map. The SFAA module aligns and fuses feature maps of different scales, resulting in a feature map that aligns with both detection and re-identification tasks while incorporating the spatiotemporal features of targets at different scales. This effectively addresses the impact of occlusion interference on pedestrian search and improves the accuracy of occluded pedestrian search.

[0119] As an optional implementation, the method of performing classification regression prediction on the initial feature map to obtain the pedestrian feature map can be as follows:

[0120] Pedestrian detection is performed on the initial feature map to obtain positive pedestrian sample features;

[0121] Pedestrian re-identification is performed on the positive pedestrian sample features to obtain a pedestrian feature map.

[0122] In this embodiment of the invention, the SFAA module is connected to both the Detection Head and the ReID Head. This paper adopts the ReID first principle, directly using the feature map output by the SFAA as the ReID feature of the pipeline, and directly constraining it through TOIMLoss.

[0123] The Detection Head is trained in two branches based on the feature map output by SFAA. One branch trains class scores and centrality, while the other predicts the pipeline. Each branch undergoes four standard convolutional operations to obtain its own feature map, which is used for different tasks. Since pedestrian detection and pedestrian re-identification are performed simultaneously and the resulting feature maps are of the same size except for the channel dimension, positive sample grid points identified as pedestrians are obtained based on the class score. These grid points are then matched one-to-one with the prediction pipeline branch and the ReID feature branch, thus achieving task alignment. The corresponding loss calculation determines whether a pipeline is a positive sample based on its class score and uses focal loss for constraint. Other loss functions calculate their respective losses based on positive samples, ignoring negative samples and the background.

[0124] Step S103: Determine the target pedestrian information corresponding to the pedestrian feature map.

[0125] This invention can accurately identify specific pedestrians from images with missing features, thereby improving the accuracy of pedestrian identification.

[0126] Exemplary device

[0127] After introducing the method of exemplary embodiments of the present invention, the following references are made. Figure 2An exemplary embodiment of the present invention provides a computer vision-based pedestrian search device, the device comprising:

[0128] The first determining unit 201 is used to determine a spatiotemporal pipeline from the acquired multi-frame continuous image; wherein the spatiotemporal pipeline includes the pedestrian to be identified in the continuous frame image;

[0129] Input unit 202 is used to input the spatiotemporal pipeline determined by the first determining unit 201 into the pedestrian search model to obtain the pedestrian feature map of the pedestrian to be identified;

[0130] The second determining unit 203 is used to determine the target pedestrian information corresponding to the pedestrian feature map obtained by the input unit 202.

[0131] As an optional implementation, the input unit 202 inputs the spatiotemporal pipeline into the pedestrian search model to obtain the pedestrian feature map of the pedestrian to be identified in the following specific ways:

[0132] The spatiotemporal pipeline is input into the pedestrian search model for feature extraction to obtain pedestrian features;

[0133] The pedestrian features are fused to obtain an initial feature map;

[0134] The initial feature map is subjected to classification and regression prediction to obtain the pedestrian feature map.

[0135] As an optional implementation, the input unit 202 inputs the spatiotemporal pipeline into the pedestrian search model for feature extraction. Specifically, the method for obtaining pedestrian features can be as follows:

[0136] The spatiotemporal pipeline is input into the pedestrian search model for convolution to obtain convolutional features;

[0137] The convolutional features are pooled to obtain pooled features;

[0138] The pooling features are processed by multiple residual block groups to obtain pedestrian features; wherein each residual block group contains multiple residual blocks, and the pedestrian features include a first pedestrian feature map, a second pedestrian feature map, and a third pedestrian feature map, wherein any two pedestrian feature maps in the first pedestrian feature map, the second pedestrian feature map, and the third pedestrian feature map have different dimensions.

[0139] As an optional implementation, the multiple residual block groups include at least a first residual block group, a second residual block group, a third residual block group, and a fourth residual block group. The input unit 202 processes the pooling features through the multiple residual block groups to obtain pedestrian features in the following specific ways:

[0140] The pooling features are processed by the first residual block group to obtain residual features;

[0141] The residual features are processed by the second residual block group to obtain the first pedestrian feature map;

[0142] The first pedestrian feature map is processed by the third residual block group to obtain the second pedestrian feature map;

[0143] The residual features are processed by the fourth residual block group to obtain the third pedestrian feature map;

[0144] The first pedestrian feature map, the second pedestrian feature map, and the third pedestrian feature map are identified as pedestrian features.

[0145] As an optional implementation, the input unit 202 fuses the pedestrian features to obtain an initial feature map in the following specific ways:

[0146] The second pedestrian feature map and the third pedestrian feature map are fused to obtain the first fused feature map;

[0147] The first pedestrian feature map and the first fused feature map are fused to obtain the second fused feature map;

[0148] The second fused feature map is convolved by a variable 3D convolutional layer to obtain an initial feature map.

[0149] As an optional implementation, the input unit 202 fuses the second pedestrian feature map and the third pedestrian feature map to obtain the first fused feature map in the following specific ways:

[0150] Perform a variable 3D convolution on the second pedestrian feature map to obtain the first sub-feature map;

[0151] The third pedestrian feature map is upsampled to obtain the second sub-feature map;

[0152] The first sub-feature map and the second sub-feature map are fused to obtain the first fused feature map;

[0153] Furthermore, the input unit 202 fuses the first pedestrian feature map and the first fused feature map to obtain the second fused feature map in the following specific ways:

[0154] Perform variable 3D convolution on the first pedestrian feature map to obtain the third sub-feature map;

[0155] Upsample the first fused feature map to obtain the fourth sub-feature map;

[0156] The third sub-feature map and the fourth sub-feature map are fused to obtain the second fused feature map.

[0157] As an optional implementation, the input unit 202 performs classification regression prediction on the initial feature map to obtain the pedestrian feature map in the following specific ways:

[0158] Pedestrian detection is performed on the initial feature map to obtain positive pedestrian sample features;

[0159] Pedestrian re-identification is performed on the positive pedestrian sample features to obtain a pedestrian feature map.

[0160] Exemplary media

[0161] After introducing the methods and apparatus of exemplary embodiments of the present invention, the following references are made. Figure 3 A computer-readable storage medium according to exemplary embodiments of the present invention will be described, please refer to... Figure 3 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it implements the steps described in the above method implementation, such as determining a spatiotemporal pipeline from multiple consecutive frame images acquired; wherein the spatiotemporal pipeline contains the pedestrian to be identified in the consecutive frame images; inputting the spatiotemporal pipeline into a pedestrian search model to obtain a pedestrian feature map of the pedestrian to be identified; and determining the target pedestrian information corresponding to the pedestrian feature map. The specific implementation of each step will not be repeated here.

[0162] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0163] Exemplary computing device

[0164] After introducing the methods, apparatus, and media of exemplary embodiments of the present invention, the following references are made. Figure 4 A computing device for pedestrian search based on computer vision according to an exemplary embodiment of the present invention.

[0165] Figure 4 A block diagram is shown of an exemplary computing device 40 suitable for implementing embodiments of the present invention. The computing device 40 may be a computer system or a server. Figure 4The computing device 40 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0166] like Figure 4 As shown, the components of computing device 40 may include, but are not limited to: one or more processors or processing units 401, system memory 402, and bus 403 connecting different system components (including system memory 402 and processing unit 401).

[0167] The computing device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device 40, including volatile and non-volatile media, and removable and non-removable media.

[0168] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 4021 and / or cache memory 4022. Computing device 40 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, ROM 4023 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 4 Not shown in the image (usually referred to as a "hard drive"). Although not shown in Figure 4 The diagram illustrates that disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disc drives for reading and writing to removable non-volatile optical discs (e.g., CD-ROMs, DVD-ROMs, or other optical media) can be provided. In these cases, each drive can be connected to bus 403 via one or more data media interfaces. System memory 402 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0169] A program / utility 4025 having a set (at least one) of program modules 4024 may be stored, for example, in system memory 402, and such program modules 4024 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment. Program modules 4024 typically perform the functions and / or methods described in the embodiments of the present invention.

[0170] The computing device 40 can also communicate with one or more external devices 404 (such as a keyboard, pointing device, display, etc.). This communication can be performed via the input / output (I / O) interface 405. Furthermore, the computing device 40 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 406. Figure 4 As shown, network adapter 406 communicates with other modules of computing device 40 (such as processing unit 401) via bus 403. It should be understood that, although... Figure 4 As not shown, it can be used in conjunction with computing device 40 with other hardware and / or software modules.

[0171] The processing unit 401 executes various functional applications and data processing by running programs stored in the system memory 402. For example, it determines a spatiotemporal pipeline from multiple consecutive frame images acquired from the dataset; wherein the spatiotemporal pipeline contains pedestrians to be identified in the consecutive frame images; it inputs the spatiotemporal pipeline into a pedestrian search model to obtain a pedestrian feature map of the pedestrian to be identified; and it determines the target pedestrian information corresponding to the pedestrian feature map. The specific implementation methods of each step will not be repeated here. It should be noted that although several units / modules or sub-units / sub-modules of a computer vision-based pedestrian search device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0172] In the description of this invention, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0173] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0174] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0175] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0176] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0177] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0178] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0179] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

Claims

1. A pedestrian search method based on computer vision, comprising: A spatiotemporal pipeline is determined from multiple consecutive frames of acquired images; wherein, the spatiotemporal pipeline includes the pedestrians to be identified in the consecutive frames of images; The spatiotemporal pipeline is input into the pedestrian search model to obtain the pedestrian feature map of the pedestrian to be identified; Determine the target pedestrian information corresponding to the pedestrian feature map; The step of inputting the spatiotemporal pipeline into the pedestrian search model to obtain the pedestrian feature map of the pedestrian to be identified includes: The spatiotemporal pipeline is input into the pedestrian search model for feature extraction to obtain pedestrian features; The pedestrian features are fused to obtain an initial feature map; The initial feature map is subjected to classification and regression prediction to obtain the pedestrian feature map; The step of inputting the spatiotemporal pipeline into the pedestrian search model for feature extraction to obtain pedestrian features includes: The spatiotemporal pipeline is input into the pedestrian search model for convolution to obtain convolutional features; The convolutional features are pooled to obtain pooled features; The pooling features are processed by multiple residual block groups to obtain pedestrian features; wherein each residual block group contains multiple residual blocks, and the pedestrian features include a first pedestrian feature map, a second pedestrian feature map, and a third pedestrian feature map, and any two pedestrian feature maps in the first pedestrian feature map, the second pedestrian feature map, and the third pedestrian feature map have different dimensions; The step of fusing the pedestrian features to obtain an initial feature map includes: The second pedestrian feature map and the third pedestrian feature map are fused to obtain a first fused feature map; The first pedestrian feature map and the first fused feature map are fused to obtain the second fused feature map; The second fused feature map is convolved by a variable 3D convolutional layer to obtain an initial feature map.

2. The pedestrian search method based on computer vision according to claim 1, wherein the plurality of residual block groups at least includes a first residual block group, a second residual block group, a third residual block group, and a fourth residual block group, and the step of processing the pooling features through the plurality of residual block groups to obtain pedestrian features includes: The pooling features are processed by the first residual block group to obtain residual features; The residual features are processed by the second residual block group to obtain the first pedestrian feature map; The first pedestrian feature map is processed by the third residual block group to obtain the second pedestrian feature map; The residual features are processed by the fourth residual block group to obtain the third pedestrian feature map; The first pedestrian feature map, the second pedestrian feature map, and the third pedestrian feature map are identified as pedestrian features.

3. The pedestrian search method based on computer vision according to claim 1, wherein fusing the second pedestrian feature map and the third pedestrian feature map to obtain a first fused feature map includes: Perform a variable 3D convolution on the second pedestrian feature map to obtain the first sub-feature map; The third pedestrian feature map is upsampled to obtain the second sub-feature map; The first sub-feature map and the second sub-feature map are fused to obtain the first fused feature map; And, the step of fusing the first pedestrian feature map and the first fused feature map to obtain the second fused feature map includes: Perform variable 3D convolution on the first pedestrian feature map to obtain the third sub-feature map; Upsample the first fused feature map to obtain the fourth sub-feature map; The third sub-feature map and the fourth sub-feature map are fused to obtain the second fused feature map.

4. The pedestrian search method based on computer vision according to claim 2 or 3, wherein performing classification and regression prediction on the initial feature map to obtain a pedestrian feature map includes: Pedestrian detection is performed on the initial feature map to obtain positive pedestrian sample features; Pedestrian re-identification is performed on the positive pedestrian sample features to obtain a pedestrian feature map.

5. A pedestrian search device based on computer vision, comprising: The first determining unit is configured to determine a spatiotemporal pipeline from multiple consecutive frame images acquired; wherein the spatiotemporal pipeline includes the pedestrian to be identified in the consecutive frame images; The input unit is used to input the spatiotemporal pipeline into the pedestrian search model to obtain the pedestrian feature map of the pedestrian to be identified. The second determining unit is used to determine the target pedestrian information corresponding to the pedestrian feature map; Specifically, the input unit inputs the spatiotemporal pipeline into the pedestrian search model to obtain the pedestrian feature map of the pedestrian to be identified in the following way: The spatiotemporal pipeline is input into the pedestrian search model for feature extraction to obtain pedestrian features; The pedestrian features are fused to obtain an initial feature map; The initial feature map is subjected to classification and regression prediction to obtain the pedestrian feature map; Specifically, the input unit inputs the spatiotemporal pipeline into the pedestrian search model for feature extraction, and the method for obtaining pedestrian features is as follows: The spatiotemporal pipeline is input into the pedestrian search model for convolution to obtain convolutional features; The convolutional features are pooled to obtain pooled features; The pooling features are processed by multiple residual block groups to obtain pedestrian features; wherein each residual block group contains multiple residual blocks, and the pedestrian features include a first pedestrian feature map, a second pedestrian feature map, and a third pedestrian feature map, and any two pedestrian feature maps in the first pedestrian feature map, the second pedestrian feature map, and the third pedestrian feature map have different dimensions; Specifically, the input unit fuses the pedestrian features to obtain the initial feature map in the following way: The second pedestrian feature map and the third pedestrian feature map are fused to obtain the first fused feature map; The first pedestrian feature map and the first fused feature map are fused to obtain the second fused feature map; The second fused feature map is convolved by a variable 3D convolutional layer to obtain an initial feature map.

6. A computing device, the computing device comprising: At least one processor, memory, and input / output unit; The memory is used to store computer programs, and the processor is used to call the computer programs stored in the memory to execute the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 4.

Images