Pedestrian detection model training method and device, and storage medium

Abstract

The present disclosure provides a pedestrian detection model training method and device and a storage medium, comprising: obtaining a training sample image, processing the training sample image by using a feature pyramid network to obtain a feature pyramid, converting the feature pyramid into an image block linear embedding sequence, inputting the image block linear embedding sequence into a pre-constructed Transformer network to output the positions and confidence of a plurality of prediction boxes, calculating a target loss value based on the positions and confidence of the actual boxes in the training sample image and the positions and confidence of the prediction boxes, and performing back propagation according to the target loss value to update the network weight. The feature pyramid network and the Transformer network are cascaded from coarse to fine, the network learns the features of each dimension in the pedestrian image, and the network can adapt to detection and positioning tasks in various scenes and environments, thereby improving the accuracy of pedestrian detection and positioning.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method, apparatus, and storage medium for training a pedestrian detection model. Background Technology

[0002] Pedestrian detection involves marking all pedestrians in an image with bounding boxes. It can be applied to pedestrian recognition and verification, as well as pedestrian tracking and retrieval in surveillance scenarios. During detection, the image is input into a pre-trained model, which outputs the coordinates of several bounding boxes containing the pedestrian's location. In practical applications, pedestrians may appear anywhere in an image, and pedestrians in the same or different images may have different sizes. Therefore, using existing pedestrian detection models to identify images with such conditions may lead to increased false detection and false negative rates. Summary of the Invention

[0003] This disclosure proposes a pedestrian detection model training method, apparatus, and storage medium, aiming to at least partially solve one of the technical problems in related technologies.

[0004] The first aspect of this disclosure proposes a method for training a pedestrian detection model, comprising: acquiring training sample images, wherein at least one pedestrian is present in the training sample images; processing the training sample images using a feature pyramid network to obtain a feature pyramid; converting the feature pyramid into a linear embedding sequence of image patches; inputting the linear embedding sequence of image patches into a pre-constructed Transformer network to output the positions and confidence scores of multiple predicted bounding boxes; calculating a target loss value based on the positions and confidence scores of the actual bounding boxes in the training sample images and the positions and confidence scores of the predicted bounding boxes; and performing backpropagation based on the target loss value to update the network weights.

[0005] A second aspect of this disclosure provides a pedestrian detection method, comprising: acquiring a target image to be predicted; inputting the target image into a pedestrian detection model trained by the pedestrian detection model training method described above to obtain first pedestrian bounding box information; performing Gaussian background modeling on the target image to obtain second pedestrian bounding box information; and determining pedestrians in the target image based on the first pedestrian bounding box information and the second pedestrian bounding box information.

[0006] A third aspect of this disclosure provides a pedestrian detection model training apparatus, comprising: a first acquisition module for acquiring training sample images, wherein at least one pedestrian is present in the training sample images; a first processing module for processing the training sample images using a feature pyramid network to obtain a feature pyramid; a transformation module for transforming the feature pyramid into a sequence of linear embeddings of image patches; a first prediction module for inputting the sequence of linear embeddings of image patches into a pre-constructed Transformer network to output the positions and confidence scores of multiple predicted bounding boxes; a loss calculation module for calculating a target loss value based on the positions and confidence scores of the actual bounding boxes in the training sample images and the positions and confidence scores of the predicted bounding boxes; and a backpropagation module for performing backpropagation based on the target loss value to update the network weights.

[0007] A fourth aspect of this disclosure provides a pedestrian detection device, comprising: a second acquisition module for acquiring a target image to be predicted; a second prediction module for inputting the target image into a pedestrian detection model trained by the aforementioned pedestrian detection model training method to obtain first pedestrian bounding box information; a modeling module for performing Gaussian background modeling on the target image to obtain second pedestrian bounding box information; and a determination module for determining pedestrians in the target image based on the first pedestrian bounding box information and the second pedestrian bounding box information.

[0008] A fifth aspect of this disclosure provides a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a pedestrian detection model training method of the present disclosure, or to perform a pedestrian detection method of the present disclosure.

[0009] A sixth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the pedestrian detection model training method disclosed in the embodiments of this disclosure, or to execute the pedestrian detection method of the embodiments of this disclosure.

[0010] In this embodiment, during model training, training sample images are acquired and processed using a feature pyramid network to obtain a feature pyramid. The feature pyramid is then converted into a linear embedding sequence of image patches, which is input into a pre-built Transformer network to output the positions and confidence scores of multiple predicted boxes. Based on the positions and confidence scores of the actual boxes in the training sample images and the positions and confidence scores of the predicted boxes, a target loss value is calculated. Backpropagation is then performed based on the target loss value to update the network weights. This allows the feature pyramid network and the Transformer network to be cascaded from coarse to fine, enabling the network to learn features in various dimensions of pedestrian images. This allows the network to adapt to detection and localization tasks in various scenarios and environments, improving the accuracy of pedestrian detection and localization.

[0011] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 This is a flowchart illustrating a pedestrian detection model training method according to an embodiment of the present disclosure;

[0014] Figure 2 This is a schematic diagram of a feature pyramid network structure provided according to an embodiment of this disclosure;

[0015] Figure 3 This is a schematic diagram of the Transformer network structure provided according to an embodiment of this disclosure;

[0016] Figure 4 This is a flowchart illustrating a pedestrian detection method according to another embodiment of the present disclosure;

[0017] Figure 5 This is a schematic diagram of a pedestrian detection model training device according to another embodiment of the present disclosure;

[0018] Figure 6 This is a schematic diagram of a pedestrian detection device according to another embodiment of the present disclosure;

[0019] Figure 7 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0020] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are used only to explain this disclosure, and should not be construed as limiting this disclosure. Rather, embodiments of this disclosure include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0021] It should be noted that the execution subject of the pedestrian detection model training method in this embodiment can be a pedestrian detection model training device. This device can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc.

[0022] Figure 1 This is a flowchart illustrating a pedestrian detection model training method according to an embodiment of the present disclosure, as shown below. Figure 1 As shown, the method includes:

[0023] S101: Obtain training sample images.

[0024] The images used to train the pedestrian detection model are referred to as training sample images. Each training sample image contains at least one pedestrian, and these images can be taken in any scene, at various poses, and from various angles; there are no restrictions on this. Furthermore, each training sample image also corresponds to the location information (e.g., coordinates) and confidence score for each pedestrian, i.e., the actual bounding box position of the pedestrian in the image. The confidence score can be set to 1. In some embodiments, the acquired training sample images can also be preprocessed, such as cropping them to a 640*640 pixel size, or other preprocessing can be performed; there are no restrictions on this.

[0025] S102: The training sample images are processed using a feature pyramid network to obtain the feature pyramid.

[0026] In this embodiment, for example, any one of ResNet18, ResNet32, or ResNet50 can be selected as the backbone neural network of the Feature Pyramid Net (FPN). If a faster detection speed is desired, a lightweight network, such as the MobileNet series network, can also be selected as the backbone neural network. There are no restrictions on this.

[0027] Once the backbone neural network is determined, training sample images can be input into the backbone neural network to obtain feature maps of each layer in the network. Then, feature maps output by each layer are selected to construct a feature pyramid.

[0028] For example, Figure 2 This is a schematic diagram of the feature pyramid network structure provided according to an embodiment of this disclosure, such as... Figure 2 As shown, the backbone neural network includes multiple convolutional layers, such as C1(Conv1(BN+ReLU)), C2(Conv2_x), C3(Conv3_x), C4(Conv4_x), and C5(Conv5_x). Each convolutional layer can produce a feature map. In this embodiment, feature maps from some layers can be selected to construct a feature pyramid, for example, three feature maps from layers C3, C4, and C5 can be selected to construct the feature pyramid. Wherein, as... Figure 2 As shown, this embodiment can use 1x1 convolution to adjust and align the number of channels in three effective feature layers C3, C4, and C5. After adjustment, Upsample and Add are used to perform upsampled feature fusion, and 1x1 convolution is performed on each fusion result to generate an FPN feature pyramid, including P3, P4, and P5.

[0029] S103: Transform the feature pyramid into a linear embedding sequence of image patches.

[0030] Specifically, the feature pyramid can be processed first. For example, P3, P4, and P5 can be reshaped into a flattened two-dimensional fragment sequence. Where (H3, W3), (H4, W4), and (H5, W5) are the resolutions of P3, P4, and P5, respectively, C is the number of channels, (P, P) is the resolution of each image fragment, and N = H * W / P 2 It represents the number of fragments generated.

[0031] Furthermore, by flattening all the image fragments, we obtain a linear embedding sequence of image patches, i.e., the patch embedding. Specifically, the image patches can be linearly projected onto the size of vector D, and the output of this projection is called the patch embedding. The patch embedding expression is as follows:

[0032]

[0033] Z0 represents the patch embedding, which can be used as input to the Transformer network; X P E represents the vector of each image patch, E pos It is a constant.

[0034] S104: Input the linear embedding sequence of image patches into a pre-built Transformer network to output the positions and confidence scores of multiple prediction boxes.

[0035] In this embodiment, a Transformer Encoder network structure can be pre-constructed as a Transformer network. This Transformer Encoder network structure may include, for example, multi-head self-attention layers, normalization layers (LN), multilayer perceptrons (MLP), residual connection layers, and any other possible computational layers, without limitation. The multi-head self-attention layers linearly connect multiple attention outputs to the desired dimension; multiple attention heads help understand local and global dependencies in the image. The multilayer perceptron (MLP) contains two Gaussian error linear unit (GELU) layers for normalization (LN), applied before each patch, without introducing any new dependencies between training images, thus improving training time and generalization performance. Residual connections are applied after each patch because they allow gradients to flow directly through the network without passing through nonlinear layers.

[0036] In some embodiments, the Transformer network may include multiple sub-encoders connected in sequence. Each sub-encoder includes a multi-head self-attention layer, a normalization layer, a multilayer perceptron, etc. The output of the previous sub-encoder is the input of the next sub-encoder. The image patch embedding sequence is the input of the first sub-encoder, that is, the input data of the multi-head self-attention layer of the first sub-encoder. Figure 3 This is a schematic diagram of the Transformer network structure provided according to an embodiment of this disclosure, such as... Figure 3 As shown, a sub-encoder can be formed by sequentially connecting a multi-head self-attention layer (multi-head attention network module), a normalization layer, a multi-layer perceptron (perceptron module), and a normalization layer. This sub-encoder can be repeatedly calculated multiple times (e.g., 7 times) to achieve the function of multiple sub-encoders.

[0037] In some embodiments, the calculation process of the multi-head self-attention layer in each sub-encoder is as follows:

[0038] Z′ l =MSA(LN(Z) l-1 ))+Z l-1

[0039] Among them, Z l-1 The output of the previous sub-encoder is LN, which represents the normalization layer, and MSA is the multi-head self-attention layer.

[0040] In other embodiments, the computation results of the multi-head self-attention layer can be used as the input to the multilayer perceptron of the sub-encoder. The computation process of the multilayer perceptron is as follows:

[0041] Zl =MLP(LN(Z′) l ))+Z′ l

[0042] Where MLP represents a multilayer perceptron, LN represents a normalization layer, and Z represents a normalization layer. l This serves as the input for the next sub-encoder.

[0043] After repeated network iterations (i.e., sequentially passing through a multi-head self-attention layer, a normalization layer, and a multilayer perceptron), the overall structure of the Transformer Encoder network is formed.

[0044] By inputting the patch embedding sequence corresponding to each training sample image into the Transformer Encoder network structure, the network can output the positions (e.g., coordinates) and confidence scores of multiple predicted boxes, i.e., the model's prediction results. Some embodiments allow... This represents a set of multiple prediction boxes, where N is the number of prediction boxes. Represents each prediction box, where, This indicates the confidence level of the predicted bounding box. Indicates the position of the prediction box, where, w represents the coordinates of the center point of the prediction box. i h i This indicates the width and height of the prediction box.

[0045] In some embodiments, let y be the ground truth set, that is, the positions and confidence scores of multiple actual boxes. The number of actual boxes can be represented by M. The number of predicted boxes N can be the same as the number of labeled actual boxes M; or the number of predicted boxes N can be greater than the number of actual boxes M, without any restriction.

[0046] S105: Calculate the target loss value based on the position and confidence of the actual bounding box in the training sample image and the position and confidence of the predicted bounding box.

[0047] In other words, the loss function is used to calculate the loss value between the predicted box and the actual box. This loss value can be called the target loss value L. The loss function can be any loss function that can be selected, and there are no restrictions on it.

[0048] In some embodiments, when the number of predicted boxes N equals the number of actual boxes M, the pedestrian classification loss value can be calculated based on the confidence levels corresponding to the predicted boxes and actual boxes respectively; and the pedestrian regression loss value can be calculated based on the positions corresponding to the predicted boxes and actual boxes respectively. Then, the final target loss value L is determined based on the pedestrian classification loss value and the pedestrian regression loss value.

[0049] In other embodiments, when the number of predicted boxes N is greater than the number of actual boxes M, this embodiment can construct a matching relation matrix (also called an assignment matrix) based on the position and confidence of the actual boxes and the position and confidence of the predicted boxes. The matching relation matrix can be represented by P∈[0,1]. M×N :express.

[0050] For example, if the Transformer Encoder network structure outputs 100 (N) predicted boxes and the number of actual boxes M is 10, then in this embodiment, a 100*10 matching relationship matrix can be constructed between the predicted boxes and the actual boxes.

[0051] Furthermore, the target loss value is calculated based on the matching relationship matrix.

[0052] In some embodiments, the candidate loss values ​​between the target predicted boxes and the actual boxes in multiple prediction boxes can be calculated iteratively based on the matching relationship matrix. The number of target predicted boxes is the same as the number of actual boxes. That is, 10 target predicted boxes are selected iteratively from 100 prediction boxes, and the loss value between the 10 actual boxes and the 10 target predicted boxes in each iteration is calculated. This loss value can be called the candidate loss value. It can be understood that a candidate loss value can be obtained in each iteration.

[0053] In some embodiments, in the operation of iteratively calculating the candidate loss values ​​of the target predicted box and the actual box in multiple prediction boxes, the pedestrian classification loss value and the pedestrian regression loss value of the target predicted box and the actual box in each iteration can be calculated first, and then the candidate loss value can be calculated based on the pedestrian classification loss value and the pedestrian regression loss value.

[0054] Specifically, the pedestrian classification loss value can be calculated based on the confidence scores corresponding to the predicted bounding box and the actual bounding box, respectively. The calculation formula is as follows:

[0055]

[0056] in, Indicates the confidence level of the actual bounding box. This indicates the confidence level of the predicted target bounding box. This represents the matching value between a set of actual bounding boxes and the target predicted bounding boxes in each iteration. For example, combining 10 candidate predicted bounding boxes with 10 actual bounding boxes yields 10 sets of combined results. This formula can be used to calculate the matching value between each set of actual bounding boxes and the target predicted bounding boxes. (Also known as the loss value), and by summing the 10 sets of matching values, we can obtain the final pedestrian classification loss value, which can be used... express.

[0057] Furthermore, this embodiment can calculate the pedestrian regression loss value based on the positions corresponding to the predicted target bounding box and the actual bounding box, respectively. In some embodiments, the calculation formula is as follows:

[0058]

[0059] Where i represents the actual number of boxes (e.g., i = 1, 2, ..., 10). Indicates the actual position of the box ( (Euclidean distance from the center coordinates) This indicates the position (Euclidean distance) of the predicted bounding box. This represents the pedestrian regression loss value.

[0060] The calculation method for smoothL1 is as follows:

[0061]

[0062] Furthermore, the pedestrian classification loss value and the pedestrian regression loss value are weighted and calculated to determine the candidate loss value. The calculation formula is as follows:

[0063]

[0064] Where α is the weight of the human classification loss value, β is the weight of the pedestrian regression loss value, and L is the candidate loss value.

[0065] Furthermore, the candidate loss value with the smallest value is determined as the target loss value. Some embodiments may employ the Sinkhorn algorithm to iteratively solve for the minimum target loss value, or other methods may be used to iteratively solve for the target loss value; there are no limitations on this.

[0066] S106: Backpropagate based on the target loss value to update the network weights.

[0067] After determining the target loss value, backpropagation can be performed based on the target loss value to update the network weights, such as updating the network weights of each layer in the feature pyramid network and / or updating the network weights in the Transformer network, until the cascaded network composed of the feature pyramid network and the Transformer network converges.

[0068] In this embodiment, during model training, training sample images are acquired and processed using a feature pyramid network to obtain a feature pyramid. The feature pyramid is then converted into a linear embedding sequence of image patches, which is input into a pre-constructed Transformer network to output the positions and confidence scores of multiple predicted boxes. Based on the positions and confidence scores of the actual boxes in the training sample images and the positions and confidence scores of the predicted boxes, a target loss value is calculated. Backpropagation is then performed based on the target loss value to update the network weights. This allows the feature pyramid network and the Transformer network to be cascaded from coarse to fine, enabling the network to learn features in various dimensions of pedestrian images. This allows the network to adapt to detection and localization tasks in various scenarios and environments, improving the accuracy of pedestrian detection and localization.

[0069] Figure 4 This is a schematic flowchart of a pedestrian detection method according to an embodiment of the present disclosure, as shown below. Figure 4 As shown, the method includes:

[0070] S401: Obtain the target image to be predicted.

[0071] The image in which the location of pedestrians needs to be predicted can be called the target image, which may contain one or more pedestrians.

[0072] S402: Input the target image into the pedestrian detection model trained by the pedestrian detection model training method described above to obtain the first pedestrian bounding box information.

[0073] In other words, the pedestrian detection model, which uses the feature pyramid network and the Transformer network cascaded and trained in the above embodiments, predicts the pedestrian location box in the target image. The prediction result can be called the first pedestrian box information or the model box, which includes, for example, the location and confidence of the first pedestrian box.

[0074] S403: Perform Gaussian background modeling on the target image to obtain the second pedestrian bounding box information.

[0075] Furthermore, embodiments of this disclosure can also perform Gaussian background modeling and motion detection on the target image to obtain second pedestrian bounding box information, also known as motion bounding box, which may include the position and confidence level of the second pedestrian bounding box.

[0076] S404: Determine the pedestrians in the target image based on the first pedestrian bounding box information and the second pedestrian bounding box information.

[0077] For example, the final true target box is obtained by weighting the confidence scores of the pedestrian motion box and the pedestrian model box.

[0078] For example, the first pedestrian box information may include the location and confidence level of 10 pedestrians, and the second pedestrian box information may also include the location and confidence level of 10 pedestrians. In this case, this embodiment performs a weighted calculation on the confidence level of each group of boxes (including a corresponding first pedestrian box and a second pedestrian box), and compares the weighted calculation result with a set threshold. If it is greater than the set threshold, the group of boxes can be determined to be the correct target box.

[0079] Therefore, the embodiments of this disclosure utilize a cascaded network of feature pyramid network and Transformer network to predict the pedestrian position in the target image, and combine the pedestrian position obtained by Gaussian background modeling and motion detection to determine the final pedestrian position, thus improving the accuracy of pedestrian detection and localization.

[0080] Figure 5 This is a schematic diagram of a pedestrian detection model training device according to another embodiment of this disclosure. Figure 5 As shown, the pedestrian detection model training device 50 includes:

[0081] The first acquisition module 501 is used to acquire training sample images, wherein at least one pedestrian exists in the training sample images;

[0082] The first processing module 502 is used to process the training sample images using a feature pyramid network to obtain a feature pyramid.

[0083] Transformation module 503 is used to transform the feature pyramid into a linear embedding sequence of image patches;

[0084] The first prediction module 504 is used to input the linear embedding sequence of image patches into a pre-built Transformer network to output the positions and confidence scores of multiple prediction boxes;

[0085] The loss calculation module 505 is used to calculate the target loss value based on the position and confidence of the actual bounding box and the position and confidence of the predicted bounding box in the training sample image; and

[0086] The backpropagation module 506 is used to perform backpropagation based on the target loss value to update the network weights.

[0087] In some embodiments, the loss calculation module 505 is specifically used for:

[0088] Construct a matching matrix based on the location and confidence level of the actual bounding boxes and the location and confidence level of the predicted bounding boxes;

[0089] The target loss value is calculated based on the matching relationship matrix.

[0090] In some embodiments, the loss calculation module 505 is specifically used for:

[0091] Based on the matching relationship matrix, the candidate loss values ​​of the target prediction box and the actual box in multiple prediction boxes are calculated iteratively, where the number of target prediction boxes is the same as the number of actual boxes;

[0092] The candidate loss value with the smallest value is determined as the target loss value.

[0093] In some embodiments, the loss calculation module 505 is specifically used to: iteratively calculate candidate loss values ​​using the Sinkhorn algorithm.

[0094] In some embodiments, the loss calculation module 505 is specifically used to: calculate the pedestrian classification loss value based on the confidence levels corresponding to the target predicted box and the actual box respectively; calculate the pedestrian regression loss value based on the positions corresponding to the target predicted box and the actual box respectively; and perform weighted calculation on the pedestrian classification loss value and the pedestrian regression loss value to determine the candidate loss value.

[0095] In some embodiments, the pedestrian classification loss value is calculated according to the following formula:

[0096]

[0097] in, This indicates the confidence level of the actual bounding box. This indicates the confidence level of the predicted target bounding box. This represents the matching value between a set of actual bounding boxes and the target predicted bounding box in each iteration, and the multiple matching values ​​in each iteration represent the pedestrian classification loss value.

[0098] In some embodiments, the pedestrian regression loss value is calculated according to the following formula:

[0099]

[0100] in, Indicates the actual position of the box. Indicates the position of the target prediction box. This represents the pedestrian regression loss value.

[0101] In some embodiments, the Transformer network includes a plurality of sub-encoders connected in sequence, each sub-encoder including a multi-head self-attention layer and a multilayer perceptron.

[0102] Figure 6 This is a schematic diagram of a pedestrian detection device according to another embodiment of the present disclosure. Figure 6 As shown, the pedestrian detection model training device 60 includes:

[0103] The second acquisition module 601 is used to acquire the target image to be predicted;

[0104] The second prediction module 602 is used to input the target image into the pedestrian detection model trained by the pedestrian detection model training method of any one of claims 1-8 above, so as to obtain the first pedestrian box information.

[0105] Modeling module 603 is used to perform Gaussian background modeling on the target image to obtain the second pedestrian bounding box information; and

[0106] The determination module 604 is used to determine pedestrians in the target image based on the first pedestrian frame information and the second pedestrian frame information.

[0107] In this embodiment, during model training, training sample images are acquired and processed using a feature pyramid network to obtain a feature pyramid. The feature pyramid is then converted into a linear embedding sequence of image patches, which is input into a pre-built Transformer network to output the positions and confidence scores of multiple predicted boxes. Based on the positions and confidence scores of the actual boxes in the training sample images and the positions and confidence scores of the predicted boxes, a target loss value is calculated. Backpropagation is then performed based on the target loss value to update the network weights. This allows the feature pyramid network and the Transformer network to be cascaded from coarse to fine, enabling the network to learn features in various dimensions of pedestrian images. This allows the network to adapt to detection and localization tasks in various scenarios and environments, improving the accuracy of pedestrian detection and localization.

[0108] According to embodiments of this disclosure, this disclosure also provides a computer device, a readable storage medium, and a computer program product.

[0109] To implement the above embodiments, this disclosure also proposes a computer program product that, when the instruction processor in the computer program product is executed, performs the pedestrian detection model training method proposed in the foregoing embodiments of this disclosure.

[0110] Figure 7 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Figure 7 The computer device 12 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0111] like Figure 7 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0112] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the MicroChannel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0113] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0114] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 7 Not shown; usually referred to as a "hard drive".

[0115] although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0116] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0117] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with computer device 12, and / or with any device that enables computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0118] The processing unit 16 executes various functional applications by running programs stored in the system memory 28, such as implementing the pedestrian detection model training method mentioned in the foregoing embodiments.

[0119] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0120] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0121] It should be noted that in the description of this disclosure, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0122] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0123] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0124] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0125] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0126] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0127] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0128] Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A method for training a pedestrian detection model, characterized in that, include: Obtain training sample images, wherein at least one pedestrian is present in the training sample images; The training sample images are processed using a feature pyramid network to obtain a feature pyramid; The feature pyramid is transformed into a linear embedding sequence of image patches; The linear embedding sequence of the image patches is input into a pre-built Transformer network to output the positions and confidence scores of multiple predicted boxes; Based on the position and confidence level of the actual bounding boxes in the training sample images, and the position and confidence level of the predicted bounding boxes, the target loss value is calculated; and Backpropagation is performed based on the target loss value to update the network weights.

2. The method as described in claim 1, characterized in that, The calculation of the target loss value based on the position and confidence of the actual bounding box in the training sample image and the position and confidence of the predicted bounding box includes: A matching matrix is ​​constructed based on the position and confidence level of the actual bounding box and the position and confidence level of the predicted bounding box; The target loss value is calculated based on the matching relationship matrix.

3. The method as described in claim 2, characterized in that, The step of calculating the target loss value based on the matching relationship matrix includes: Based on the matching relationship matrix, the candidate loss values ​​of the target predicted boxes and the actual boxes in the plurality of predicted boxes are calculated iteratively, wherein the number of target predicted boxes is the same as the number of actual boxes; The candidate loss value that is the smallest is determined as the target loss value.

4. The method as described in claim 3, characterized in that, The iterative calculation of the candidate loss values ​​between the target predicted box and the actual box in the plurality of predicted boxes includes: The candidate loss value is calculated iteratively using the Sinkhorn algorithm.

5. The method as described in claim 3, characterized in that, The iterative calculation of the candidate loss values ​​between the target predicted box and the actual box in the plurality of predicted boxes includes: Calculate the pedestrian classification loss value based on the confidence scores corresponding to the predicted target bounding box and the actual bounding box, respectively. Calculate the pedestrian regression loss value based on the positions corresponding to the predicted target bounding box and the actual bounding box, respectively. The candidate loss value is determined by weighting the pedestrian classification loss value and the pedestrian regression loss value.

6. The method as described in claim 5, characterized in that, in, The pedestrian classification loss value is calculated using the following formula. in, This indicates the confidence level of the actual bounding box. This indicates the confidence level of the predicted target bounding box. This represents the pedestrian classification loss value.

7. The method as described in claim 5, characterized in that, in, The pedestrian regression loss value is calculated using the following formula. in, This indicates the position of the actual box. Indicates the position of the target prediction box. This represents the pedestrian regression loss value.

8. The method as described in claim 1, characterized in that, in, The Transformer network comprises a multi-head self-attention layer, a normalization layer, and a multilayer perceptron connected in sequence.

9. A pedestrian detection method, characterized in that, include: Obtain the target image to be predicted; The target image is input into the pedestrian detection model trained by the pedestrian detection model training method according to any one of claims 1-8 above, so as to obtain the first pedestrian bounding box information; Gaussian background modeling is performed on the target image to obtain the second pedestrian bounding box information; as well as Based on the first pedestrian frame information and the second pedestrian frame information, the pedestrians in the target image are determined.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the pedestrian detection model training method according to any one of claims 1-8, or to execute the pedestrian detection method according to claim 9.

Images