Controllable pedestrian generation method, device, and medium based on generative adversarial neural networks

Abstract

This invention discloses a controllable pedestrian generation method, device, and medium based on a generative adversarial neural network (GAN). The method improves the synthesis effect by having a discriminator and a generator compete against each other. The generator has an overall encoder-decoder structure, and the decoder incorporates a three-branch structure: a foreground branch synthesizes pedestrian feature maps, a background branch synthesizes street scene feature maps, and a mask branch synthesizes a segmentation mask for pedestrians, which guides the fusion of feature maps from the foreground and background branches. The discriminator includes a pedestrian discriminator and a mask discriminator, used for adversarial training to improve the realism of the synthesized pedestrians and masks, and to measure whether the mask can accurately segment pedestrians in the synthesized image. The model further constrains the input mask to be consistent with the synthesized mask through a loss function, thereby controlling the synthesis of pedestrian images according to a specified semantic mask. This invention can be widely applied in the field of pedestrian generation technology.

Description

Technical Field

[0001] This invention relates to the field of pedestrian generation technology, and in particular to a controllable pedestrian generation method, device and medium based on generative adversarial neural networks. Background Technology

[0002] In recent years, generative adversarial neural networks (GANs) have demonstrated excellent performance in generating diverse and high-fidelity images. To enhance the constraint on the structure of generated images, many works have attempted to use the structural mask of the input model as control information, training the model to synthesize images that conform to its structure. This allows for the segmentation of specified content in images using semantic masks, serving downstream tasks.

[0003] When the amount of data is sufficient, the effect of controllable image generation is quite ideal. However, in practical applications, the training of controllable image generation neural networks is subject to some limitations. For street scene pedestrian datasets, images can be captured by vehicle-mounted cameras, and common annotation methods include low-cost detection box annotations. However, semantic segmentation annotation generally does not consume a lot of human, material, and financial resources. Semantic segmentation annotation is a challenging task. While vehicle-mounted cameras can easily obtain a large amount of rich pedestrian data by capturing street scene images and annotating pedestrian detection boxes, most existing controllable image generation neural networks rely on datasets that match masks and images. Even if semantic segmentation annotation is performed on pedestrians in some images, it is difficult to provide a large amount of data for training the generation model, making it impossible to achieve high-quality controllable pedestrian generation at a low cost. Summary of the Invention

[0004] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a controllable pedestrian generation method, device and medium based on generative adversarial neural networks.

[0005] The first technical solution adopted in this invention is:

[0006] A controllable pedestrian generation method based on generative adversarial neural networks includes the following steps:

[0007] Obtain the pedestrian instance dataset and the pedestrian pose mask dataset, and denote the pedestrian instance dataset as... The pedestrian pose mask dataset is denoted as

[0008] Construct a mapping network M to map the input control vector z, which follows a Gaussian distribution, to an intermediate vector w;

[0009] Constructing a mask encoder E m Used for input pedestrian pose mask x m Perform feature e i Extraction;

[0010] Construct a three-branch generator network G to generate data based on the intermediate vector w and features of the pedestrian pose mask. To generate pedestrian images and the corresponding pedestrian segmentation mask

[0011] The generated pedestrian images Pedestrian segmentation mask As fake data, pedestrian instance dataset X p Pedestrian pose mask dataset X m The data in the image is used as real data to train the image discriminator D. img and mask discriminator D mask The image discriminator D is among them. img Used to identify generated pedestrian images and images x from the pedestrian instance dataset p Mask discriminator D mask Used to identify the generated pedestrian segmentation mask and the mask x from the pedestrian pose mask dataset m ;

[0012] Through image discriminator D img and mask discriminator D mask With the mapping network M used for synthesizing pedestrian images and masks, and the mask encoder E m The three-branch generator network G uses adversarial interactions to constrain the learning of the neural network. When the adversarial learning reaches equilibrium, the generator can synthesize pedestrian images of different styles based on the input style vector and control mask. and its corresponding mask

[0013] Furthermore, the pedestrian instance dataset X p Obtained from the pedestrian detection dataset: Based on the pedestrian detection bounding box annotations in the pedestrian detection dataset, extract the corresponding regions in the dataset images to construct the pedestrian instance dataset X. p ;

[0014] The pedestrian pose mask dataset X m The data in this dataset comes from multiple different pedestrian segmentation datasets.

[0015] Furthermore, a multilayer perceptron (MLP) is used as the network structure for the mapping network M;

[0016] The intermediate vector w output by the mapping network M, after affine transformation, serves as a style control condition and is input into each synthesis block of the three-branch generator network G to influence the generated pedestrian images. Clothing.

[0017] Furthermore, a multi-layer convolutional neural network is used as the mask encoder E. m The network structure;

[0018] Input pedestrian pose mask x m Features are extracted through convolution. This will be incorporated into the synthesis blocks on the backbone of the three-branch generator network G and the synthesis blocks on the mask branch to influence the generated pedestrian images. and the generated pedestrian segmentation mask The structure, i.e., pedestrian pose; where L represents the mask encoder E. m The number of convolutional layers.

[0019] Furthermore, the three-branch generation network G consists of four parts: the backbone, the foreground branch, the background branch, and the mask branch.

[0020] Each branch is implemented by multiple synthesizer blocks, which adopt the structure of synthesizer blocks in StyleGAN2; the synthesizer blocks of the backbone synthesize low-resolution common feature maps f based on the style obtained by affine transformation of the intermediate vector w. i out And a pedestrian pose mask x is introduced under the control of factor α. m Extracting features at the corresponding resolution e j After reaching a certain resolution, the three-branch generator network G differentiates into a foreground branch, a background branch, and a mask branch; the synthesizer blocks in the mask branch continue to receive style and pedestrian pose masks x. m Features e i Synthesize pedestrian segmentation mask Next, pedestrian segmentation mask It controls the fusion of feature maps between the foreground branch composite block and the background branch composite block at the corresponding resolution; after achieving a satisfactory resolution, the fused feature map c... fused It will be converted into generated pedestrian images

[0021] Furthermore, the fused feature map c fused The expression is:

[0022]

[0023] Where σ is the downsampling function, c f c b c represents the output feature maps of the foreground composite block and the background composite block at the corresponding resolution, respectively. fused and c b These will be used as inputs for the next foreground and background composites, respectively.

[0024] Furthermore, the image discriminator D imgand mask discriminator D mask It adopts the same discriminator structure as StyleGAN2;

[0025] The image discriminator D img For the generated pedestrian images Scoring is performed on images x from a pedestrian instance dataset with the expectation that the score will be close to 0. p Score the results and aim for a score close to 1;

[0026] The mask discriminator D mask The generated pedestrian segmentation mask Scoring is performed on masks x from a pedestrian pose mask dataset with the expectation that the score will be close to 0. m The score is assigned and the desired score is close to 1.

[0027] Furthermore, the mask mapping network M and the mask encoder E m Three-branch generator network G and image discriminator D img and mask discriminator D mask To enhance the realism of the synthetic data, an adversarial approach is employed, and the loss function is expressed as follows:

[0028]

[0029] Where S represents the softplus function; Z represents the expected value; Z represents the set of control vectors of the input mapping network M that follow a Gaussian distribution.

[0030] In addition, a reconstruction loss for the mask was introduced. To enhance the mask x according to the input m The ability to synthesize pedestrians in corresponding postures.

[0031] The second technical solution adopted in this invention is:

[0032] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a controllable pedestrian generation method based on a generative adversarial neural network as described above.

[0033] The third technical solution adopted in this invention is:

[0034] A computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a controllable pedestrian generation method based on a generative adversarial neural network as described above.

[0035] The fourth technical solution adopted in this invention is:

[0036] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned controllable pedestrian generation method based on a generative adversarial neural network.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] (1) This invention combines a mapping network M and a mask encoder E. m Three-branch generator network G, image discriminator D img and mask discriminator D mask Five neural networks, through adversarial joint training among three of them, ultimately achieved controllable high-definition pedestrian image synthesis, enabling the generated pedestrian images to be segmented based on masks and better serve downstream semi-supervised pedestrian detection tasks.

[0039] (2) Most existing methods for controlling pedestrian synthesis using input masks rely on pixel-by-pixel semantic annotation of training data. They cannot effectively synthesize pedestrians in a specified pose from unsegmented and unannotated images and lack generalization ability on pedestrian datasets containing only detection box annotations. This invention frees mask-controlled pedestrian synthesis from dependence on pixel-by-pixel semantically annotated training data, reducing the training set requirement from semantic segmentation datasets to object detection datasets, and has good generalization ability on different pedestrian datasets.

[0040] (3) This invention utilizes a mapping network M and a mask encoder E. m The style and structure information of the synthesized pedestrian images are controlled separately, and decoupled control of the style and structure of the generated images is achieved. This is of great significance for improving downstream semi-supervised pedestrian detection tasks, and can improve the detection model's ability to detect pedestrians with different clothing in different scenes. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a model structure diagram of a controllable pedestrian generation method based on a generative adversarial neural network in an embodiment of the present invention. Detailed Implementation

[0043] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0044] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0045] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0046] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0047] To address the existing technical problems, this invention provides a controllable pedestrian generation method, device, and medium based on generative adversarial neural networks. In this invention, the model, through appropriate structural constraints, achieves the synthesis of pedestrian images according to a specified semantic mask without performing pixel-by-pixel semantic annotation on the pedestrian image dataset. This allows for the full utilization of a large amount of street scene pedestrian data to synthesize realistic and diverse pedestrian images, and the generated data can be applied to pedestrian detection model data augmentation.

[0048] Specifically, this invention improves the synthesis effect by employing an adversarial approach between the discriminator and the generator. The generator is an encoder-decoder structure, with a three-branch structure introduced in the decoder: the foreground branch synthesizes pedestrian feature maps, the background branch synthesizes street scene feature maps, and the mask branch synthesizes a segmentation mask for pedestrians, which guides the fusion of feature maps from the foreground and background branches. The discriminator includes a pedestrian discriminator and a mask discriminator, used for adversarial training to improve the realism of the synthesized pedestrians and masks, and to measure whether the mask can accurately segment pedestrians in the synthesized image. The model further constrains the input mask to be consistent with the synthesized mask through a loss function, thereby controlling the synthesis of pedestrian images according to a specified semantic mask.

[0049] Example 1

[0050] like Figure 1 As shown, this embodiment provides a controllable pedestrian generation method based on generative adversarial neural networks. Utilizing pedestrian data from a pedestrian detection dataset and data from an independent pedestrian pose mask dataset, the model is trained through an adversarial process between a generator and a discriminator to synthesize realistic and diverse pedestrian images that conform to specified pose constraints, thereby serving downstream tasks. The method specifically includes the following steps:

[0051] S1. Prepare a pedestrian instance dataset and a pedestrian pose mask dataset, with no matching relationship between the two datasets. Let the pedestrian instance dataset be denoted as... The pedestrian pose mask dataset is denoted as

[0052] In some embodiments, pedestrian instance dataset X p This can be obtained from a pedestrian detection dataset. Specifically, based on the pedestrian detection bounding box annotations in the pedestrian detection dataset, regions at corresponding locations in the dataset images are extracted to construct a pedestrian instance dataset X. p Pedestrian pose mask dataset X m The data in the dataset can come from multiple different pedestrian segmentation datasets.

[0053] For example, the pedestrian instance data comes from the CityPersons dataset, whose training set contains 2975 RGB street scene images with pedestrian detection bounding boxes. Pedestrian pose mask data comes from multiple datasets such as CityScapes and Penn-FUDAN. Based on the pedestrian's location, pedestrian images and pose masks can be extracted from the dataset, and their resolution is uniformly changed to 128×128 to obtain the pedestrian instance dataset. Pedestrian pose mask dataset In this embodiment, the number of pedestrian instances N p =12329, Number of pedestrian pose masks N m =2645.

[0054] S2. Train a mapping network M:z→w implemented by a neural network to map the input control vector z~N(0,I) that follows a Gaussian distribution to the intermediate vector w.

[0055] As an optional implementation, an MLP is chosen as the network structure for the mapping network M. The intermediate vector w output by the mapping network M, after affine transformation, is input as a "style" control condition into each synthesis block of the three-branch generator network G, which can affect the generated pedestrian image. Clothing.

[0056] S3. Train a mask encoder implemented by a neural network. For the input pedestrian pose mask x m ∈X m Perform feature e i Extraction.

[0057] As an optional implementation, a multi-layer convolutional neural network is selected as the mask encoder E. m The network structure. The input pedestrian pose mask x m Features are extracted through convolution. It will be incorporated into the synthesis blocks on the backbone of the three-branch generator network G and the synthesis blocks of the mask branch, which can affect the generated pedestrian images. and the generated pedestrian segmentation mask The structure, i.e., the pedestrian's posture. In this embodiment, the mask encoder E m The number of convolutional layers, i.e. the number of output features, is L = 5. e1, e2, e3, e4, and e5 represent feature maps with resolutions of 64×64, 32×32, 16×16, 8×8, and 4×4, respectively.

[0058] S4. Train a three-branch generator network implemented by a neural network. Based on the intermediate vector w obtained in step S2 and the features of the pedestrian pose mask obtained in step S3 To generate pedestrian images and the corresponding pedestrian segmentation mask

[0059] As a specific implementation, the three-branch generation network G consists of four parts: the backbone, the foreground branch, the background branch, and the mask branch. Each branch is implemented by multiple synthesis blocks, which adopt the structure of synthesis blocks in StyleGAN2. The synthesis blocks of the backbone synthesize low-resolution 4×4, 8×8, 16×16, and 32×32 common feature maps f based on the "style" obtained by affine transformation of the intermediate vector w. i out And a pedestrian pose mask x is introduced under the control of factor α. m Extracting features at the corresponding resolution e j The formula is:

[0060]

[0061] in This represents the input feature map for the next synthesis block. In this embodiment, the control factor α = 0.9.

[0062] After reaching a 32×32 resolution, the three-branch generation network G differentiates into a foreground branch, a background branch, and a mask branch. The composite blocks in the mask branch continue to receive "style" data and the pedestrian pose mask x. m Features e j The system synthesizes feature maps with resolutions of 64×64 and 128×128, and then converts the 128×128 feature map into a 128×128 pedestrian segmentation mask. Next, pedestrian segmentation mask It controls the feature map fusion of the foreground branch composite block and the corresponding resolution background branch composite block at a 64×64 resolution, using the following formula:

[0063]

[0064] Where σ is the downsampling function. These represent the 64×64 resolution feature maps output by the foreground composite block and the background composite block at the corresponding resolution, respectively. This represents the fused feature map, and and These will be used as inputs for the next foreground and background composite blocks, respectively. Furthermore, the pedestrian segmentation mask... It controls the feature map fusion of the foreground branch composite block and the corresponding resolution background branch composite block at a 128×128 resolution, using the following formula:

[0065]

[0066] Fusion Feature Map It will be converted into a 128×128 resolution RGB pedestrian image.

[0067] S5. The pedestrian images generated in step S4 Pedestrian segmentation mask As fake data, pedestrian instance dataset X p Pedestrian pose mask dataset X m The data in the image is used as real data to train an image discriminator D implemented by a neural network. img :x→[0,1] and mask discriminator D mask :x→[0,1], where the image discriminator D img Used to identify generated pedestrian images and images x from the pedestrian instance dataset p ∈X p Mask discriminator D mask Used to identify the generated pedestrian segmentation mask and the mask x from the pedestrian pose mask dataset m ∈X m .

[0068] As an optional implementation, image discriminator D img and mask discriminator D mask It adopts the same discriminator structure as StyleGAN2. Image discriminator D img For the generated pedestrian images Scoring is performed on images x from a pedestrian instance dataset with the expectation that the score will be close to 0. p ∈X p Scoring is performed and the expected score is close to 1; Mask discriminator D mask The generated pedestrian segmentation mask Scoring is performed on masks x from a pedestrian pose mask dataset with the expectation that the score will be close to 0. m ∈X m The score is assigned and the desired score is close to 1.

[0069] S6, via image discriminator D img and mask discriminator D mask With the mapping network M used for synthesizing pedestrian images and masks, and the mask encoder E m A three-branch generative network G is used. The adversarial learning among the three branches constrains the neural network's learning. When the adversarial learning among the three branches reaches a balance, the generator can then learn based on the input style vector z and the control mask x. m Synthesizing a large number of pedestrian images with different styles and its corresponding mask

[0070] Specifically, the mask mapping network M and the mask encoder E m Three-branch generator network G and image discriminator D img and mask discriminator D mask To enhance the realism of synthetic data through adversarial analysis, the loss function formulas are as follows:

[0071]

[0072] Where S represents the softplus function S(t) = log(1 + exp(t)). Additionally, a mask reconstruction loss is introduced. To enhance the mask x according to the input m The ability to synthesize pedestrians in corresponding postures.

[0073] After training, the pedestrian instance dataset X, constructed from the CityPersons dataset, is used. p The performance of this method was quantitatively evaluated using Fréchet Inception Distance (FID) and Inception Score (IS). FID represents the similarity between the generated image and the real image in terms of feature distribution; a lower value indicates a more realistic generated image. IS represents the overall distribution of the generated image; a higher value indicates a more realistic and diverse generated image. The evaluation showed that this invention significantly outperformed the benchmark method on both evaluation criteria, making it worthy of widespread adoption.

[0074] In summary, this embodiment provides a controllable pedestrian generation method based on generative adversarial neural networks. This method achieves the synthesis of pedestrian images according to a specified semantic mask without performing pixel-by-pixel semantic annotation on the pedestrian image dataset. Considering the large amount of semantic information in the intermediate feature maps of the generative model, we introduce a mask encoder and a mapping network to edit the structural and stylistic semantics in the intermediate feature maps, respectively. Specifically, this invention adopts a three-branch encoding and decoding structure in the generative model: the mask encoder encodes the mask image controlling the structural semantics, the mapping network encodes the vector controlling the stylistic semantics, and the decoder's backbone synthesizes a low-resolution basic feature map, while the three branches synthesize high-resolution foreground, background, and mask images, respectively. The mapping network feeds the stylistic semantics into the feature maps synthesized by the decoder, while the mask encoder feeds the structural semantics into the decoder's backbone and the mask branches. The mask branches influence the feature map fusion of the foreground and background branches, thereby achieving control over the structure and style of the generated image. This invention enables the synthesis of a large number of pedestrian images with different styles but corresponding poses by specifying an arbitrary mask, without requiring semantic segmentation and annotation of the pedestrian dataset. The synthesized pedestrians can be segmented by the specified mask, and can be applied to downstream tasks such as training semi-supervised pedestrian detection models in traffic scenarios.

[0075] Example 2

[0076] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 1 This paper presents a controllable pedestrian generation method based on generative adversarial neural networks.

[0077] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0078] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0079] Since this electronic device is an electronic device corresponding to a controllable pedestrian generation method based on generative adversarial neural networks in this embodiment of the invention, and the principle of solving the problem by this electronic device is similar to that of this method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0080] Example 3

[0081] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 1 This paper presents a controllable pedestrian generation method based on generative adversarial neural networks.

[0082] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0083] Since this storage medium is the storage medium corresponding to a controllable pedestrian generation method based on a generative adversarial neural network in an embodiment of the present invention, and the principle of the storage medium in solving the problem is similar to that of the method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0084] Example 4

[0085] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of a controllable pedestrian generation method based on a generative adversarial neural network according to various exemplary embodiments of this application as described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0086] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0087] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0088] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A generatable pedestrian generation method based on a generative adversarial neural network, characterized by, The method comprises the following steps: An instance dataset of pedestrians and a pose mask dataset of pedestrians are acquired, and the instance dataset of pedestrians is denoted as , and the pose mask dataset of pedestrians is denoted as ; Constructing a mapping network for mapping an input control vector subject to a Gaussian distribution into an intermediate vector Constructing a mask encoder , for performing feature extraction on an input pedestrian pose mask ; Constructing a three-branch generative network for generating pedestrian images and corresponding pedestrian segmentation masks from an intermediate vector and features of a pedestrian pose mask ; wherein denotes a number of convolutional layers of a mask encoder ; generated pedestrian images and pedestrian segmentation masks pedestrian instance dataset and pedestrian pose mask dataset training image discriminator and mask discriminator ; wherein the image discriminator is configured to discriminate between generated pedestrian images and images from the pedestrian instance dataset , the mask discriminator is configured to discriminate between generated pedestrian segmentation masks and masks from the pedestrian pose mask dataset ; Through the image discriminator and the mask discriminator and the mapping network for synthesizing the pedestrian image and the mask , the mask encoder , the three-branch generation network , the learning of the neural network is constrained through the confrontation between them, when the confrontation learning reaches balance, the generator can synthesize the pedestrian image of different styles according to the input style vector and the control mask and the corresponding mask thereof ; The three-branch generative network The four parts are trunk, foreground branch, background branch and mask branch. Each branch is implemented by multiple synthesizer blocks, which adopt the structure of synthesizer blocks in StyleGAN2; the synthesizer blocks of the trunk are based on the intermediate vectors. The style obtained through affine transformation is used to synthesize a low-resolution common feature map. and in control factors Pedestrian pose mask introduced under regulation Feature extraction at corresponding resolution Once the feature maps synthesized from the backbone reach a certain resolution, the three-branch generator network... The process is divided into foreground, background, and mask branches; the composite block in the mask branch continues to receive style and pedestrian pose masks. Features Synthesize pedestrian segmentation mask Next, pedestrian segmentation mask. It controls the fusion of feature maps between the foreground branch composite block and the background branch composite block at the corresponding resolution; after achieving a satisfactory resolution, the feature maps are fused. It will be converted into generated pedestrian images .

2. The generatable pedestrian generation method based on the generative adversarial neural network according to claim 1, characterized in that, The pedestrian instance data set The pedestrian instance data set is obtained from the pedestrian detection data set: according to the pedestrian detection frame label in the pedestrian detection data set, an area at a corresponding position in the data set image is extracted to construct the pedestrian instance data set ; The pedestrian pose mask dataset The data in the dataset comes from a plurality of different pedestrian segmentation datasets.

3. The generatable pedestrian generation method based on the generative adversarial neural network according to claim 1, characterized in that, Using a multi-layer perceptron, MLP, as a mapping network network structure; Mapping network Output intermediate vector , after affine transformation as a style control condition, input three-branch generation network In each synthesis block, to affect the generated pedestrian image Clothing.

4. The generatable pedestrian generation method based on the generative adversarial neural network according to claim 1, characterized in that, Using a multi-layer convolutional neural network as a mask encoder network structure; Input pedestrian pose mask Features are extracted by convolution The synthetic block and the synthetic block of the mask branch on the main stem of the three-branch generation network will be introduced to affect the structure of the generated pedestrian image and the generated pedestrian segmentation mask , that is, the pedestrian pose.

5. The generatable pedestrian generation method based on the generative adversarial neural network according to claim 1, characterized in that, The fused feature map The expression is: wherein, is a downsampling function, , respectively represent the output feature maps of the foreground synthesis block and the background synthesis block of the corresponding resolution; and will be respectively taken as the input of the next foreground synthesis block and the background synthesis block.

6. The generatable pedestrian generation method based on the generative adversarial neural network according to claim 1, characterized in that, The image discriminator and the mask discriminator Uses the same architecture as the discriminator of StyleGAN2; The image discriminator The generated pedestrian images Scored and expected score close to 0, images from pedestrian instance dataset Scored and expected score close to 1; The mask discriminator The generated pedestrian segmentation mask Score and expect score close to 0, mask from pedestrian pose mask dataset Score and expect score close to 1.

7. The generatable pedestrian generation method based on the generative adversarial neural network according to claim 1, characterized in that, The mapping network of the mask , Mask encoder , Three-branch generation network With the image discriminator And the mask discriminator Conduct confrontation to improve the authenticity of the synthesized data, and the expression of the loss function is: wherein, represents a softplus function; represents an expected value; represents an input mapping network of a set of control vectors that are subject to a Gaussian distribution; Additionally, a reconstruction loss of the mask is introduced to enforce the mask according to the input the ability to synthesize corresponding pose pedestrians.

8. An electronic device, comprising: The electronic device comprises a processor and a memory, and the memory stores at least one program, which is loaded and executed by the processor to implement the method in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which is loaded and executed by the processor to implement the method in any one of claims 1 to 7.

Images