Method, apparatus and storage medium for generating pedestrian images

Abstract

The present invention relates to a method for generating pedestrian images, comprising: obtaining an image dataset of a scene area and classifying the scene area of the image data, wherein the scene area is semantically classified into pedestrian-accessible areas and other areas; training an image region segmentation model using training data and scene region classification information; obtaining a target background image; performing semantic segmentation on the target background image using the trained image region segmentation model to obtain and output a pedestrian-accessible area of the target background image; obtaining a target pedestrian image; determining a synthesis position for the target pedestrian image within the pedestrian-accessible area; and synthesizing the target pedestrian image into the target background image. The present invention also relates to a corresponding device and a machine-readable storage medium.

Description

Technical Field

[0001] The present invention relates to the fields of computer technology and autonomous driving, and in particular to a method and apparatus for generating pedestrian images, particularly in an autonomous driving scenario. Background Art

[0002] An artificial neural network (ANN) is a mathematical model that mimics the behavioral characteristics of animal neural networks and learns data characteristics through distributed, parallel processing of large amounts of data. ANNs process information by adjusting the relationships between multiple nodes within the system, relying on the complexity of the system. They also possess self-learning and adaptive capabilities.

[0003] Here, artificial neural networks can also be used to learn to generate target images. Although existing generative networks can generate high-quality images of people, existing generative networks do not perform semantic recognition analysis on the background, and cannot ensure the semantic information and physical rationality of the generated images, making the generated images unnatural and realistic. When generating pedestrian images in autonomous driving scenarios, existing generative models cannot determine the reasonable position and size of pedestrians synthesized in the background. Therefore, the synthesized pedestrian images may appear in unreasonable areas such as the sky and trees, or the size of the synthesized pedestrians may not conform to the imaging principle of large near and small far away, making the synthesized images unrealistic and natural. In view of the above shortcomings, there is a need for an improved method for generating pedestrian images in autonomous driving scenarios. Summary of the Invention

[0004] Against this backdrop, the present invention proposes a method, a corresponding apparatus, and a machine-readable storage medium for generating pedestrian images, particularly in autonomous driving scenarios. The method and apparatus of the present invention enable the generation of semantically consistent, natural, and realistic pedestrian images based on the recognition and analysis results of background images and a generative model. This approach can at least partially overcome the shortcomings of existing solutions.

[0005] According to a first aspect of the present invention, a method for generating a pedestrian image is provided, the method comprising the following steps:

[0006] Acquire an image data set of a scene area, and classify the scene area of the image data, wherein the scene area is classified into a pedestrian-accessible area and other areas according to semantics;

[0007] Use training data and scene region classification information to train the image region segmentation model;

[0008] Get the target background image;

[0009] Performing semantic segmentation on the target background image using a trained image region segmentation model, thereby obtaining a pedestrian feasible region of the target background image and outputting the region;

[0010] Obtain target pedestrian image;

[0011] Determining a synthesis position of the target pedestrian image in the pedestrian feasible area; and

[0012] The target pedestrian image is synthesized into the target background image.

[0013] The present invention can automatically provide reasonable pedestrian positions and sizes based on the recognition and analysis of background images, so that the synthesized image can be better controlled to have consistent semantics and physical rationality, so that the quality of the synthesized image is higher than that of the existing technology and the image is more natural and realistic.

[0014] In the method according to the present invention, scene areas can be divided into pedestrian-accessible areas and other areas. Pedestrian-accessible areas refer to areas where pedestrians are located and meet semantic rationality. Examples of such areas include crosswalks and intersections. Conversely, areas such as the sky and trees that do not meet the semantic and physical rationality of pedestrian-accessible areas are classified as other areas.

[0015] According to an exemplary embodiment, during the step of determining the synthesis position of the target pedestrian image within the pedestrian-accessible area, depth information of the synthesis position of the target pedestrian image is extracted, a reasonable size of the target pedestrian image is determined based on the depth information, and the target pedestrian image is synthesized into the target background image based on the synthesis position and size of the target pedestrian image. Thus, based on the depth information, a reasonable size of the target pedestrian image that conforms to imaging principles can be provided, making the synthesis of the target pedestrian image into the target background image more natural and realistic.

[0016] According to an exemplary embodiment, the method further includes the step of optimizing the synthesized image toward a natural and realistic appearance. This achieves the following technical advantage: the connection between the target pedestrian image and the background image is more natural and realistic. The synthesized image can be optimized using an effect enhancement model. For example, the synthesized image can be optimized based on a generative adversarial network model or, alternatively, using an image Gaussian filter.

[0017] According to another exemplary embodiment, the image region segmentation model is a known U-Net network structure. Thus, when using the known U-Net network structure, semantic segmentation of the target image can be performed simply and quickly with the help of a fully convolutional network.

[0018] According to an optional exemplary embodiment, the depth information is extracted by a depth sensor, and the depth of objects in the environment can be accurately measured by utilizing, for example, binocular stereo vision technology, radar detection technology, TOF (time of flight) technology, and depth detection technology based on structured light.

[0019] According to an exemplary embodiment, the method can be applied in different scenarios, in particular for synthesizing pedestrian images in an autonomous driving scenario.

[0020] According to a second aspect of the present invention, a corresponding device for generating a pedestrian image is provided, the device comprising: a feasible area acquisition unit, which is used to perform semantic segmentation on the target background image and obtain a pedestrian feasible area; a target pedestrian image acquisition unit, which is used to acquire a target pedestrian image; and a synthesis unit, which is used to synthesize the target pedestrian image into the target background image.

[0021] According to an exemplary embodiment, the apparatus further includes a depth acquisition unit configured to acquire depth information within a pedestrian-accessible area of the target background image, and synthesize the target pedestrian image into the target background image based on the depth information.

[0022] According to an optional exemplary embodiment, the device further includes an image optimization unit, which is used to optimize the synthesized image.

[0023] In an exemplary embodiment, the feasible region acquisition unit is an image region segmentation model based on a U-Net network structure.

[0024] In another exemplary embodiment, the image optimization unit is an image generation model based on an adversarial generative network model, or alternatively, an image optimization unit based on an effect enhancement network model.

[0025] According to yet another aspect, a machine-readable storage medium is provided, on which program instructions are stored. When the program instructions are executed by a processor, the medium is used to perform one of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be described in more detail below with reference to the accompanying drawings, so that the principles, features and advantages of the present invention can be better understood. In the accompanying drawings:

[0027] Figure 1 A schematic flow chart of a method for generating pedestrian images in an autonomous driving scenario is shown;

[0028] Figure 2 A schematic structural diagram of a device for generating pedestrian images in an autonomous driving scenario is shown. DETAILED DESCRIPTION

[0029] In order to make the technical problems, technical solutions and beneficial technical effects to be solved by the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and multiple exemplary embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0030] Figure 1 A schematic flow chart of the method of the present invention for generating pedestrian images in an autonomous driving scenario is shown.

[0031] In the first step S1, an image dataset of a scene area is acquired and the scene area of the image data is classified. The scene area is semantically classified into pedestrian-accessible areas and other areas. Here, pedestrian-accessible areas include, for example, crosswalks and intersections, while areas such as the sky and trees that do not meet the semantic and physical rationality of pedestrian-accessible areas are classified as other areas.

[0032] In the second step S2, the image region segmentation model is trained using the training data and the scene region classification information. The image region segmentation model can be implemented in a variety of ways. Here, the present invention exemplarily uses the U-Net network structure, but alternatively, any model network structure that can perform semantic segmentation on an image can also be used. The image region segmentation model is trained with the aid of trained image data and the corresponding segmentation region classification information, and the U-Net network is updated according to the loss function. The training is stopped when the number of updates reaches a preset number of steps, thereby obtaining a pedestrian feasible region segmentation model, which can be used, for example, to analyze the pedestrian feasible region that conforms to semantic and physical rationality in the background image output in an autonomous driving scenario.

[0033] In the third step S3, a target background image is acquired.

[0034] In the fourth step S4, the pedestrian feasible area segmentation model trained in the second step S2 is used to perform semantic segmentation on the target background image, thereby obtaining the pedestrian feasible area of the target background image and outputting it.

[0035] In the fifth step S5, a target pedestrian image is obtained. Here, the target pedestrian image can come from pedestrian data generated by a pedestrian generation network, or can be taken from pedestrian data in a real image.

[0036] In the sixth step S6, the synthesis position of the target pedestrian image is randomly determined in the pedestrian feasible area. Here, the present invention exemplarily uses a random function to generate two-dimensional random numbers within a specific value range as the coordinates of the pedestrian image synthesis position.

[0037] Optionally, in the seventh step S7, the depth information of the synthesis position of the target pedestrian image is extracted, and the reasonable size of the target pedestrian image is determined based on the depth information. The depth information is, for example, the depth information extracted by other depth sensors in the multi-sensor system, or it can also be derived from the true value of the depth information that has been marked. In the present invention, the depth information of a specific position is estimated by way of example using a camera model and camera parameters. After knowing that the synthesis position of the pedestrian image in the background image is (w, h) and the depth z (where w represents the horizontal coordinate of the image coordinate system, and h represents the vertical coordinate of the image coordinate system), according to the height distribution of the pedestrian:

[0038]

[0039] Randomly select a pedestrian height value y. According to the camera imaging principle, the size and height of the target pedestrian image are:

[0040]

[0041] The reasonable size of the target pedestrian image is determined by the synthesized position and the pedestrian's true height value.

[0042] In the eighth step S8, the target pedestrian image is synthesized into the target background image. Optionally, the synthesizing of the target pedestrian image into the target background image is implemented according to the synthesizing position and size of the target pedestrian image.

[0043] In an exemplary embodiment of the present invention, for example, the target pedestrian image is first resized proportionally according to the size requirements, and then the resized pedestrian image is added to the specified position of the target background image, so that the pixel point (0, a / 2) in the original coordinate system of the pedestrian image coincides with the pixel point (w, h) at the synthesized position of the background image, where a represents the image width of the pedestrian image after the size change, and (0, a / 2) represents the bottom center position of the pedestrian image.

[0044] Optionally, in the ninth step S9, the synthesized image is optimized using an effect enhancement model, thereby making the connection between the target pedestrian image and the background image more natural and realistic. The effect enhancement model can be implemented, for example, by using an image Gaussian filter. Alternatively, the present invention exemplarily uses a generative adversarial network as an image effect enhancement model. The model consists of a generator G and a discriminator D, and the input is the synthesized image x input , the output is y=G(x input ), the training data is the real pedestrian image x in the autonomous driving scenario real During the training process, a loss function needs to be calculated to update the parameters of the neural network. Specifically, the loss function used in the present invention includes the following two parts:

[0045] The first part is adversarial loss:

[0046] L GAN (G, D) = E y [log D(y)]+E x [log(1-D(G(x)))],

[0047] The second part is the reconstruction loss:

[0048]

[0049] The total loss function is:

[0050]

[0051] By constructing the loss function in this way, the connection between the character image and the background image is made more natural and realistic while retaining the image content.

[0052] Finally, the synthesized image can optionally be further processed using the trained effect enhancement model to obtain and output an optimized synthesized image.

[0053] exist Figure 2 Figure 2 shows a schematic diagram of a device for generating pedestrian images in an autonomous driving scenario. The device includes: a feasible region acquisition unit for segmenting a background image and obtaining a semantically consistent pedestrian feasible region; a target pedestrian image acquisition unit for acquiring a target pedestrian image; and a synthesis unit for synthesizing the generated target pedestrian image into a target background image.

[0054] According to an exemplary embodiment, the apparatus further includes a depth acquisition unit configured to acquire depth information within a pedestrian-accessible area of the target background image.

[0055] According to another exemplary embodiment, the apparatus further comprises an image optimizing unit, which is configured to optimize the synthesized image in a direction of naturalness and realism so as to achieve realistic effects.

[0056] Optionally, the feasible area acquisition unit is an image area segmentation model based on a U-Net structure, and the trained area segmentation model can be used to segment the background area to obtain a pedestrian feasible area.

[0057] Optionally, the target image optimization unit is an image generation model based on an adversarial generative network model. The trained image optimization unit can optimize the connecting contour part of the target pedestrian image and the background part, making the synthesized image more natural and realistic.

[0058] The present invention also relates to a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the processors are capable of performing the method according to the present invention. The computer-readable storage medium may include, for example, a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0059] Other advantages and alternative embodiments of the present invention will be readily apparent to those skilled in the art. Therefore, the present invention, in its broader sense, is not limited to the specific details, representative configurations, and exemplary embodiments shown and described. Rather, various modifications and substitutions may be made by those skilled in the art without departing from the basic spirit and scope of the present invention.

Claims

1. A method for generating a pedestrian image, the method comprising the following steps: Acquire an image data set of a scene area and classify the scene area of the image data, wherein the scene area is semantically classified into a pedestrian feasible area and other areas; train an image area segmentation model using the training data and scene area classification information; acquire a target background image; perform semantic segmentation on the target background image using the trained image area segmentation model to obtain a pedestrian feasible area of the target background image and output it; acquire a target pedestrian image; determine a synthesis position of the target pedestrian image in the pedestrian feasible area; and synthesize the target pedestrian image into the target background image.

2. The method according to claim 1, wherein Determining the synthesis position of the target pedestrian image in the pedestrian feasible area includes the following steps: extracting depth information of the synthesis position of the target pedestrian image, determining a reasonable size of the target pedestrian image based on the depth information; and synthesizing the target pedestrian image into the target background image based on the synthesis position and size of the target pedestrian image.

3. The method according to claim 1 or 2, wherein: The method further includes the following step: optimizing the synthesized image in a direction of natural realism, wherein the optimization of the synthesized image is based on a generative adversarial network model or is implemented through an image Gaussian filter.

4. The method according to claim 1 or 2, wherein: The image region segmentation model is a U-Net network structure.

5. The method according to claim 3, wherein The image region segmentation model is a U-Net network structure.

6. The method according to claim 2, wherein: The depth information is extracted by means of a depth sensor.

7. The method according to any one of claims 1, 2, 5 and 6, wherein: The method is used for pedestrian image synthesis in an autonomous driving scenario.

8. The method according to claim 3, wherein: The method is used for pedestrian image synthesis in an autonomous driving scenario.

9. The method according to claim 4, wherein: The method is used for pedestrian image synthesis in an autonomous driving scenario.

10. A device for generating a pedestrian image, the device comprising: A feasible area acquisition unit, which is used to perform semantic segmentation on the target background image and obtain a pedestrian feasible area; A target pedestrian image acquisition unit, which is used to acquire a target pedestrian image; and a synthesis unit, which is used to synthesize the target pedestrian image into the target background image.

11. The device according to claim 10, wherein The device also includes: a depth acquisition unit, which is used to obtain depth information within the pedestrian-accessible area of the target background image for output to the synthesis unit; and / or an image optimization unit, which is used to optimize the synthesized image in a natural and realistic direction.

12. The device according to claim 11, wherein The feasible region acquisition unit is an image region segmentation model based on a U-Net network structure, and / or the image optimization unit is an image generation model based on an adversarial generative network model or an image Gaussian filter. 13 . A machine-readable storage medium having program instructions stored thereon, for executing the method according to claim 1 when the program instructions are executed by a processor.

Images