A street view based child image synthesis method

Abstract

The application provides a street view-based child image synthesis method, comprising the following steps: detecting the body orientation attribute of a child in a foreground image dataset of the child, and detecting an optimal reference adult target and its body orientation attribute in a street view image dataset with an adult; extracting any two foreground images with the same body orientation attribute and street view images; scaling the foreground images according to the size of the detection box of the optimal reference adult target; and pasting the foreground images in the surrounding area of the optimal reference adult target in the street view images. The street view-based child image synthesis method of the application solves the problem of incoordination between the foreground image and the surrounding environment content of the pasting area of the street view image in the synthesized image content, guarantees the naturalness and rationality of the foreground and the background, ensures the reasonable body proportion of the adult and the child, and thus can be used for synthesizing child images in different street view scenes in different countries and cities, and improves the generalization ability of detecting children in various street view scenes.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and more specifically, relates to a method for synthesizing children's images based on street view.

[0002] Street View Technology

[0003] As Convolutional Neural Networks (CNNs) are increasingly used in computer vision, their reliance on image data is also growing. The amount of image data significantly impacts the accuracy of CNNs across different tasks. However, in real-world computer vision tasks, not all scene image data is readily available. Some task scenes may only have a few or dozens of images, which is far from sufficient for CNNs requiring thousands, tens of thousands, hundreds of thousands, millions, or even tens of millions of images. This limited image data restricts the application of CNNs. Therefore, obtaining specific scene image data is crucial. In practice, acquiring image data for various scenes requires enormous human and financial resources, making it unprofitable. Image synthesis effectively solves this problem. Image synthesis involves cutting out a specific foreground image from one image and pasting it onto another image (background) to create a new image. Image synthesis enables the creation of more images from fewer images, providing strong support for scenarios with extremely limited image data and offering ample training data for CNN models. Besides data augmentation, image compositing technology has a wide range of applications, such as background replacement for portraits, virtual social interaction, artistic creation, and automatic generation of advertising images.

[0004] In urban management systems, convolutional neural networks are used to detect missing children in street view images. To improve detection accuracy, a large amount of image data is required, necessitating image synthesis. However, due to the greater complexity of street view environments compared to conventional environments—different countries, cities, and streets have different styles—traditional image synthesis methods often result in inconsistencies, unnaturalness, and illogicalities in the foreground and background of synthesized images of children in street view scenes. For example, the foreground may be pasted into various unreasonable locations such as the sky, a river, a wall, or a car; the foreground size may be too large or too small; and children with inappropriate orientations may be placed on the target background street view image, leading to inconsistencies with the background content. These issues severely reduce the generalization ability of CNN models for detecting children in various street view scenarios. Summary of the Invention

[0005] The purpose of this application is to provide a method for synthesizing children's images based on street view, so as to solve the technical problems of unreasonable synthesis of children's images based on street view scenes in the traditional image synthesis process, which affects the generalization ability of detecting children in street view scenes.

[0006] To achieve the above objectives, the technical solution adopted in this application is: to provide a method for synthesizing children's images based on street view, comprising:

[0007] Detect the body orientation attributes of children in a foreground image dataset of children, and detect the best reference adult target and its body orientation attributes in a street view image dataset with adults;

[0008] Extract any two foreground images and street view images with the same body orientation attribute;

[0009] The foreground image is scaled proportionally to the detection box size of the best-referenced adult target.

[0010] Paste the foreground image into the area around the best reference adult target in the street view image.

[0011] Preferably, the body orientation attribute includes at least four categories: forward, backward, left, and right.

[0012] Preferably, the method for detecting the best reference adult target in a street view image dataset containing adults includes the following steps:

[0013] Detect all adult targets in a street view image dataset containing adults;

[0014] The highest-scoring adult target among all adult targets is selected as the best reference adult target.

[0015] Preferably, the method for scaling the foreground image proportionally according to the detection box size of the optimal reference adult target includes the following steps:

[0016] Obtain the bounding box coordinates of the optimal reference adult target;

[0017] Calculate the width and / or height of the detection box for the optimal reference adult target;

[0018] Scale the foreground image until its width and / or height are in a reasonable ratio to the width and / or height of the detection box of the best reference adult target.

[0019] Preferably, the method for calculating the reasonable ratio includes the following steps:

[0020] Detect the age of children in the foreground image;

[0021] Set an appropriate ratio based on age.

[0022] Preferably, after scaling the foreground image proportionally to the detection box size of the optimal reference adult target, the method further includes the step of:

[0023] Perform ground segmentation processing on the street view image to segment out the ground region;

[0024] Based on the shape and size of the mask image of the scaled foreground image, candidate regions of the same shape and size are randomly selected around the detection box of the best reference adult target in the street view image.

[0025] Determine if the bottom of the foreground pasting candidate area is within the ground area;

[0026] If not, then randomly select a foreground pasting candidate region of the same shape and size around the detection box of the best reference adult target in the street view image, and re-determine whether the bottom of the foreground pasting candidate region is within the ground area; if so, then select the current foreground pasting candidate region as the pasting region of the foreground image.

[0027] Preferably, after selecting the current foreground pasting candidate region as the pasting region of the foreground image, the method further includes the following steps:

[0028] The intersection and union ratio (OCR) of the pasted area with the detection box of the best reference adult target is calculated.

[0029] If the intersection-union ratio is less than the threshold, the final pasting region is determined; otherwise, a foreground pasting candidate region of the same shape and size is randomly selected around the detection box of the best reference adult target in the street view image, and the bottom of the foreground pasting candidate region is re-determined to be within the ground area.

[0030] Preferably, after scaling the foreground image proportionally to the detection box size of the optimal reference adult target, the method further includes the step of:

[0031] Perform an erosion operation on the mask image of the foreground image;

[0032] The mask image of the foreground image after the erosion operation is subjected to Gaussian blur.

[0033] Preferably, after scaling the foreground image proportionally to the detection box size of the optimal reference adult target, the method further includes the step of:

[0034] Select the foreground image in the paste area of ​​the street view image;

[0035] The pasted area is expanded and cropped in the street view image at a certain ratio with the pasted area as the center to obtain a local background image containing the pasted area;

[0036] The color and illumination information of the local background image are calculated, and an adaptive thresholding method is used to adjust the color and illumination information of the foreground image to match the local background image.

[0037] Preferably, the method for pasting a foreground image into the area surrounding the optimal reference adult target in a street view image includes the following steps:

[0038] Paste the foreground image with adjusted color and lighting information, along with its mask, into the pasting area of ​​the street view image;

[0039] Obtain street view images with both children and adults.

[0040] The street view-based child image synthesis method provided in this application, compared with the prior art, synthesizes a new street view image dataset containing both children and adults by extracting any two foreground images of children with the same body orientation attribute and a street view image with adults. This solves the problem of inconsistency between the foreground image and the surrounding environment of the pasted area of ​​the street view image in the synthesized image content, ensuring the naturalness and rationality of the foreground and background. At the same time, the foreground image is scaled proportionally according to the detection box size of the best reference adult target to ensure that the body proportions of adults and children are reasonable. In this way, it can synthesize child images in street view scenes of different countries and cities, and improve the generalization ability of detecting children in various street view scenes. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A schematic flowchart illustrating the street view-based child image synthesis method provided in this application embodiment;

[0043] Figure 2 Example diagram of a foreground image of a child provided for an embodiment of this application;

[0044] Figure 3 for Figure 2 A mask of the foreground image of the child in the image;

[0045] Figure 4 Example diagram of a street view image with an adult provided for an embodiment of this application;

[0046] Figure 5 for Figure 2 Foreground image and Figure 4 The composite image of the street view. Detailed Implementation

[0047] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0048] It should be noted that when a component is referred to as being "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as being "connected to" another component, it can be directly connected to or indirectly connected to that other component.

[0049] It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0050] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0051] Please refer to the following: Figures 1 to 5 The method for synthesizing children's images based on street view, as provided in the embodiments of this application, will now be described. The method for synthesizing children's images based on street view includes:

[0052] Step S1.0: Detect the body orientation attribute of children in the foreground image dataset of children, and detect the best reference adult target and its body orientation attribute in the street view image dataset with adults;

[0053] Step S2.0: Extract any two foreground images and street view images with the same body orientation attribute;

[0054] Step S3.0: Scale the foreground image proportionally according to the detection box size of the best reference adult target;

[0055] Step S4.0: Paste the foreground image into the area around the best reference adult target in the street view image.

[0056] It is understood that in step S1.0, the detection of the body orientation attribute can be determined by the human body orientation model. The human body orientation model is a classification model that can output multiple different body orientation attributes to replace manual annotation, improve image synthesis efficiency, and is suitable for large-scale image synthesis. Foreground images of children are placed in their respective foreground image category files according to the classification of their body orientation attributes and their corresponding mask images; street view images containing adults are placed in their respective street view image category files according to the classification of their body orientation attributes.

[0057] In step S2.0, any two foreground images and street view images with the same body orientation attribute are extracted, which can be combined to create a large number of new street view images featuring children and adults. For example, if there are 100 foreground images with the body orientation attribute of "forward" and 50 street view images with the body orientation attribute of "forward", then after extracting any two foreground images with the body orientation attribute of "forward" and the street view images and combining them, 5000 street view images featuring children and adults can be obtained.

[0058] Since the adults in the street view image and the children in the foreground image have the same body orientation attribute, the orientation of the selected pedestrians is determined, and the foreground image with the same orientation is randomly selected according to the orientation result. This ensures that the orientation of the children in the background image is the same as the body orientation attribute of the surrounding pedestrians, and avoids the problem of inconsistency between the foreground image in the composite image content and the content around the pasted area due to inconsistent body orientation attributes.

[0059] The large number of street view images obtained above, including those featuring children and adults, and those showing adults supervising children, can be labeled with features such as the positional relationship, actions, and distance between children and adults in the street view images. This can be used to train a lost child detection model, improving its detection accuracy and reducing the difficulty of acquiring the dataset. When applied to an urban management and monitoring system, the model can detect lost children on various streets, automatically triggering alarms to help parents find their children promptly.

[0060] In step S3.0, since the size ratio of the child in the foreground image to the adult in the street view image is uncertain, it is necessary to scale the foreground image proportionally according to the detection box size of the optimal reference adult target. The foreground image and the corresponding mask image are scaled to ensure that the foreground target is pasted into the background image at a reasonable size, and to ensure that the body proportions of the adult and the child are coordinated.

[0061] In step S4.0, the foreground image is pasted in the area around the best reference adult target in the street view image, so that the child can better integrate into the street view and avoid the foreground image of the child being randomly pasted in various unreasonable positions such as the sky, river, wall, car, etc., further improving the rationality.

[0062] The street view-based child image synthesis method provided in this application, compared with the prior art, synthesizes a new street view image dataset containing both children and adults by extracting any two foreground images of children with the same body orientation attribute and a street view image with adults. This solves the problem of inconsistency between the foreground image and the surrounding environment of the pasted area of ​​the street view image in the synthesized image content, ensuring the naturalness and rationality of the foreground and background. At the same time, the foreground image is scaled proportionally according to the detection box size of the best reference adult target to ensure that the body proportions of adults and children are reasonable. In this way, it can synthesize child images in street view scenes of different countries and cities, and improve the generalization ability of detecting children in various street view scenes.

[0063] In another embodiment of this application, in step S1.0, the body orientation attribute includes at least four categories: forward, backward, left, and right.

[0064] Understandably, in step S1.0, the body orientation attribute of children in the foreground image dataset of children is detected, and the foreground image dataset of children is divided into four categories: forward, backward, left and right based on the detection results; the body orientation attribute of adults in the street view image dataset of adults is detected, and the street view image dataset of adults is divided into four categories: forward, backward, left and right based on the detection results.

[0065] In another embodiment of this application, in step S2.0, the method for detecting the best reference adult target in a street view image dataset containing adults includes the following steps:

[0066] Detect all adult targets in a street view image dataset containing adults;

[0067] The highest-scoring adult target among all adult targets is selected as the best reference adult target.

[0068] It is understandable that each street view image containing adults may contain multiple adult targets. Therefore, selecting the adult target with the highest detection score among all adult targets as the best reference adult target can reduce the probability of misidentification. On the other hand, the human body shape of the best reference adult target is relatively clear and the whole is relatively complete, which is beneficial for scaling the foreground image proportionally as a reference in step S3.0.

[0069] In another embodiment of this application, in step S3.0, the method of scaling the foreground image proportionally according to the detection box size of the optimal reference adult target includes the following steps:

[0070] Obtain the bounding box coordinates of the optimal reference adult target;

[0071] Calculate the width and / or height of the detection box for the optimal reference adult target;

[0072] Scale the foreground image until its width and / or height are in a reasonable ratio to the width and / or height of the detection box of the best reference adult target.

[0073] Furthermore, the method for calculating the reasonable ratio includes the following steps:

[0074] Detect the age of children in the foreground image;

[0075] Set an appropriate ratio based on age.

[0076] Understandably, in the same street view, to avoid a sense of incongruity, the length, width, and height of a child's body should be smaller than that of an adult. Therefore, before pasting the child's foreground image onto the street view image, the child's foreground image needs to be scaled so that the width and / or height of the foreground image are in a reasonable ratio to the width and / or height of the detection box of the best reference adult target.

[0077] To ensure the most reasonable body proportions between children and adults, an age-detection CNN (Convolutional Neural Network) can be used to detect the age of the child in the foreground image. The older the detected child is, the larger the ratio of the child's width and / or height to that of the adult, but the ratio should not exceed 1; the younger the detected child is, the smaller the ratio of the child's width and / or height to that of the adult, but the ratio should be less than 0.2.

[0078] In another embodiment of this application, after scaling the foreground image proportionally according to the detection box size of the optimal reference adult target in step S3.0, the following step is further included:

[0079] Perform ground segmentation processing on the street view image to segment out the ground region;

[0080] Based on the shape and size of the mask image of the scaled foreground image, candidate regions of the same shape and size are randomly selected around the detection box of the best reference adult target in the street view image.

[0081] Determine if the bottom of the foreground pasting candidate area is within the ground area;

[0082] If not, then randomly select a foreground pasting candidate region of the same shape and size around the detection box of the best reference adult target in the street view image, and re-determine whether the bottom of the foreground pasting candidate region is within the ground area; if so, then select the current foreground pasting candidate region as the pasting region of the foreground image.

[0083] Furthermore, after selecting the current foreground pasting candidate region as the pasting region for the foreground image, the following steps are also included:

[0084] The intersection and union ratio (OCR) of the pasted area with the detection box of the best reference adult target is calculated.

[0085] If the intersection-union ratio is less than the threshold, the final pasting region is determined; otherwise, a foreground pasting candidate region of the same shape and size is randomly selected around the detection box of the best reference adult target in the street view image, and the bottom of the foreground pasting candidate region is re-determined to be within the ground area.

[0086] Understandably, the first step is to determine if the bottom of the foreground pasting candidate region is within the ground area. This helps detect if the child in the foreground image is located on a road in the street view, preventing the child from standing on a car, rooftop, or in the sky, thus ensuring the child's position is reasonable. Next, the intersection-union (IoU) ratio of the pasting region to the detection bounding box of the best reference adult target is calculated to ensure that the foreground image, after being pasted into the street view image, does not cause large-scale occlusion of the adult target within the street view image. These steps are iterated repeatedly until a reasonable pasting region for the foreground image is obtained in the street view image, allowing the child's foreground image to seamlessly integrate into the street view image containing adults.

[0087] In another embodiment of this application, after scaling the foreground image proportionally according to the detection box size of the optimal reference adult target in step S3.0, the following step is further included:

[0088] Perform an erosion operation on the mask image of the foreground image;

[0089] The mask image of the foreground image after the erosion operation is subjected to Gaussian blur.

[0090] It is understandable that performing an erosion operation on the mask image of the foreground image can optimize the edge parts of the mask image; performing a Gaussian blur on the mask image of the foreground image after the erosion operation further optimizes the edge parts of the mask image, resulting in a more natural transition of the edge parts after the foreground image is pasted and composited.

[0091] In another embodiment of this application, after scaling the foreground image proportionally according to the detection box size of the optimal reference adult target in step S3.0, the following step is further included:

[0092] Select the foreground image in the paste area of ​​the street view image;

[0093] The pasted area is expanded and cropped in the street view image at a certain ratio with the pasted area as the center to obtain a local background image containing the pasted area;

[0094] The color and illumination information of the local background image are calculated, and an adaptive thresholding method is used to adjust the color and illumination information of the foreground image to match the local background image.

[0095] Understandably, by expanding and cropping the pasted area in the street view image proportionally around the pasted area, a local background image containing the pasted area is obtained. The color and lighting information of the pasted area can then be accurately calculated. An adaptive thresholding method is used to adjust the color and lighting information of the foreground image to match the local background image, achieving local harmony between the foreground and background. This local harmony method avoids global processing of the background image, significantly reducing computational load and increasing computational speed. Furthermore, by using local information from the foreground pasted area to harmonize and match the foreground's color and lighting information to adapt to the current pasted area, without altering the global distribution of color and lighting information in the background image, the synthesized image becomes more realistic and natural.

[0096] Further, in step S4.0, the method of pasting the foreground image into the area surrounding the optimal reference adult target in the street view image includes the following steps:

[0097] Paste the foreground image with adjusted color and lighting information, along with its mask, into the pasting area of ​​the street view image;

[0098] Obtain street view images with both children and adults.

[0099] Understandably, the pasting method uses OpenCV functions to perform the pasting operation, ultimately resulting in a relatively realistic composite image of a child in a street scene.

[0100] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for synthesizing children's images based on street view, characterized in that, include: Detect the body orientation attributes of children in a foreground image dataset of children, and detect the best reference adult target and its body orientation attributes in a street view image dataset with adults; Extract any two foreground images and street view images with the same body orientation attribute; The foreground image is scaled proportionally based on the detection bounding box size of the optimal reference adult target, including the following steps: Ground segmentation is performed on the street view image to segment the ground region. Based on the shape and size of the mask image of the scaled foreground image, foreground pasting candidate regions of the same shape and size are randomly selected around the detection box of the best reference adult target in the street view image. It is determined whether the bottom of the foreground pasting candidate region is within the ground region. If not, foreground pasting candidate regions of the same shape and size are randomly selected again around the detection box of the best reference adult target in the street view image, and the bottom of the foreground pasting candidate regions is re-determined. If yes, the current foreground pasting candidate region is selected as the pasting region of the foreground image. The intersection-union ratio (IUR) of the pasting region and the detection box of the best reference adult target is calculated. If the IUR is less than the threshold, the pasting region is finally determined. If not, foreground pasting candidate regions of the same shape and size are randomly selected again around the detection box of the best reference adult target in the street view image, and the bottom of the foreground pasting candidate regions is re-determined. Paste the foreground image into the area around the best reference adult target in the street view image.

2. The street view-based child image synthesis method as described in claim 1, characterized in that, The body orientation attributes include at least four categories: forward, backward, left, and right.

3. The street view-based child image synthesis method as described in claim 1, characterized in that, The method for detecting the best reference adult target in a street view image dataset containing adults includes the following steps: Detect all adult targets in a street view image dataset containing adults; The highest-scoring adult target among all adult targets is selected as the best reference adult target.

4. The street view-based child image synthesis method as described in claim 1, characterized in that, The method for scaling the foreground image proportionally based on the detection box size of the optimal reference adult target includes the following steps: Obtain the bounding box coordinates of the optimal reference adult target; Calculate the width and / or height of the detection box for the optimal reference adult target; Scale the foreground image until its width and / or height are in a reasonable ratio to the width and / or height of the detection box of the best reference adult target.

5. The street view-based child image synthesis method as described in claim 4, characterized in that, The method for calculating the reasonable ratio includes the following steps: Detect the age of children in the foreground image; Set an appropriate ratio based on age.

6. The method for synthesizing children's images based on street view as described in any one of claims 1 to 5, characterized in that, After scaling the foreground image proportionally to the detection box size of the optimal reference adult target, the following steps are also included: Perform an erosion operation on the mask image of the foreground image; The mask image of the foreground image after the erosion operation is subjected to Gaussian blur.

7. The street view-based child image synthesis method as described in claim 6, characterized in that, After scaling the foreground image proportionally to the detection box size of the optimal reference adult target, the following steps are also included: Select the foreground image in the paste area of ​​the street view image; The pasted area is expanded and cropped in the street view image at a certain ratio with the pasted area as the center to obtain a local background image containing the pasted area; The color and illumination information of the local background image are calculated, and an adaptive thresholding method is used to adjust the color and illumination information of the foreground image to match the local background image.

8. The method for synthesizing children's images based on street view as described in claim 7, characterized in that, The method for pasting a foreground image into the area surrounding the best reference adult target in a street view image includes the following steps: Paste the foreground image with adjusted color and lighting information, along with its mask, into the pasting area of ​​the street view image; Obtain street view images with both children and adults.

Images