A sample image generation method and device, and an autonomous driving device

Abstract

Embodiments of the present application provide a sample image generation method and device and an automatic driving device, and relate to the technical field of image processing. The method comprises: obtaining a road image of an actual scene, and determining a superimposition region in the road image for superimposing an obstacle; obtaining, according to the road image, illumination information of the actual scene and parameters of a camera for collecting the road image; generating a virtual scene including an obstacle model based on the parameters of the camera, the illumination information and region information of the superimposition region; rendering the virtual scene to obtain an obstacle image; and performing image fusion on the road image and the obstacle image with the road image as a background image and the obstacle image as a foreground image to obtain a sample image. The sample image generation scheme provided by the embodiments of the present application can generate a sample image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a sample image generation method, apparatus, and autonomous driving device. Background Technology

[0002] Road maintenance, fallen objects, and other factors can cause obstacles to appear on the road, posing safety hazards to vehicles. To improve driving safety, obstacle detection can be performed on images captured by vehicle-mounted image acquisition equipment and roadside image acquisition equipment to promptly detect obstacles in the road, reminding drivers to avoid them in advance or alerting cleaning personnel to remove them.

[0003] In related technologies, obstacle detection models are often used to detect obstacles. To improve the detection accuracy of these models, they need to be pre-trained using a large number of sample road images containing obstacles. However, since collecting such a large number of sample road images is difficult, a sample image generation scheme is needed to generate sample road images containing obstacles. Summary of the Invention

[0004] The purpose of this invention is to provide a sample image generation method, apparatus, and autonomous driving device to generate sample images. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of the present invention provide a sample image generation method, the method comprising:

[0006] Obtain a road image of the actual scene and determine the stacking area in the road image for stacking obstacles;

[0007] Based on the road image, obtain the lighting information of the actual scene and the parameters of the camera that acquired the road image;

[0008] Based on the camera parameters, lighting information, and area information of the overlay area, a virtual scene including obstacle models is generated. The position of the obstacle models in the virtual scene is determined according to the camera parameters and the area information.

[0009] The virtual scene is rendered to obtain obstacle images;

[0010] Using the road image as the background image and the obstacle image as the foreground image, the road image and the obstacle image are fused to obtain a sample image.

[0011] In one embodiment of the present invention, the road image and the obstacle image are the same size;

[0012] The step of fusing the road image as the background image and the obstacle image as the foreground image to obtain a sample image includes:

[0013] In the obstacle image, determine the foreground region where the obstacle is located, and in the road image, determine the background region outside the same location as the foreground region;

[0014] Obtain a sample image in which the foreground region and the background region are fused according to their positions.

[0015] In one embodiment of the present invention, determining the foreground region where the obstacle is located in the obstacle image and determining the background region outside the same location in the road image includes:

[0016] In the obstacle image, determine the foreground region where the obstacle is located and the first edge region adjacent to the foreground region;

[0017] In the road image, a second edge region is determined at the same location as the first edge region, and a background region is determined in the road image outside the same location as the foreground region and the first edge region;

[0018] Obtaining a sample image in which the foreground region and the background region are fused by position includes:

[0019] Based on the distance weighting coefficient, the first edge region and the second edge region are fused to obtain a sample image containing the foreground region, the background region, and the fused region. The distance weighting coefficient is determined based on the distance between each pixel in the first edge region and the edge pixels in the foreground region.

[0020] In one embodiment of the present invention, determining the foreground region where the obstacle is located in the obstacle image and determining the background region outside the same location in the road image includes:

[0021] Generate an obstacle mask map including a first region, wherein the first region reflects the area where the obstacle is located in the obstacle image;

[0022] Based on the obstacle mask map, determine the foreground region where the obstacle is located in the obstacle image;

[0023] Based on the obstacle mask map, a background region outside the same location as the foreground region is determined in the road image.

[0024] In one embodiment of the present invention, the method further includes:

[0025] The edges of the first region in the obstacle mask image are expanded.

[0026] The expanded regions resulting from the expansion process are blurred.

[0027] The step of determining the foreground region where the obstacle is located in the obstacle image based on the obstacle mask image includes:

[0028] In the obstacle image, a third region is identified that is at the same location as the first region, and a fourth region is identified that is at the same location as the inflated region;

[0029] The step of determining the background region outside the same location as the foreground region in the road image based on the obstacle mask map includes:

[0030] In the road image, a fifth region is identified that is at the same location as the second region, and a sixth region is identified that is at the same location as the expansion region, wherein the second region is the region in the obstacle image other than the first region and the expansion region;

[0031] Obtaining a sample image in which the foreground region and the background region are fused by position includes:

[0032] Using the weighting coefficients determined by the pixel values ​​of each pixel in the expansion region, the fourth and sixth regions are fused to obtain a sample image containing the third region, the fifth region, and the fused region.

[0033] In one embodiment of the present invention, the blurring of the expanded region resulting from the expansion process includes:

[0034] The pixel values ​​of the pixels in the expansion region are adjusted based on the distance between the pixels in the expansion region and the edge pixels in the first region, the pixel values ​​of the pixels in the first region, and the pixel values ​​of the pixels in the second region.

[0035] In one embodiment of the present invention, generating a virtual scene including an obstacle model based on the camera parameters, lighting information, and area information of the overlay area includes:

[0036] Based on the camera parameters and the lighting information, a temporary virtual scene is constructed;

[0037] The obstacle model is superimposed on the target area in the temporary virtual scene to obtain the superimposed virtual scene, wherein the target area is determined according to the camera parameters and the area information of the superimposed area.

[0038] In one embodiment of the present invention, determining the stacking area in the road image for stacking obstacles includes:

[0039] Detect the road region and the object region containing a preset type of object in the road image;

[0040] From the regions within the road image that belong to the road region but not to the object region, determine the stacking region for stacking obstacles.

[0041] In one embodiment of the present invention, before performing image fusion on the road image and the obstacle image, the method further includes:

[0042] Based on the pixel values ​​of each pixel in the foreground region where the obstacle is located in the obstacle image, and the pixel values ​​of each pixel in the corresponding region of the foreground region in the road image, tone mapping is performed on each pixel in the foreground region.

[0043] In one embodiment of the present invention, the step of performing tone mapping processing on each pixel in the foreground region based on the pixel values ​​of each pixel in the foreground region where the obstacle is located in the obstacle image and the corresponding region in the road image includes:

[0044] Calculate the first pixel mean of each pixel in the foreground region where the obstacle is located in the obstacle image;

[0045] Calculate the second pixel mean of each pixel in the region corresponding to the foreground region in the road image;

[0046] The pixel values ​​of each pixel in the foreground region are adjusted based on the ratio of the first pixel mean to the second pixel mean.

[0047] In one embodiment of the present invention, the step of fusing the road image as a background image and the obstacle image as a foreground image to obtain a sample image includes:

[0048] Using the road image as the background image and the obstacle image as the foreground image, the road image and the obstacle image are fused to obtain a first intermediate image;

[0049] The first intermediate image is blurred as a whole to obtain the processed sample image.

[0050] In one embodiment of the present invention, the step of performing overall blurring processing on the first intermediate image to obtain the processed sample image includes:

[0051] The first intermediate image is blurred as a whole to obtain the processed second intermediate image;

[0052] Based on the image parameters of the road image, the image parameters of the second intermediate image are adjusted to obtain the adjusted sample image.

[0053] In one embodiment of the present invention, the method further includes:

[0054] Obtain at least one of the following information as annotation information for the sample image:

[0055] The sample image contains information about the region where the obstacle is located and information about the object within the obstacle.

[0056] Secondly, embodiments of the present invention also provide a sample image generation apparatus, the apparatus comprising:

[0057] The image acquisition module is used to acquire road images of the actual scene and determine the stacking area in the road image for stacking obstacles;

[0058] The information acquisition module is used to obtain the illumination information of the actual scene and the parameters of the camera that acquired the road image based on the road image;

[0059] The scene generation module is used to generate a virtual scene including obstacle models based on the parameters of the camera, lighting information and area information of the overlay area. The position of the obstacle models in the virtual scene is determined according to the parameters of the camera and the area information.

[0060] The scene rendering module is used to render the virtual scene to obtain obstacle images;

[0061] The image fusion module is used to fuse the road image as the background image and the obstacle image as the foreground image to obtain a sample image.

[0062] In one embodiment of the present invention, the road image and the obstacle image are the same size;

[0063] The image fusion module includes:

[0064] The region determination submodule is used to determine the foreground region where the obstacle is located in the obstacle image, and to determine the background region outside the same location as the foreground region in the road image;

[0065] The location fusion submodule is used to obtain a sample image in which the foreground region and the background region are fused according to their locations.

[0066] In one embodiment of the present invention, the region determination submodule is specifically used for:

[0067] In the obstacle image, a foreground region where the obstacle is located and a first edge region adjacent to the foreground region are determined; in the road image, a second edge region at the same position as the first edge region is determined, and a background region outside the same position as the foreground region and the first edge region is determined in the road image;

[0068] The location fusion submodule is specifically used for:

[0069] Based on the distance weighting coefficient, the first edge region and the second edge region are fused to obtain a sample image containing the foreground region, the background region, and the fused region. The distance weighting coefficient is determined based on the distance between each pixel in the first edge region and the edge pixels in the foreground region.

[0070] In one embodiment of the present invention, the region determination submodule includes:

[0071] A mask image generation unit is used to generate an obstacle mask image including a first region, wherein the first region reflects the region where the obstacle is located in the obstacle image;

[0072] A foreground determination unit is used to determine the foreground region where the obstacle is located in the obstacle image based on the obstacle mask image;

[0073] The background determination unit is used to determine a background region outside the same location as the foreground region in the road image based on the obstacle mask map.

[0074] In one embodiment of the present invention, the apparatus further includes:

[0075] An expansion processing module is used to expand the edges of the first region in the obstacle mask image;

[0076] The fuzzing module is used to fuzz the expanded regions generated by the dilation process.

[0077] The foreground determination unit is specifically used to determine, in the obstacle image, a third region that is at the same position as the first region and a fourth region that is at the same position as the dilated region;

[0078] The background determination unit is specifically used to determine, in the road image, a fifth region at the same location as the second region and a sixth region at the same location as the dilated region, wherein the second region is the region in the obstacle image other than the first region and the dilated region;

[0079] The location fusion submodule is specifically used to fuse the fourth region and the sixth region using weight coefficients determined by the pixel values ​​of each pixel in the dilated region, to obtain a sample image containing the third region, the fifth region, and the fused region.

[0080] In one embodiment of the present invention, the blurring module is specifically used to adjust the pixel value of the pixel in the expansion region based on the distance between the pixel in the expansion region and the edge pixel in the first region, the pixel value of the pixel in the first region, and the pixel value of the pixel in the second region.

[0081] In one embodiment of the present invention, the scene generation module is specifically used to construct a temporary virtual scene based on the camera parameters and the lighting information; and to overlay an obstacle model onto a target area in the temporary virtual scene to obtain the overlaid virtual scene, wherein the target area is determined according to the camera parameters and the area information of the overlaid area.

[0082] In one embodiment of the present invention, the image acquisition module is specifically used to acquire a road image of an actual scene; detect the road region and the object region where a preset type of object is located in the road image; and determine a stacking region for stacking obstacles from the regions in the road image that belong to the road region but do not belong to the object region.

[0083] In one embodiment of the present invention, the apparatus further includes:

[0084] The tone mapping module is used to perform tone mapping processing on each pixel in the foreground region based on the pixel values ​​of each pixel in the foreground region where the obstacle is located in the obstacle image and the pixel values ​​of each pixel in the corresponding region of the foreground region in the road image before performing image fusion on the road image and the obstacle image.

[0085] In one embodiment of the present invention, the tone mapping module is specifically used for:

[0086] Calculate the first pixel mean of each pixel in the foreground region where the obstacle is located in the obstacle image; calculate the second pixel mean of each pixel in the road image corresponding to the foreground region; adjust the pixel value of each pixel in the foreground region based on the ratio of the first pixel mean to the second pixel mean.

[0087] In one embodiment of the present invention, the image fusion module includes:

[0088] The image fusion submodule is used to perform image fusion on the road image as the background image and the obstacle image as the foreground image to obtain a first intermediate image;

[0089] The overall blur submodule is used to perform overall blurring on the first intermediate image to obtain the processed sample image.

[0090] In one embodiment of the present invention, the overall blurring submodule is specifically used to perform overall blurring processing on the first intermediate image to obtain a processed second intermediate image; and to adjust the image parameters of the second intermediate image based on the image parameters of the road image to obtain an adjusted sample image.

[0091] In one embodiment of the present invention, the apparatus further includes:

[0092] The sample annotation module is used to obtain at least one of the region information of the area where the obstacle is located and the item information of the obstacle in the sample image, as annotation information of the sample image.

[0093] Thirdly, embodiments of the present invention also provide an autonomous driving device, including a processor, wherein the processor runs an obstacle detection model, the obstacle detection model being trained based on sample images generated by any of the method steps described in the first aspect above.

[0094] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods described in the first aspect above.

[0095] Beneficial effects of the embodiments of the present invention:

[0096] As can be seen from the above, when generating sample images using the scheme provided in this embodiment of the invention, a virtual scene including obstacle models is generated based on the obtained camera parameters, lighting information, and area information of the overlay area. Since the camera parameters and lighting information are obtained from road images, the virtual scene can be regarded as a simulated actual scene. Rendering the virtual scene in this way can obtain obstacle images with high realism. Image fusion of road images and obstacle images can not only obtain sample images, but these sample images can also be regarded as images obtained from roads with obstacles placed in the actual scene. Therefore, the obtained sample images have high realism. Training the obstacle detection model based on the sample images with high realism can improve the training efficiency and recall rate of the obstacle detection model. Attached Figure Description

[0097] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0098] Figure 1a A schematic flowchart of the first sample image generation method provided in an embodiment of the present invention;

[0099] Figure 1b An obstacle image provided in an embodiment of the present invention;

[0100] Figure 1c A road image provided in an embodiment of the present invention;

[0101] Figure 1d A sample image provided for an embodiment of the present invention;

[0102] Figure 2 A schematic flowchart of the second sample image generation method provided in an embodiment of the present invention;

[0103] Figure 3 A schematic flowchart of the third sample image generation method provided in the embodiments of the present invention;

[0104] Figure 4 A schematic flowchart of the fourth sample image generation method provided in the embodiments of the present invention;

[0105] Figure 5 A schematic flowchart of the fifth sample image generation method provided in this embodiment of the invention;

[0106] Figure 6a A schematic flowchart illustrating the sixth sample image generation method provided in this embodiment of the invention;

[0107] Figure 6b An obstacle mask image provided for an embodiment of the present invention;

[0108] Figure 7a A flowchart illustrating the seventh sample image generation method provided in this embodiment of the invention;

[0109] Figure 7b An obstacle mask image before blurring is provided in an embodiment of the present invention;

[0110] Figure 7c An obstacle mask image after blurring is provided in an embodiment of the present invention;

[0111] Figure 8A flowchart illustrating the eighth sample image generation method provided in this embodiment of the invention;

[0112] Figure 9 A flowchart illustrating the ninth sample image generation method provided in this embodiment of the invention;

[0113] Figure 10 A flowchart illustrating the tenth sample image generation method provided in this embodiment of the invention;

[0114] Figure 11 A flowchart illustrating the eleventh sample image generation method provided in this embodiment of the invention;

[0115] Figure 12 A schematic flowchart of the twelfth sample image generation method provided in this embodiment of the invention;

[0116] Figure 13 A flowchart illustrating the thirteenth sample image generation method provided in this embodiment of the invention;

[0117] Figure 14a A flowchart illustrating the fourteenth sample image generation method provided in this embodiment of the invention;

[0118] Figure 14b A schematic diagram illustrating the workflow of a fusion module provided in an embodiment of the present invention;

[0119] Figure 15 This is a schematic diagram of the structure of the first sample image generation device provided in an embodiment of the present invention;

[0120] Figure 16 This is a schematic diagram of the structure of the second sample image generation device provided in an embodiment of the present invention;

[0121] Figure 17 This is a schematic diagram of the structure of the third sample image generation device provided in the embodiments of the present invention;

[0122] Figure 18 This is a schematic diagram of the structure of the fourth sample image generation device provided in the embodiments of the present invention;

[0123] Figure 19 This is a schematic diagram of the structure of the fifth sample image generation device provided in the embodiments of the present invention;

[0124] Figure 20 This is a schematic diagram of the sixth sample image generation device provided in an embodiment of the present invention;

[0125] Figure 21 This is a schematic diagram of the structure of the seventh sample image generation device provided in the embodiments of the present invention;

[0126] Figure 22 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0127] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.

[0128] See Figure 1a , Figure 1a This is a flowchart illustrating the first sample image generation method provided in an embodiment of the present invention. The method includes the following steps S101-S105.

[0129] Step S101: Obtain a road image of the actual scene and determine the stacking area in the road image for stacking obstacles.

[0130] The aforementioned road images are images from actual scenes, i.e., images obtained by collecting images of real roads.

[0131] The aforementioned obstacles can be items such as tires, bricks, trash cans, cardboard boxes, water pipes, and buckets.

[0132] Specifically, after obtaining the aforementioned road image, the stackable area in the road image can be determined, and then the stacking area for stacking obstacles can be determined within the determined stackable area.

[0133] There are various ways to obtain the aforementioned road images. For example, one can obtain road images captured by a camera, or obtain road images from a repository containing road images. Alternatively, road images can be obtained through other means, which are not limited in this embodiment of the invention.

[0134] In one embodiment of the present invention, the stackable area in the road image can be determined by any of the following three implementation methods, thereby determining the stacking area for stacking obstacles.

[0135] In the first implementation, non-overlayable areas in the road image can be detected, such as the area where vehicles are located, the area where green belts are located, and the area where the sky is located. After detecting the non-overlayable areas, the areas in the road image other than the non-overlayable areas can be determined as overlayable areas. Then, the overlay area for overlaying obstacles can be determined in the determined overlayable areas.

[0136] In the second implementation, since obstacles usually appear on roads, road surface detection can be performed on the road image to obtain the area where the road is located in the road image, and the area where the road is located can be determined as a stackable area. Then, a stacking area for stacking obstacles can be determined in the determined stackable area.

[0137] In the third implementation method, it can also be achieved through subsequent... Figure 3 In the illustrated embodiment, steps S101B-S101C determine the stacking area for stacking obstacles.

[0138] Step S102: Based on the road image, obtain the lighting information of the actual scene and the parameters of the camera that acquired the road image.

[0139] The aforementioned lighting information may include information such as light intensity and light angle.

[0140] The camera parameters mentioned above can include the camera's intrinsic and extrinsic parameters. The camera's intrinsic parameters include the camera's pitch angle, circumferential angle, and the distance between the camera's optical center and the corresponding position of the image center pixel, as well as the camera's focal length and field of view.

[0141] Specifically, illumination estimation can be performed on the aforementioned road images to obtain illumination information for the actual scene. In addition, camera parameter estimation can be performed on the aforementioned road images to obtain the parameters of the camera that acquired the road images.

[0142] Illumination estimation and camera parameter estimation of road images can be achieved using existing estimation algorithms and deep learning-based estimation models, which will not be detailed here.

[0143] For example, when estimating camera parameters in road images, this can be achieved using existing methods for calibrating camera intrinsic and extrinsic parameters.

[0144] Step S103: Based on the camera parameters, lighting information, and area information of the overlay area, generate a virtual scene including obstacle models.

[0145] The position of the obstacle model in the virtual scene is determined based on the camera parameters and area information.

[0146] The area information of the aforementioned overlay area may include the location and size of the overlay area in the road image.

[0147] The virtual scene described above can be viewed as a three-dimensional scene simulating the actual scene described above. The camera parameters are used to simulate the perspective effect of the actual scene described above. The lighting information is used to simulate the lighting and shadow effects of the actual scene described above. The area information of the overlay area is used to determine the position of the obstacle model in the virtual scene.

[0148] In one embodiment of the present invention, a virtual scene including an obstacle model can be generated by any of the following two implementation methods.

[0149] In the first implementation method, it can be achieved through subsequent... Figure 9 In the illustrated embodiment, steps S103A-S103B generate a virtual scene including obstacles and models, which will not be described in detail here.

[0150] In the second implementation, obstacle models can be predetermined. Based on the camera parameters and the area information of the stacked area, the position of the obstacle model in the virtual scene can be determined. Thus, based on the camera parameters, lighting information, area information of the stacked area, the determined obstacle model, and the position of the obstacle model, a virtual scene with the obstacle model at the determined position can be generated.

[0151] In one embodiment of the present invention, when determining the obstacle model, a user-built model can be obtained and used as the obstacle model. Alternatively, an obstacle model of a preset obstacle category can be selected from a pre-built obstacle model database. The models included in the obstacle model database can be user-built or downloaded from the internet. The models stored in the obstacle model database can be, but are not limited to, models in formats such as .obj, .fbx, .MD2, and .MD3. The obstacle model database can store, but is not limited to, models of road obstacles such as tires, bricks, trash cans, cardboard boxes, water pipes, and buckets.

[0152] When determining the aforementioned preset types, obstacle types that the obstacle recognition model has a weaker ability to recognize can be identified as preset types. For example, if the obstacle recognition model has a weak ability to recognize cardboard boxes, then cardboard boxes can be identified as a preset type.

[0153] The above preset types can also be any obstacle type specified by the user.

[0154] In one embodiment of the present invention, since the virtual scene is usually a three-dimensional scene, when determining the position of the obstacle model in the virtual scene, the two-dimensional position information of the stacked area in the road image can be obtained. Then, combined with the camera parameters, the obtained two-dimensional position information is converted into three-dimensional position information in the virtual scene, thereby determining the position of the obstacle in the virtual scene.

[0155] Step S104: Render the virtual scene to obtain obstacle images.

[0156] The size of the obstacle image can be the same as or different from the size of the road image.

[0157] Specifically, after obtaining the aforementioned virtual scene, existing rendering techniques can be used to render the virtual scene and obtain obstacle images.

[0158] The virtual scene described above can be seen as a three-dimensional scene simulating the actual scene described above. Therefore, rendering the virtual scene can produce an image with a realism that is close to that of the image in the real scene. Thus, the obstacle image described above can be considered as a virtual image data with a realism that reaches the level of a photograph.

[0159] In one embodiment of the present invention, a virtual scene can be rendered using ray tracing rendering technology to obtain an obstacle image.

[0160] Compared to rasterization rendering, ray tracing rendering can simulate the physical principles of real lighting and imaging, thereby improving the realism of the rendered image.

[0161] In addition, the location of obstacle models in the virtual scene is fixed. Therefore, the annotation information of obstacle models can be generated based on the location information of obstacle models. After the obstacle image is rendered, the area where the obstacle is located can be determined in the obstacle image based on the annotation information of the obstacle models.

[0162] Figure 1b This is an obstacle image obtained by rendering a virtual scene. Figure 1b As can be seen, the area where the obstacle is located is the irregular region at the bottom of the image.

[0163] Step S105: Using the road image as the background image and the obstacle image as the foreground image, perform image fusion on the road image and the obstacle image to obtain a sample image.

[0164] When the obstacle image and the road image are of different sizes, the correspondence between the pixels in the superimposed area of ​​the road image and the pixels in the obstacle image can be obtained. Then, based on the correspondence, the pixel values ​​of the pixels in the obstacle area of ​​the obstacle image are mapped to the corresponding pixels in the road image to obtain the mapped road image, which is used as a sample image.

[0165] Given that the obstacle image and the road image are the same size, subsequent... Figure 4 In the illustrated embodiment, steps S105A-S105B fuse the road image and the obstacle image, which will not be described in detail here.

[0166] Figure 1c This is a road image from a real-world scenario. Figure 1d For Figure 1c The road image shown is a background image. Figure 1b The obstacle image shown is the foreground image. Figure 1b and Figure 1c Sample images obtained by image fusion Figure 1d In the image, the foreground is the area where the obstacle is located in the third lane on the left, and the background is the area other than the area where the obstacle is located.

[0167] As can be seen from the above, when generating sample images using the scheme provided in this embodiment of the invention, a virtual scene including obstacle models is generated based on the obtained camera parameters, lighting information, and area information of the overlay area. Since the camera parameters and lighting information are obtained from road images, the virtual scene can be regarded as a simulated actual scene. Rendering the virtual scene in this way can obtain obstacle images with high realism. Image fusion of road images and obstacle images can not only obtain sample images, but these sample images can also be regarded as images obtained from roads with obstacles placed in the actual scene. Therefore, the obtained sample images have high realism. Training the obstacle detection model based on the sample images with high realism can improve the training efficiency and recall rate of the obstacle detection model.

[0168] Furthermore, the obstacle model mentioned above can be any obstacle model in the obstacle model library or an obstacle model manually constructed by the user. This allows for the generation of various sample images containing different obstacles. These sample images can then be used to train the obstacle detection model, improving its ability to detect various obstacles. Alternatively, a large number of sample images containing specific types of obstacles can be generated based on the training requirements of the obstacle detection model. This allows for targeted training of the model based on the generated sample images, improving its ability to detect specific types of obstacles. Since the background of the sample images comes from actual road images, background modeling is unnecessary, eliminating background modeling costs. Moreover, the acquisition of these road images is relatively easy, allowing for the acquisition of a large number of images of various road types. The sample images are then fused with the road images and obstacle images. Training the obstacle recognition model based on these sample images improves the model's ability to recognize obstacles on various roads. Furthermore, the sample images are obtained simply by fusing the road images and obstacle images, eliminating the need for manual acquisition of sample images or the need to manually simulate roads with obstacles. Therefore, the sample image generation scheme provided in this embodiment of the invention reduces the difficulty and risk of acquiring sample images. The obstacle model generated using the above method has stronger detection capabilities, and when applied to fields such as autonomous driving and navigation, the obstacle avoidance capabilities of autonomous driving devices will be stronger.

[0169] After obtaining the above sample images, the annotation information of the above sample images can be further obtained.

[0170] In one embodiment of the present invention, see Figure 2 The flowchart of the second sample image generation method is provided. In this embodiment, the above method further includes the following step S106.

[0171] Step S106: Obtain at least one of the region information of the area where the obstacle is located and the item information of the obstacle in the sample image, as the annotation information of the sample image.

[0172] The item information of the aforementioned obstacles may include information such as obstacle type, size, and color.

[0173] The obstacle model can be predetermined, therefore, the item information of the obstacle corresponding to the obstacle model can be obtained in advance.

[0174] From the above Figure 1a As can be seen from the description of step S104 in the illustrated embodiment, after rendering the virtual scene to obtain the obstacle image, the area where the obstacle is located can be determined in the obstacle image based on the annotation information of the obstacle model in the virtual scene.

[0175] As can be seen from the above, when generating sample images using the solution provided in this embodiment of the invention, at least one of the region information of the area where the obstacle is located and the item information of the obstacle can be obtained as annotation information for the sample image. Using labeled sample images to detect obstacles can improve the training effect of the obstacle detection model. Furthermore, this solution eliminates the need for manual annotation of sample images, thereby reducing the annotation cost.

[0176] When determining the stacking area for overlaying obstacles in road images, in addition to the aforementioned methods... Figure 1a In addition to the method mentioned in step S101 of the illustrated embodiment, it can also be done through the following Figure 3 In the illustrated embodiment, steps S101B-101C determine the stacking area of ​​the stacked obstacles.

[0177] In one embodiment of the present invention, see Figure 3 The flowchart of the third sample image generation method is provided. In this embodiment, the above step S101 can be achieved through the following steps S101A-S101C.

[0178] Step S101A: Obtain road images of the actual scene.

[0179] For details on how to obtain road images, please refer to the aforementioned methods. Figure 1a The description of step S101 in the illustrated embodiment will not be repeated here.

[0180] Step S101B: Detect the road region and the object region containing the preset type of object in the road image.

[0181] Since obstacles usually refer to objects that appear on the road, rather than objects that appear in non-road areas such as vehicles, green belts, or the sky, the objects of the above preset type can be vehicles, green belts, the sky, etc., and the area where the objects of the preset type are located can be understood as a non-stackable area.

[0182] Specifically, object detection can be performed on road images to detect the road region where the road is located and the object region where objects of a preset type are located.

[0183] Step S101C: Determine the stacking area for stacking obstacles from the areas within the road image that belong to the road area but not to the object area.

[0184] Specifically, since a road region may include one or more object regions, such as the area where a vehicle is located, after detecting the road region and the object region, the superimposed regions that belong to the road region but do not belong to the object region can be determined in the road image, and then the overlapping region can be determined in the determined region.

[0185] In one embodiment of the present invention, after determining the superimposed area, the superimposed area can be determined in the superimposed area according to the preset selection conditions.

[0186] The above selection criteria can be set according to various aspects such as area, shape, and location. For example, the area of ​​the superimposed area is greater than the preset area, the superimposed area is a rectangular or circular area, the boundary of the superimposed area intersects with the upper boundary of the superimposed area, etc.

[0187] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, the road area and the object area where the preset type of object is located in the road image are detected. Since obstacles usually refer to items appearing on the road, the above-mentioned object area can be regarded as a non-overlapping area in the road image. In this way, the area in the road image that belongs to the road area but does not belong to the object area is used to determine the stacking area for stacking obstacles, which conforms to the logic of the actual scene, thereby improving the realism of the generated sample image.

[0188] The following section explains the specific implementation method for fusing road and obstacle images when the road and obstacle images are the same size.

[0189] In one embodiment of the present invention, see Figure 4The flowchart of the fourth sample image generation method is provided. In this embodiment, the above step S105 can be implemented through the following steps S105A-S105B.

[0190] Step S105A: Determine the foreground region where the obstacle is located in the obstacle image, and determine the background region outside the same location in the road image.

[0191] Specifically, as described in the foregoing embodiments, after rendering the virtual scene to obtain obstacle images, the foreground region where the obstacle is located can be determined in the obstacle image based on the annotation information of the obstacle model in the virtual scene. Since the obstacle image and the road image are the same size, a corresponding region with the same position as the foreground region can be determined in the road image, and then the background region other than the determined corresponding region can be determined in the road image.

[0192] Step S105B: Obtain a sample image with foreground and background regions fused by position.

[0193] In the first implementation, for each pixel in the corresponding region, the pixel value of that pixel can be adjusted to the pixel value of the pixel in the foreground region that is at the same position as that pixel, so as to obtain the adjusted background image, which can be used as a sample image.

[0194] In the second implementation, for each pixel in the region corresponding to the background region in the obstacle image, the pixel value of that pixel can be adjusted to the pixel value of the pixel in the background region that is at the same position as that pixel, to obtain the adjusted obstacle image, which can be used as a sample image.

[0195] In the third implementation, an image containing a first fusion region and a second fusion region can also be generated as a sample image. The first fusion region has the same regional information as the foreground region, such as position, size, and pixel values ​​of the included pixels, and the second fusion region has the same regional information as the background region.

[0196] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, if the obstacle image and the road image are the same size, the foreground region where the obstacle is located is determined in the obstacle image, and the background region outside the same position of the foreground region is determined in the road image. In this way, the road image and the obstacle image can be accurately fused based on the positions of the foreground region and the background region to obtain a sample image fused with the foreground region and the background region.

[0197] To make the generated sample images more natural and avoid abrupt and unrealistic edges, the boundary between the foreground and background areas can be blurred to create a gradient at the boundary, thereby improving the realism of the sample images.

[0198] In one embodiment of the present invention, see Figure 5 The present invention provides a flowchart of the fifth sample image generation method. In this embodiment, step S105A can be implemented by the following steps S105A1-S105A2, and step S105B can be implemented by the following step S105B1.

[0199] Step S105A1: Determine the foreground region where the obstacle is located and the first edge region adjacent to the foreground region in the obstacle image.

[0200] For methods of determining the foreground region where obstacles are located, please refer to the above. Figure 4 Step S105A in the illustrated embodiment will not be described again here.

[0201] The first edge region mentioned above is adjacent to the foreground region, which can be understood as the first edge region being adjacent to the boundary of the foreground region. Therefore, after determining the foreground region, the boundary of the foreground region can be further determined, and the first edge region adjacent to the boundary of the foreground region can be determined based on the position of the boundary of the foreground region.

[0202] When determining the first edge region adjacent to the boundary of the foreground region, pixels whose distance from the boundary of the foreground region is less than a preset pixel distance can be determined in other regions of the obstacle image besides the foreground region, and the region formed by the determined pixels is taken as the first edge region.

[0203] The preset pixel distance can be 3, 4, or any other number of pixels.

[0204] Step S105A2: Determine the second edge region at the same location as the first edge region in the road image, and determine the background region outside the same location as the foreground region and the first edge region in the road image.

[0205] Since the road image and the obstacle image are the same size, for any region in the obstacle image, based on the location information of that region in the obstacle image, a corresponding region in the road image with the same location can be determined. Thus, after determining the first edge region in the obstacle image, the location information of the first edge region can be obtained, and based on this location information, a region with the same location information in the road image can be determined as the second edge region. Similarly, based on the location information of the foreground region in the obstacle image, a region corresponding to the foreground region can be determined in the road image. After determining the region corresponding to the foreground region and the second edge region in the road image, the background region outside the same location as the foreground region and the first edge region can be determined in the road image.

[0206] After determining the foreground region, background region, first edge region, and second edge region, a sample image can be obtained through the following steps S105B1.

[0207] Step S105B1: Based on the distance weight coefficient, fuse the first edge region and the second edge region to obtain a sample image containing the foreground region, the background region and the fused region.

[0208] The distance weighting coefficient is determined based on the distance between each pixel in the first edge region and the edge pixels in the foreground region.

[0209] Specifically, for each pixel in the first edge region, a weighting coefficient can be determined based on the distance between the pixel and the edge pixels in the foreground region to fuse the pixel and the pixels in the second edge region at the same position as the pixel. Based on the weighting coefficients of each pixel, the first edge region and the second edge region are fused to obtain the fused region. Thus, a sample image with a foreground region, a background region, and a fused region can be obtained according to their positions.

[0210] For example, if pixels e1 and f1 exist in the first edge region, and pixel e1 corresponds to pixel e2 in the second edge region, and pixel f1 corresponds to pixel f2 in the second edge region, then based on the distance between pixel e1 and the edge pixels of the foreground region, the weight coefficient for merging pixels e1 and e2 is determined as k1, and based on the distance between pixel f1 and the edge pixels of the foreground region, the weight coefficient for merging pixels f1 and f2 is determined as k2. Assuming that after merging the first and second edge regions, pixel e3 corresponds to pixel e1 in the merged region, and pixel f3 corresponds to pixel f1 in the merged region, then when merging the first and second edge regions, the pixel value of pixel e3 in the merged region can be determined based on the pixel values ​​of pixels e1 and e2 and the weight coefficient k1, and the pixel value of pixel f3 in the merged region can be determined based on the pixel values ​​of pixels f1 and f2 and the weight coefficient k2.

[0211] The method for obtaining sample images with foreground, background and fused regions according to location can be found in step S105B above, and will not be repeated here.

[0212] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, after determining the first edge region and the second edge region, the first edge region and the second edge region are fused based on the distance weight coefficient. The distance weight coefficient is determined according to the distance between each pixel in the first edge region and the edge pixels in the foreground region. In this way, the pixel value of each pixel in the fused region can be adjusted according to the pixel distance. If the distance between the pixel in the first edge region and the edge pixels in the foreground region is closer, the pixel value of the pixel corresponding to that pixel in the fused region is closer to the pixel value of the edge pixels in the foreground region. If the distance between the pixel in the first edge region and the edge pixels in the foreground region is farther, the pixel value of the pixel corresponding to that pixel in the fused region is closer to the pixel value of the edge pixels in the background region. In this way, a pixel value gradient band can be formed between the background region and the foreground region, thereby improving the realism of the sample image.

[0213] In one embodiment of the present invention, see Figure 6a The document provides a flowchart of the sixth sample image generation method. In this embodiment, step S105A can be achieved through the following steps S105A3-S105A5.

[0214] Step S105A3: Generate an obstacle mask map including the first region.

[0215] The first region reflects the area where the obstacle is located in the obstacle image.

[0216] The obstacle mask image mentioned above is usually a binary image, where the pixels in the first region and the pixels in the other region are two types of pixels with different pixel values.

[0217] Specifically, after rendering the obstacle image from the virtual scene, the position of each pixel in the obstacle image often corresponds to its position in the virtual scene, and according to the aforementioned... Figure 1a As can be seen from the description of step S104 in the illustrated embodiment, the position of the obstacle model in the virtual scene is determined. Therefore, the position information of the region corresponding to the virtual position in the obstacle image can be determined based on the virtual position of the obstacle model in the virtual scene, thereby generating an obstacle mask map including the first region based on the determined position information.

[0218] In addition, when rendering virtual scenes, besides generating obstacle images, obstacle mask maps can also be generated based on the position of the obstacle models in the virtual scene.

[0219] Figure 6b In order to be consistent with the above Figure 1b The obstacle image shown corresponds to the obstacle mask image. Figure 6bIn the image, the first region is white, reflecting the area where the obstacle is located.

[0220] Step S105A4: Based on the obstacle mask map, determine the foreground region where the obstacle is located in the obstacle image.

[0221] Specifically, after generating the obstacle mask map, the position information of the first region in the obstacle mask map can be obtained, and the region with the same position information can be determined in the obstacle image, thereby determining that the region is the foreground region where the obstacle is located.

[0222] Step S105A5: Based on the obstacle mask map, determine the background area outside the same location as the foreground area in the road image.

[0223] Specifically, similar to step S105A4 above, in the road image, an area with the same location as the first area can be determined, thereby determining other areas besides the determined area, and then determining the other areas as background areas.

[0224] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, since the first region in the generated obstacle mask image can reflect the region where the obstacle is located in the obstacle image, the foreground region where the obstacle is located can be accurately determined in the obstacle image based on the obstacle mask image, and the background region outside the same position of the foreground region can be accurately determined in the road image based on the obstacle mask image. Thus, based on the more accurate foreground and background regions, the accuracy of the sample image obtained after fusing the road image and the obstacle image can be improved.

[0225] After fusing obstacle and road images, the pixel values ​​at the boundary between the foreground and background regions in the resulting sample image may differ significantly, resulting in abrupt edges and low realism in the sample image.

[0226] To address this problem, in one embodiment of the present invention, see [reference needed]. Figure 7a The document provides a flowchart of the seventh sample image generation method. In this embodiment, the method further includes the following steps S107-S108, and the above steps S105A4 can be implemented by the following steps S105A4a, S105A5 can be implemented by the following steps S105A5a, and S105B can be implemented by the following steps S105B2.

[0227] Step S107: Dilate the edges of the first region in the obstacle mask image.

[0228] Expanding the edge of the first region can be understood as expanding the edge of the first region outward in the direction from the first region to other regions.

[0229] Specifically, edge pixels located at the edge of the first region can be identified. Then, following the direction from the first region to other regions, pixels whose distance from the edge pixels is less than a preset distance can be identified as new edge pixels in the first region to achieve dilation. The region where these edge pixels are located is the dilated region generated by the dilation process.

[0230] For example, the preset distance can be the distance of 3, 4 or other numbers of pixels.

[0231] Step S108: Blur the expanded region generated by the expansion process.

[0232] Specifically, since the obstacle mask image is usually a binary image, the pixels in the first region and the pixels in the other region are two types of pixels with different pixel values. Blurring the dilated region can be understood as adjusting the pixel values ​​of the pixels in the dilated region to pixel values ​​between the two types of pixel values ​​mentioned above.

[0233] For example, if the pixel value of a pixel in the first region is 1 and the pixel value of a pixel in other regions is 0, the obstacle mask image can be regarded as a grayscale image. By blurring the above-mentioned dilated region, the grayscale value of the pixel in the dilated region can be adjusted to any grayscale value between 0 and 1, such as 0.3, 0.5, or 0.8. In this case, blurring the dilated region can make the processed dilated region transparent.

[0234] In addition, when adjusting the pixel values ​​of pixels in the expansion region, you can adjust the pixel values ​​of each pixel to the same pixel value, or you can adjust the pixel values ​​of different pixels according to their distance from the original edge pixels of the first region.

[0235] For details on how to adjust the pixel values ​​of pixels in the expanded region according to their distance from the original edge pixels of the first region, please refer to the following. Figure 8 Step S108A in the illustrated embodiment will not be described in detail here.

[0236] Figure 7b This is an obstacle mask image before blurring. Figure 7c This is a blurred obstacle mask image, compared to... Figure 7b and Figure 7c It can be seen that, Figure 7b In the image, the division between the white and black areas is clear. Figure 7cThere is a blurred area between the white and black areas.

[0237] After dilating the edges of the first region in the obstacle mask image, the obstacle mask image includes the first region, the dilated region, and the second region other than these two regions. Thus, when determining the region in the obstacle image, the above step S105A4 can be achieved based on the first region and the dilated region through the following steps S105A4a.

[0238] Step S105A4a: In the obstacle image, determine the third region that is in the same position as the first region and the fourth region that is in the same position as the dilated region.

[0239] Specifically, the position information of the first region in the obstacle mask image can be obtained, and then the region with the same position information in the obstacle image can be determined as the third region. Similarly, the fourth region can be determined in the same way.

[0240] For example, if the first region is located between row a and row b of the obstacle mask image, then the region between row a and row b in the obstacle image can be defined as the third region.

[0241] When determining a region in a road image, step S105A5 can be achieved based on the second region and the dilated region described above, through the following steps S105A5a.

[0242] Step S105A5a: In the road image, determine the fifth region that is in the same position as the second region and the sixth region that is in the same position as the dilation region.

[0243] This step is similar to step S105A4a above, and will not be repeated here.

[0244] After determining the third, fourth, fifth, and sixth regions, step S105B can be implemented through the following step S105B2.

[0245] Step S105B2: Using the weight coefficients determined by the pixel values ​​of each pixel in the dilated region, fuse the fourth region and the sixth region to obtain a sample image containing the third region, the fifth region, and the fused region.

[0246] Specifically, the above sample image can be seen as being stitched together from the third region, the fifth region, and the fusion region.

[0247] The aforementioned expansion region, fourth region, sixth region, and fusion region are regions with the same position in the obstacle mask image, obstacle image, road image, and fusion region, respectively. For each pixel in the fusion region, a pixel with the same position as that pixel can be found in the other three regions.

[0248] For simplicity, the pixels in the fused region are referred to as pixel m, the pixels in the dilated region that are at the same position as pixel m are referred to as pixel m1, the pixels in the fourth region that are at the same position as pixel m are referred to as pixel m2, and the pixels in the sixth region that are at the same position as pixel m are referred to as pixel m3.

[0249] When merging the fourth and sixth regions, for each pixel m, the weight coefficients of the pixel values ​​of pixels m2 and m3 can be determined based on the pixel value of pixel m1. Thus, the pixel value of pixel m is calculated by weighting the pixel values ​​of pixel m2 and pixel m3.

[0250] For example, if the pixel value of pixel m1 is 0.4, the weight coefficient of pixel m2 is determined to be 0.4, and the weight coefficient of pixel m3 is 0.6. If the pixel value of pixel m2 is k1 and the pixel value of pixel m3 is k2, then the pixel value of pixel m is 0.4K1+0.6K2.

[0251] After calculating the above-mentioned fusion region, the third region, the fifth region, and the fusion region can be stitched together according to their respective locations to obtain the stitched sample image.

[0252] As can be seen from the above, when generating a sample image using the scheme provided in the embodiments of the present invention, the sample image includes a third region, a fifth region, and a fusion region. The fusion region is obtained by fusing the fourth region and the sixth region with a weight coefficient determined by the pixel values ​​of each pixel in the dilated region. The pixel values ​​of the pixels in the fusion region can be regarded as being obtained by fusing some pixel values ​​of the pixels at the same position in the fourth region with some pixel values ​​of the pixels at the same position in the sixth region, thereby reducing the difference between the pixel values ​​of adjacent pixels in the sample image, reducing the abruptness between the foreground region and the background region in the sample image, and improving the realism of the sample image.

[0253] Furthermore, since an image can be represented by a matrix composed of the pixel values ​​of each pixel, the road image, obstacle image, and obstacle mask image can be regarded as three different matrices, and the above steps S105A4a, S105A5a, and S105B2 can be implemented by matrix calculation.

[0254] In one embodiment of the present invention, steps S105A4a, S105A5a, and S105B2 described above can be implemented by the following expressions to obtain a sample image:

[0255] S = A * B + A ′ *

[0256] Where S represents the sample image correspondence matrix, A represents the obstacle mask image correspondence matrix, B represents the obstacle image correspondence matrix, and A ′ Let A represent the matrix obtained by inverting matrix A, and C represent the matrix corresponding to the road image.

[0257] In one case, A ′ The value of each element in A is 1 minus the value of the element at the same position in A.

[0258] When the obstacle mask image is a grayscale image containing pixels with pixel values ​​of 0 and 1, step S105B2 can be considered as fusing the obstacle image and the road image based on the grayscale values ​​of each pixel in the obstacle mask image.

[0259] The following explains the specific implementation method of adjusting the pixel values ​​of pixels in the expanded region according to the distance between the pixels and the original edge pixels of the first region.

[0260] In one embodiment of the present invention, see Figure 8 The document provides a flowchart of the eighth sample image generation method. In this embodiment, the above step S108 can be achieved through the following step S108A.

[0261] Step S108A: Adjust the pixel values ​​of the pixels in the expansion region based on the distance between the pixels in the expansion region and the edge pixels in the first region, the pixel values ​​of the pixels in the first region, and the pixel values ​​of the pixels in the second region.

[0262] Specifically, within the expansion region, if the distance between a pixel and the edge pixels of the first region is smaller, the pixel value of that pixel can be adjusted to a pixel value that is less different from the pixel value of the pixels in the first region. If the distance between a pixel and the edge pixels of the first region is larger, the pixel value of that pixel can be adjusted to a pixel value that is similar to the pixel value of the pixels in the second region.

[0263] For example, if there is a row of pixels n1, n2, n3, n4, and n5 in an obstacle mask image, where pixel n1 belongs to the second region and has a pixel value of 0, pixel n5 belongs to the first region and has a pixel value of 1, and pixels n2, n3, and n4 belong to the blending region, then for pixel n2, since the distance between pixel n2 and pixel n5 is relatively large, the pixel value of pixel n2 can be adjusted to a pixel value that is small compared to 0, such as 0.1 or 0.2. For pixel n3, since the distance between pixel n3 and pixel n5 is relatively small compared to the corresponding distance of pixel n2, the pixel value of pixel n3 can be adjusted to 0.4, 0.5, or 0.6. For pixel n4, since the distance between pixel n4 and pixel n5 is relatively small, the pixel value of pixel n4 can be adjusted to a pixel value that is small compared to 1, such as 0.7 or 0.8.

[0264] As can be seen from the above, when generating sample images using the scheme provided by the present invention, the pixel values ​​of the pixels in the expansion region are adjusted according to the distance between the pixels in the expansion region and the edge pixels in the first region, the pixel values ​​of the pixels in the first region, and the pixel values ​​of the pixels in the second region. The adjusted expansion region can be regarded as a gradient band between the first region and the second region. Thus, in the obtained sample image containing the third region, the sixth region, and the fusion region, the fusion region can also be regarded as a gradient band between the third region and the sixth region, thereby eliminating the abruptness of the edges caused by fusion in the sample image. Therefore, the sample image generation scheme provided by the embodiments of the present invention can improve the realism of the generated sample image.

[0265] In generating the virtual scene described above, in addition to the method mentioned in step S103, it can also be done through the following methods: Figure 9 In the illustrated embodiment, steps S103A-S103B generate a virtual scene.

[0266] In one embodiment of the present invention, see Figure 9 The flowchart of the ninth sample image generation method is provided. In this embodiment, the above step S103 can be achieved through the following steps S103A-S103B.

[0267] Step S103A: Construct a temporary virtual scene based on camera parameters and lighting information.

[0268] Specifically, based on camera parameters and lighting information, temporary virtual scenes can be constructed using existing scene building techniques.

[0269] In addition, when constructing temporary virtual scenes, a temporary virtual scene can be constructed for each road image obtained, or a temporary virtual scene can be constructed for multiple road images in the same real scene. For example, the same camera can capture multiple road images in a short period of time. Since the acquisition time difference of the captured road images is small, the camera parameters and lighting information can be considered to be consistent. Therefore, it can be assumed that the real scene in which the multiple road images are located is the same. Thus, a temporary virtual scene can be constructed for the multiple road images based on the camera parameters and lighting information obtained from any one of the road images.

[0270] Step S103B: Overlay the obstacle model onto the target area in the temporary virtual scene to obtain the overlaid virtual scene.

[0271] The target area is determined based on camera parameters and the area information of the overlay area.

[0272] Specifically, the target area where the obstacle model is located in the virtual scene can be determined based on the camera parameters and the area information of the overlay area. Then, based on the model data of the obstacle model, the data of the target area in the virtual scene can be adjusted to the model data of the obstacle model, thereby obtaining the virtual scene described above.

[0273] Determining the target area and adjusting the target area data to the obstacle model data can be considered as overlaying the obstacle model onto a temporary virtual scene. After overlaying the model onto the scene, the obstacle models in the scene can be marked, so that the obstacle mask map can be obtained in subsequent processes.

[0274] The determination of the target area based on camera parameters and the area information of the superimposed area can be found in the description of step S103 above, and will not be repeated here.

[0275] As can be seen from the above, when generating sample images using the solution provided in the embodiments of the present invention, a blank temporary virtual scene can be constructed based on the camera parameters and lighting information. After the temporary virtual scene is constructed, obstacle models can be superimposed on the target area in the temporary virtual scene to accurately obtain a virtual scene including obstacle models. In this way, rendering the virtual scene can accurately obtain obstacle images, thereby fusing obstacle images and road images to achieve sample image generation.

[0276] Since the color tone of the road image may be inconsistent with the color tone of the rendered obstacle image, fusing the two images with inconsistent color tones may result in a color tone difference between the foreground and background in the obtained sample image, thereby reducing the realism of the sample image.

[0277] In view of the above, in one embodiment of the present invention, see... Figure 10 The document provides a flowchart of the tenth sample image generation method. In this embodiment, before fusing the road image and obstacle image, the method further includes the following step S109.

[0278] Step S109: Based on the pixel values ​​of each pixel in the foreground region where the obstacle is located in the obstacle image, and the pixel values ​​of each pixel in the corresponding region of the foreground region in the road image, perform tone mapping processing on each pixel in the foreground region.

[0279] Specifically, based on the pixel values ​​of each pixel in the foreground region where the obstacle is located in the obstacle image, the hue information of the foreground region can be calculated. Based on the pixel values ​​of each pixel in the corresponding region of the foreground region in the road image, the hue information of the corresponding region can be calculated. Based on these two types of hue information, the hue mapping relationship between the foreground region and its corresponding region can be obtained. Thus, based on this hue mapping relationship, hue mapping processing can be performed on each pixel in the foreground region.

[0280] For details on the implementation of tone mapping for each pixel in the foreground region, please refer to the following sections. Figure 11 Steps S109A-S109C in the illustrated embodiment will not be described in detail here.

[0281] Alternatively, existing tone mapping techniques can be used to perform tone mapping on each pixel in the foreground area.

[0282] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, after performing tone mapping processing on each pixel in the foreground region based on the pixel values ​​of each pixel in the foreground region and the pixel values ​​of each pixel in the corresponding region of the foreground region in the road image, the tone of the foreground region can be kept consistent with the tone of the corresponding region of the foreground region in the road image. In this way, after fusing the road image and the obstacle image to obtain the sample image, it can be ensured that the tone of each region in the sample image is the same, thereby improving the realism of the sample image.

[0283] The following section explains the specific implementation method of tone mapping for each pixel in the foreground region.

[0284] In one embodiment of the present invention, see Figure 11 The document provides a flowchart of the eleventh sample image generation method. In this embodiment, the above step S109 can be achieved through the following steps S109A-S109C.

[0285] Step S109A: Calculate the first pixel mean of each pixel in the foreground region where the obstacle is located in the obstacle image.

[0286] Specifically, the foreground region where the obstacle is located can be determined in the obstacle image above, and then the average pixel value of each pixel in the foreground region can be calculated to obtain the first pixel average value.

[0287] For implementation methods of determining the foreground region where the obstacle is located in the obstacle image, please refer to the foregoing. Figure 4 The description of step S105A in the illustrated embodiment will not be repeated here.

[0288] Step S109B: Calculate the second pixel mean of each pixel in the region corresponding to the foreground region in the road image.

[0289] The area in a road image that corresponds to the foreground area is the area in the road image that is in the same position as the foreground area.

[0290] This step is similar to step S109A above, and will not be described in detail here.

[0291] Step S109C: Adjust the pixel values ​​of each pixel in the foreground region based on the ratio of the average value of the first pixel to the average value of the second pixel.

[0292] Specifically, after obtaining the first pixel mean and the second pixel mean, the ratio of the second pixel mean and the second pixel mean can be calculated, and the pixel value of each pixel in the foreground region can be adjusted based on the calculated ratio.

[0293] For example, if the average value of the first pixel is 64 and the average value of the second pixel is 32, then the ratio of the average value of the first pixel to the average value of the second pixel is 2, and the pixel value of each pixel in the foreground area can be adjusted to half of the original pixel value.

[0294] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, the ratio of the first pixel mean and the second pixel mean can be regarded as the hue ratio of the foreground region and the corresponding region in the road image. Based on the above ratio, adjusting the pixel values ​​of each pixel in the foreground region can adjust the hue of the foreground region to be similar to the hue of the corresponding region in the road image, thereby making the hues of the foreground and background in the fused sample image nearly consistent. Therefore, the sample image generation scheme provided in the embodiments of the present invention can improve the realism of the sample image.

[0295] The rendered obstacle images are often highly sharp, but the sharpness of different road images may vary. This may result in inconsistent sharpness between the foreground and background in the sample image, leading to a decrease in the realism of the sample image.

[0296] In view of the above, in one embodiment of the present invention, see... Figure 12The flowchart of the twelfth sample image generation method is provided. In this embodiment, the above step S105 can be achieved through the following steps S105C-S105D.

[0297] Step S105C: Using the road image as the background image and the obstacle image as the foreground image, perform image fusion on the road image and the obstacle image to obtain the first intermediate image.

[0298] This step is similar to step S105 above, and will not be repeated here.

[0299] Step S105D: Perform overall blurring on the first intermediate image to obtain the processed sample image.

[0300] The overall blurring of the first intermediate image can be achieved using existing image processing techniques, such as image filtering techniques.

[0301] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, after fusing to obtain the first intermediate image, performing overall blurring processing on the first intermediate image can make the clarity of the foreground and background in the processed sample image more consistent, thereby improving the realism of the sample image.

[0302] After blurring the first intermediate image, the brightness, contrast, and other tonal information of the resulting sample image may change, leading to a decrease in the realism of the sample image.

[0303] In view of the above, in one embodiment of the present invention, see... Figure 13 The flowchart of the thirteenth sample image generation method is provided. In this embodiment, the above step S105D can be achieved through the following steps S105D1-S105D2.

[0304] Step S105D1: Perform overall blurring on the first intermediate image to obtain the processed second intermediate image.

[0305] This step is similar to step S105D above, and will not be repeated here.

[0306] Step S105D2: Based on the image parameters of the road image, adjust the image parameters of the second intermediate image to obtain the adjusted sample image.

[0307] The aforementioned image parameters may include parameters such as chroma, brightness, contrast, and sharpness. For example, adjusting the brightness and / or contrast of a road image.

[0308] Specifically, the image parameters of the road image can be obtained, and then the image parameters of the second intermediate image can be adjusted to the obtained image parameters to obtain the adjusted image, which is used as the sample image.

[0309] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, after fusing the road image and the obstacle image to obtain the first intermediate image, the first intermediate image is subjected to overall blurring and image parameter adjustment in sequence. This makes the blur level and color tone of the processed sample image more consistent with the road image, thereby improving the realism of the sample image.

[0310] In one embodiment of the present invention, see Figure 14a A flowchart illustrating the fourteenth sample image generation method is provided. Users can generate sample images using the sample image generation system, which includes a superimposed region calculation module, a light camera parameter estimation module, a rendering module, and a fusion module. Figure 14a In this process, users can input road images into the overlayable area calculation module and the light camera parameter estimation module. The overlayable area calculation module is used to determine the overlayable areas in the road image and the overlay area for stacking obstacles within the overlayable area, and outputs the location information of the overlay area. The light camera parameter estimation module is used to calculate the camera's intrinsic and extrinsic parameters and illumination information. The rendering module obtains the above location information, camera intrinsic and extrinsic parameters, illumination information, and the 3D model of the obstacle selected by the user, and generates a virtual scene. The virtual scene is rendered to obtain the rendered image and mask image of the obstacle. The fusion module obtains the above rendered image, mask image, and road image, and fuses the rendered image and road image according to the mask image, outputting the result image and obstacle location information and type information, that is, outputting a sample image with obstacle location and type as annotation information.

[0311] Figure 14b The above diagram illustrates the workflow of the fusion module. After obtaining the rendered image, mask image, and road image, the fusion module performs edge dilation and blurring processing on the mask image and performs tone mapping on the rendered image. The tone mapping implementation method can be found in steps S109A-S109C above. After the mask image and rendered image are processed, the road image and rendered image are fused according to the mask image to obtain a fused image. Then, the fused image is sequentially subjected to overall blurring and brightness and contrast adjustment to finally obtain the sample image.

[0312] Corresponding to the above-described sample image generation method, this embodiment of the invention also provides a sample image generation apparatus.

[0313] In one embodiment of the present invention, see Figure 15 A schematic diagram of the structure of a first sample image generation device is provided, the device comprising:

[0314] The image acquisition module 1501 is used to acquire a road image of the actual scene and determine the stacking area in the road image for stacking obstacles;

[0315] The information acquisition module 1502 is used to obtain the illumination information of the actual scene and the parameters of the camera that acquired the road image based on the road image;

[0316] The scene generation module 1503 is used to generate a virtual scene including an obstacle model based on the parameters of the camera, lighting information and area information of the overlay area. The position of the obstacle model in the virtual scene is determined according to the parameters of the camera and the area information.

[0317] The scene rendering module 1504 is used to render the virtual scene to obtain obstacle images;

[0318] The image fusion module 1505 is used to perform image fusion on the road image as the background image and the obstacle image as the foreground image to obtain a sample image.

[0319] As can be seen from the above, when generating sample images using the scheme provided in this embodiment of the invention, a virtual scene including obstacle models is generated based on the obtained camera parameters, lighting information, and area information of the overlay area. Since the camera parameters and lighting information are obtained from road images, the virtual scene can be regarded as a simulated actual scene. Rendering the virtual scene in this way can obtain obstacle images with high realism. Image fusion of road images and obstacle images can not only obtain sample images, but these sample images can also be regarded as images obtained from roads with obstacles placed in the actual scene. Therefore, the obtained sample images have high realism. Training the obstacle detection model based on the sample images with high realism can improve the training efficiency and recall rate of the obstacle detection model.

[0320] Furthermore, the obstacle model mentioned above can be any obstacle model in the obstacle model library or an obstacle model manually constructed by the user. This allows for the generation of various sample images containing different obstacles. These generated sample images can then be used to train the obstacle detection model, improving its ability to detect various obstacles. Alternatively, based on the training requirements of the obstacle detection model, a large number of sample images containing specific types of obstacles can be generated, allowing for targeted training of the model and improving its ability to detect specific types of obstacles. Moreover, the road images mentioned above are relatively easy to collect and can be collected in large quantities. Sample images can be obtained by fusing road and obstacle images, eliminating the need for manual sample image collection. Therefore, applying the sample image generation scheme provided in this embodiment of the invention can reduce the difficulty of obtaining sample images.

[0321] In one embodiment of the present invention, see Figure 16 A schematic diagram of the structure of a second sample image generation device is provided. In this embodiment, the road image and the obstacle image are the same size.

[0322] The image fusion module 1505 includes:

[0323] The region determination submodule 1505A is used to determine the foreground region where the obstacle is located in the obstacle image, and to determine the background region outside the same location as the foreground region in the road image;

[0324] The position fusion submodule 1505B is used to obtain a sample image in which the foreground region and the background region are fused by position.

[0325] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, if the obstacle image and the road image are the same size, the foreground region where the obstacle is located is determined in the obstacle image, and the background region outside the same position of the foreground region is determined in the road image. In this way, the road image and the obstacle image can be accurately fused based on the positions of the foreground region and the background region to obtain a sample image fused with the foreground region and the background region.

[0326] In one embodiment of the present invention, the region determination submodule 1505A is specifically used for:

[0327] In the obstacle image, a foreground region where the obstacle is located and a first edge region adjacent to the foreground region are determined; in the road image, a second edge region at the same position as the first edge region is determined; and in the road image, a background region other than the foreground region and the first edge region at the same position is determined.

[0328] The location fusion submodule 1505B is specifically used for:

[0329] Based on the distance weighting coefficient, the first edge region and the second edge region are fused to obtain a sample image containing the foreground region, the background region, and the fused region. The distance weighting coefficient is determined based on the distance between each pixel in the first edge region and the edge pixels in the foreground region.

[0330] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, after determining the first edge region and the second edge region, the first edge region and the second edge region are fused based on the distance weight coefficient. The distance weight coefficient is determined according to the distance between each pixel in the first edge region and the edge pixels in the foreground region. In this way, the pixel value of each pixel in the fused region can be adjusted according to the pixel distance. If the distance between the pixel in the first edge region and the edge pixels in the foreground region is closer, the pixel value of the pixel corresponding to that pixel in the fused region is closer to the pixel value of the edge pixels in the foreground region. If the distance between the pixel in the first edge region and the edge pixels in the foreground region is farther, the pixel value of the pixel corresponding to that pixel in the fused region is closer to the pixel value of the edge pixels in the background region. In this way, a pixel value gradient band can be formed between the background region and the foreground region, thereby improving the realism of the sample image.

[0331] In one embodiment of the present invention, see Figure 17 A schematic diagram of a third sample image generation device is provided. In this embodiment, the region determination submodule 1505A includes:

[0332] The mask image generation unit 1505A1 is used to generate an obstacle mask image including a first region, wherein the first region reflects the region where the obstacle is located in the obstacle image;

[0333] The foreground determination unit 1505A2 is used to determine the foreground region where the obstacle is located in the obstacle image based on the obstacle mask image;

[0334] Background determination unit 1505A3 is used to determine a background region outside the same position as the foreground region in the road image based on the obstacle mask map.

[0335] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, since the first region in the generated obstacle mask image can reflect the region where the obstacle is located in the obstacle image, the foreground region where the obstacle is located can be accurately determined in the obstacle image based on the obstacle mask image, and the background region outside the same position of the foreground region can be accurately determined in the road image based on the obstacle mask image. Thus, based on the more accurate foreground and background regions, the accuracy of the sample image obtained after fusing the road image and the obstacle image can be improved.

[0336] In one embodiment of the present invention, see Figure 18 A schematic diagram of a fourth sample image generation device is provided. In this embodiment, the device further includes:

[0337] The dilation processing module 1506 is used to dilate the edge of the first region in the obstacle mask image;

[0338] The fuzzing module 1507 is used to fuzz the expanded region generated by the expansion process;

[0339] The foreground determination unit 1505A2 is specifically used to determine, in the obstacle image, a third region at the same position as the first region and a fourth region at the same position as the dilated region;

[0340] The background determination unit 1505A3 is specifically used to determine, in the road image, a fifth region at the same position as the second region and a sixth region at the same position as the expansion region, wherein the second region is the region in the obstacle image other than the first region and the expansion region;

[0341] The location fusion submodule 1505B is specifically used to fuse the fourth region and the sixth region using a weight coefficient determined by the pixel value of each pixel in the dilated region, to obtain a sample image containing the third region, the fifth region, and the fused region.

[0342] As can be seen from the above, when generating a sample image using the scheme provided in the embodiments of the present invention, the sample image includes a third region, a fifth region, and a fusion region. The fusion region is obtained by fusing the fourth region and the sixth region using a weight coefficient determined by the pixel values ​​of each pixel in the dilated region. Therefore, the pixel values ​​of the pixels in the fusion region can be regarded as being obtained by fusing some pixel values ​​of the pixels at the same position in the fourth region with some pixel values ​​of the pixels at the same position in the sixth region, thereby reducing the difference between the pixel values ​​of adjacent pixels in the sample image, reducing the abruptness between the foreground region and the background region in the sample image, and improving the realism of the sample image.

[0343] In one embodiment of the present invention, the fuzzing processing module 1507 is specifically used for:

[0344] The pixel values ​​of the pixels in the expansion region are adjusted based on the distance between the pixels in the expansion region and the edge pixels in the first region, the pixel values ​​of the pixels in the first region, and the pixel values ​​of the pixels in the second region.

[0345] As can be seen from the above, when generating sample images using the scheme provided by the present invention, the pixel values ​​of the pixels in the expansion region are adjusted according to the distance between the pixels in the expansion region and the edge pixels in the first region, the pixel values ​​of the pixels in the first region, and the pixel values ​​of the pixels in the second region. The adjusted expansion region can be regarded as a gradient band between the first region and the second region. Thus, in the obtained sample image containing the third region, the sixth region, and the fusion region, the fusion region can also be regarded as a gradient band between the third region and the sixth region, thereby eliminating the abruptness of the edges caused by fusion in the sample image. Therefore, the sample image generation scheme provided by the embodiments of the present invention can improve the realism of the generated sample image.

[0346] In one embodiment of the present invention, the scene generation module 1503 is specifically used for:

[0347] Based on the camera parameters and the lighting information, a temporary virtual scene is constructed;

[0348] The obstacle model is superimposed on the target area in the temporary virtual scene to obtain the superimposed virtual scene, wherein the target area is determined according to the camera parameters and the area information of the superimposed area.

[0349] As can be seen from the above, when generating sample images using the solution provided in the embodiments of the present invention, a blank temporary virtual scene can be constructed based on the camera parameters and lighting information. After the temporary virtual scene is constructed, obstacle models can be superimposed on the target area in the temporary virtual scene to accurately obtain a virtual scene including obstacle models. In this way, rendering the virtual scene can accurately obtain obstacle images, thereby fusing obstacle images and road images to achieve sample image generation.

[0350] In one embodiment of the present invention, the image acquisition module 1501 is specifically used for:

[0351] Obtain road images of the actual scene;

[0352] Detect the road region and the object region containing a preset type of object in the road image;

[0353] From the regions within the road image that belong to the road region but not to the object region, determine the stacking region for stacking obstacles.

[0354] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, the road area and the object area where the preset type of object is located in the road image are detected. Since obstacles usually refer to items appearing on the road, the above-mentioned object area can be regarded as a non-overlapping area in the road image. In this way, the area in the road image that belongs to the road area but does not belong to the object area is used to determine the stacking area for stacking obstacles, which conforms to the logic of the actual scene, thereby improving the realism of the generated sample image.

[0355] In one embodiment of the present invention, see Figure 19 A schematic diagram of a fifth sample image generation device is provided. In this embodiment, the device further includes:

[0356] The tone mapping module 1508 is used to perform tone mapping processing on each pixel in the foreground region based on the pixel values ​​of each pixel in the foreground region where the obstacle is located in the obstacle image and the pixel values ​​of each pixel in the corresponding region of the foreground region in the road image before performing image fusion on the road image and the obstacle image.

[0357] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, after performing tone mapping processing on each pixel in the foreground region based on the pixel values ​​of each pixel in the foreground region and the pixel values ​​of each pixel in the corresponding region of the foreground region in the road image, the tone of the foreground region can be kept consistent with the tone of the corresponding region of the foreground region in the road image. In this way, after fusing the road image and the obstacle image to obtain the sample image, it can be ensured that the tone of each region in the sample image is the same, thereby improving the realism of the sample image.

[0358] In one embodiment of the present invention, the tone mapping module 1508 is specifically used to: calculate the first pixel mean of each pixel in the foreground region where the obstacle is located in the obstacle image; calculate the second pixel mean of each pixel in the road image corresponding to the foreground region; and adjust the pixel value of each pixel in the foreground region based on the ratio of the first pixel mean to the second pixel mean.

[0359] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, the ratio of the first pixel mean and the second pixel mean can be regarded as the hue ratio of the foreground region and the corresponding region in the road image. Based on the above ratio, adjusting the pixel values ​​of each pixel in the foreground region can adjust the hue of the foreground region to be similar to the hue of the corresponding region in the road image, thereby making the hues of the foreground and background in the fused sample image nearly consistent. Therefore, the sample image generation scheme provided in the embodiments of the present invention can improve the realism of the sample image.

[0360] In one embodiment of the present invention, see Figure 20 A schematic diagram of the sixth sample image generation device is provided. In this embodiment, the image fusion module 1505 includes:

[0361] The image fusion submodule 1505C is used to perform image fusion on the road image as the background image and the obstacle image as the foreground image to obtain a first intermediate image;

[0362] The overall blur submodule 1505D is used to perform overall blurring on the first intermediate image to obtain the processed sample image.

[0363] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, after fusing to obtain the first intermediate image, performing overall blurring processing on the first intermediate image can make the clarity of the foreground and background in the processed sample image more consistent, thereby improving the realism of the sample image.

[0364] In one embodiment of the present invention, the overall fuzzy submodule 1505D is specifically used for:

[0365] The first intermediate image is blurred as a whole to obtain the processed second intermediate image;

[0366] Based on the image parameters of the road image, the image parameters of the second intermediate image are adjusted to obtain the adjusted sample image.

[0367] As can be seen from the above, when generating sample images using the scheme provided in the embodiments of the present invention, after fusing the road image and the obstacle image to obtain the first intermediate image, the first intermediate image is subjected to overall blurring and image parameter adjustment in sequence. This makes the blur level and color tone of the processed sample image more consistent with the road image, thereby improving the realism of the sample image.

[0368] In one embodiment of the present invention, see Figure 21A schematic diagram of the structure of a seventh sample image generation device is provided. In this embodiment, the device further includes:

[0369] The sample annotation module 1509 is used to obtain at least one of the region information of the area where the obstacle is located and the item information of the obstacle in the sample image, as the annotation information of the sample image.

[0370] As can be seen from the above, when generating sample images using the solution provided in this embodiment of the invention, at least one of the region information of the area where the obstacle is located and the item information of the obstacle can be obtained as annotation information for the sample image. Using labeled sample images to detect obstacles can improve the training effect of the obstacle detection model. Furthermore, this solution eliminates the need for manual annotation of sample images, thereby reducing the annotation cost.

[0371] This invention also provides an autonomous driving device, including a processor, wherein the processor runs an obstacle detection model, which is trained based on sample images generated by the scheme mentioned in the foregoing method embodiments.

[0372] In this approach, because the sample images generated by the method mentioned in the preceding embodiments have high realism, training the obstacle detection model based on these highly realistic sample images can improve the model's training efficiency and accuracy. Furthermore, applying the method mentioned in the preceding embodiments can generate a large number of sample images, allowing for thorough training of the obstacle detection model and thus improving its reliability.

[0373] This invention also provides an electronic device, such as... Figure 22 As shown, it includes a processor 2201, a communication interface 2202, a memory 2203, and a communication bus 2204. The processor 2201, the communication interface 2202, and the memory 2203 communicate with each other through the communication bus 2204.

[0374] Memory 2203 is used to store computer programs;

[0375] When processor 2201 executes the program stored in memory 2203, it performs the following steps:

[0376] Obtain a road image of the actual scene and determine the stacking area in the road image for stacking obstacles;

[0377] Based on the road image, obtain the lighting information of the actual scene and the parameters of the camera that acquired the road image;

[0378] Based on the camera parameters, lighting information, and area information of the overlay area, a virtual scene including obstacle models is generated. The position of the obstacle models in the virtual scene is determined according to the camera parameters and the area information.

[0379] The virtual scene is rendered to obtain obstacle images;

[0380] Using the road image as the background image and the obstacle image as the foreground image, the road image and the obstacle image are fused to obtain a sample image.

[0381] Other schemes for generating sample images by the processor 2201 executing the program stored in the memory 2203 are the same as those mentioned in the aforementioned method embodiments, and will not be described again here.

[0382] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0383] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0384] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0385] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), graphics processing units (GPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0386] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described sample image generation methods.

[0387] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the sample image generation methods described above.

[0388] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0389] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0390] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0391] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for generating sample images, characterized in that, The method includes: Obtain a road image of the actual scene and determine the stacking area in the road image for stacking obstacles; Based on the road image, obtain the lighting information of the actual scene and the parameters of the camera that acquired the road image; Based on the camera parameters, lighting information, and area information of the overlay area, a virtual scene including obstacle models is generated. The position of the obstacle models in the virtual scene is determined according to the camera parameters and the area information. The virtual scene is rendered to obtain obstacle images; Using the road image as the background image and the obstacle image as the foreground image, the road image and the obstacle image are fused to obtain a sample image; Wherein, the road image and the obstacle image are of the same size; the step of fusing the road image as the background image and the obstacle image as the foreground image to obtain a sample image includes: In the obstacle image, a foreground region containing the obstacle and a first edge region adjacent to the foreground region are determined; in the road image, a second edge region at the same position as the first edge region is determined, and a background region outside the same position of the foreground region and the first edge region is determined in the road image; based on a distance weighting coefficient, the first edge region and the second edge region are fused to obtain a sample image containing the foreground region, the background region, and the fused region, wherein the distance weighting coefficient is determined based on the distance between each pixel in the first edge region and the edge pixels in the foreground region; or, Generate an obstacle mask image including a first region, wherein the first region reflects the area where the obstacle is located in the obstacle image; dilate the edges of the first region in the obstacle mask image; blur the dilated region resulting from the dilation process; determine a third region in the obstacle image that is at the same position as the first region and a fourth region that is at the same position as the dilated region; determine a fifth region in the road image that is at the same position as the second region and a sixth region that is at the same position as the dilated region, wherein the second region is the region in the obstacle image other than the first region and the dilated region; fuse the fourth region and the sixth region using a weight coefficient determined by the pixel values ​​of each pixel in the dilated region to obtain a sample image including the third region, the fifth region, and the fused region.

2. The method according to claim 1, characterized in that, The blurring of the expanded region resulting from the expansion process includes: The pixel values ​​of the pixels in the expansion region are adjusted based on the distance between the pixels in the expansion region and the edge pixels in the first region, the pixel values ​​of the pixels in the first region, and the pixel values ​​of the pixels in the second region.

3. The method according to any one of claims 1-2, characterized in that, The process of generating a virtual scene including an obstacle model based on the camera parameters, lighting information, and region information of the overlay area includes: Based on the camera parameters and the lighting information, a temporary virtual scene is constructed; The obstacle model is superimposed on the target area in the temporary virtual scene to obtain the superimposed virtual scene, wherein the target area is determined according to the camera parameters and the area information of the superimposed area.

4. The method according to any one of claims 1-2, characterized in that, Determining the stacking area for overlaying obstacles in the road image includes: Detect the road region and the object region containing a preset type of object in the road image; From the regions within the road image that belong to the road region but not to the object region, determine the stacking region for stacking obstacles.

5. The method according to any one of claims 1-2, characterized in that, Before performing image fusion on the road image and the obstacle image, the method further includes: Based on the pixel values ​​of each pixel in the foreground region where the obstacle is located in the obstacle image, and the pixel values ​​of each pixel in the corresponding region of the foreground region in the road image, tone mapping is performed on each pixel in the foreground region.

6. The method according to claim 5, characterized in that, The step of performing tone mapping processing on each pixel in the foreground region based on the pixel values ​​of each pixel in the foreground region where the obstacle is located in the obstacle image and the corresponding region in the road image includes: Calculate the first pixel mean of each pixel in the foreground region where the obstacle is located in the obstacle image; Calculate the second pixel mean of each pixel in the region corresponding to the foreground region in the road image; The pixel values ​​of each pixel in the foreground region are adjusted based on the ratio of the first pixel mean to the second pixel mean.

7. The method according to any one of claims 1-2, characterized in that, The step of fusing the road image as the background image and the obstacle image as the foreground image to obtain a sample image includes: Using the road image as the background image and the obstacle image as the foreground image, the road image and the obstacle image are fused to obtain a first intermediate image; The first intermediate image is blurred as a whole to obtain the processed sample image.

8. The method according to claim 7, characterized in that, The step of performing overall blurring processing on the first intermediate image to obtain the processed sample image includes: The first intermediate image is blurred as a whole to obtain the processed second intermediate image; Based on the image parameters of the road image, the image parameters of the second intermediate image are adjusted to obtain the adjusted sample image.

9. The method according to any one of claims 1-2, characterized in that, The method further includes: Obtain at least one of the following information as annotation information for the sample image: The sample image contains information about the region where the obstacle is located and information about the object within the obstacle.

10. A sample image generation apparatus, characterized in that, The device includes: The image acquisition module is used to acquire road images of the actual scene and determine the stacking area in the road image for stacking obstacles; The information acquisition module is used to obtain the illumination information of the actual scene and the parameters of the camera that acquired the road image based on the road image; The scene generation module is used to generate a virtual scene including obstacle models based on the parameters of the camera, lighting information and area information of the overlay area. The position of the obstacle models in the virtual scene is determined according to the parameters of the camera and the area information. The scene rendering module is used to render the virtual scene to obtain obstacle images; The image fusion module is used to fuse the road image as the background image and the obstacle image as the foreground image to obtain a sample image; Wherein, the road image and the obstacle image are the same size; the image fusion module is specifically used for: In the obstacle image, a foreground region containing the obstacle and a first edge region adjacent to the foreground region are determined; in the road image, a second edge region at the same position as the first edge region is determined, and a background region outside the same position of the foreground region and the first edge region is determined in the road image; based on a distance weighting coefficient, the first edge region and the second edge region are fused to obtain a sample image containing the foreground region, the background region, and the fused region, wherein the distance weighting coefficient is determined based on the distance between each pixel in the first edge region and the edge pixels in the foreground region; or, Generate an obstacle mask image including a first region, wherein the first region reflects the area where the obstacle is located in the obstacle image; dilate the edges of the first region in the obstacle mask image; blur the dilated region resulting from the dilation process; determine a third region in the obstacle image that is at the same position as the first region and a fourth region that is at the same position as the dilated region; determine a fifth region in the road image that is at the same position as the second region and a sixth region that is at the same position as the dilated region, wherein the second region is the region in the obstacle image other than the first region and the dilated region; fuse the fourth region and the sixth region using a weight coefficient determined by the pixel values ​​of each pixel in the dilated region to obtain a sample image including the third region, the fifth region, and the fused region.

11. An automatic driving device, characterized in that, The system includes a processor in which an obstacle detection model is run, the obstacle detection model being trained based on sample images generated by the method steps of any one of claims 1-9.

Images