Image generation method and device

Abstract

An embodiment of the present invention provides an image generation method and device, the method comprising: acquiring a first background image, the first background image comprising at least image information, a drivable road surface area, and a first position of a lane line in the drivable road surface area; acquiring a first foreground image, the first foreground image comprising at least road surface information in a target lane, the road surface information comprising road signs and / or tire marks on the road surface; and fusing the first foreground image with the target lane of the first background image to generate a target image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image generating method and device. Background Art

[0002] At present, driverless technology has become an important trend in the automotive industry. The development of driverless cars can not only improve traffic efficiency and reduce traffic accidents, but also provide more convenience for people's travel. However, the realization of driverless driving requires not only a highly intelligent control system, but also a large amount of high-quality road data to train the visual perception model to ensure that the vehicle can travel safely and efficiently. Among them, lane line detection plays a vital role in autonomous driving perception technology. However, in the real road environment, there are usually various potential interference factors around the lane line, such as text, arrows, tire marks, no-parking lines, etc. on the road surface. These interference factors reduce the accuracy and robustness of lane line detection, thereby threatening the reliability of the autonomous driving system.

[0003] Most of the current road surface data is obtained from high-precision map road networks, and lane lines in various scenarios need to be accurately labeled to train lane line detection models. The above method of obtaining road surface data is not only limited by map data, resulting in low generation efficiency, but also consumes a lot of manpower and time. Summary of the invention

[0004] The embodiments of the present invention provide an image generation method and device to solve the problem that the existing road surface data acquisition method not only consumes a lot of manpower and time but also has low generation efficiency.

[0005] In one aspect, a method for generating an image is provided, the method comprising:

[0006] Acquire a first background image, wherein the first background image includes at least: image information, a drivable road surface area, and a first position of a lane line in the drivable road surface area;

[0007] Acquire a first foreground image, wherein the first foreground image at least includes: road surface information in the target lane, wherein the road surface information includes: road signs and / or tire marks on the road surface;

[0008] The first foreground image is fused with the target lane of the first background image to generate a first target image.

[0009] In another aspect, an image generating device is provided, the device comprising:

[0010] A first acquisition module, configured to acquire a first background image, wherein the first background image includes at least: image information, a drivable road surface area, and a first position of a lane line in the drivable road surface area;

[0011] A second acquisition module is used to acquire a first foreground image, wherein the first foreground image at least includes: road surface information in the target lane, and the road surface information includes: road signs and / or tire marks on the road surface;

[0012] The first generating module is used to fuse the first foreground image with the target lane of the first background image to generate a first target image.

[0013] On the other hand, an electronic device is provided, including a memory, a transceiver, and a processor:

[0014] The memory is used to store computer programs; the transceiver is used to send and receive data under the control of the processor; the processor is used to read the computer program in the memory and execute the image generation method as described above.

[0015] On the other hand, a processor-readable storage medium is provided, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable a processor to execute the above-mentioned image generation method.

[0016] One of the above technical solutions has the following advantages or beneficial effects:

[0017] By acquiring a first background image and a first foreground image, the first background image at least includes: image information, a drivable road surface area and the first position of the lane line in the drivable road surface area, and the first foreground image at least includes: road surface information in the target lane, and the road surface information includes: road signs and / or tire marks on the road surface; the first foreground image is merged with the target lane of the first background image to generate a first target image. The above scheme can merge the foreground image of various potential interference factors with the background image to generate the first target image of various potential interference factors, better simulate the visual interference in different environments, so as to train the lane line detection model to better deal with various interference factors and improve the robustness of lane line detection; and the whole process does not require manual annotation of road surface information, which reduces the cost of data acquisition and annotation and improves the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 One of the step flow charts of the image generation method provided by an embodiment of the present invention;

[0020] Figure 2 A second flowchart of the image generation method provided by an embodiment of the present invention;

[0021] Figure 3 A flow chart of a method for obtaining a background image library provided by an embodiment of the present invention;

[0022] Figure 4 A flow chart of a method for acquiring a foreground image library provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of a sliding window process provided by an embodiment of the present invention;

[0024] Figure 6 One of the detailed step flow charts of the image generation method provided by an embodiment of the present invention;

[0025] Figure 7 One of the schematic diagrams of the target image provided by the embodiment of the present invention;

[0026] Figure 8 A second detailed flowchart of the image generation method provided by an embodiment of the present invention;

[0027] Fig. 9 A second schematic diagram of a target image provided by an embodiment of the present invention;

[0028] Fig.10 A structural block diagram of an image generating device provided by an embodiment of the present invention;

[0029] Fig.11 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In the embodiments of the present invention, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0031] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] Specifically, the embodiment of the present invention provides an image generation method, such as Figure 1 and Figure 2 As shown, the following steps may be specifically included:

[0034] Step 101: Acquire a first background image, wherein the first background image includes at least: image information, a drivable road surface area, and a first position of a lane line in the drivable road surface area.

[0035] Specifically, the first background image includes not only the image information of the original image itself, but also the drivable road area extracted from the original image, and the first position of the lane line in the drivable road area in the original image. In addition, the first background image may also include other required information, which is not specifically limited here. Thus, at least the drivable road area in the original image of the first background image can be known, and the position of the lane line in the drivable road area can also be known.

[0036] Step 102: Acquire a first foreground image, wherein the first foreground image at least includes road surface information in a target lane, wherein the road surface information includes road signs and / or tire marks on the road surface.

[0037] Specifically, the first foreground image refers to a foreground image of a target lane, and the first foreground image includes but is not limited to road surface information in the target lane, wherein the road surface information includes but is not limited to at least one of road signs and tire marks on the road surface in the target lane. The road signs include but are not limited to: arrows, texts, no-parking lines, etc., and the tire marks refer to tire marks left on the road surface by the vehicle during driving.

[0038] It should be noted that the steps of obtaining the first background image in step 101 and obtaining the first foreground image in step 102 are not limited in order, and step 101 may be performed first, or step 102 may be performed first, or they may be performed simultaneously.

[0039] Step 103: Fusing the first foreground image with the target lane of the first background image to generate a first target image.

[0040] Specifically, the first foreground image is fused with the target lane of the first background image, that is, the first foreground image is placed in the target lane of the first background image, thereby obtaining a first target image that is a combination of the first foreground image and the first background image.

[0041] Through the above scheme, the foreground image is used as an interference factor, and the foreground and background of different images are fused to generate the first target images of various potential interference factors, so as to better simulate the visual interference in different environments, so as to train the lane line detection model to better deal with various interference factors and improve the robustness of lane line detection; and the whole process does not require manual labeling of road surface information, which reduces the cost of data acquisition and labeling and improves the efficiency of model training.

[0042] As an optional specific embodiment, the method may further include:

[0043] Transforming the first foreground image to obtain a second foreground image;

[0044] The second foreground image is fused with the target lane of the first background image to generate a second target image.

[0045] Specifically, after obtaining the first foreground image, the first foreground image may be transformed to obtain a transformed second foreground image, and the second foreground image may be fused with the target lane of the first foreground image, that is, the second foreground image is placed in the target lane of the first background image, thereby obtaining a second target image that is a combination of the second foreground image and the first background image. By using different transformation processing methods on the first foreground image, a variety of different foreground images may be obtained, that is, by transforming a foreground image and fusing it with the first background image, target images of various potential interference factors may be generated, thereby increasing the richness and randomness of the data.

[0046] The transformation processing includes but is not limited to reduction processing, enlargement processing, rotation processing, etc.

[0047] As an optional specific embodiment, Figure 2 As shown, the step 101 of obtaining the first background image may specifically include:

[0048] Step A1: Get background image library;

[0049] Step A2: upon receiving a first input, obtaining the first background image from the background image library.

[0050] Specifically, the background image library includes a plurality of background images. If a first input is received, a first background image is obtained from the background image library, and the first background image is one of the background images in the background image library.

[0051] The first input may be a selection input of a first background image in a background image library.

[0052] As an optional specific embodiment, Figure 3As shown, the above step A1 obtains the background image library, which may specifically include the following steps A11 to A14:

[0053] Step A11: Acquire M first images, where M is an integer greater than or equal to 1.

[0054] Specifically, M first images of the road surface are obtained through a high-precision map road network or by photographing with a camera of a collection vehicle.

[0055] Step A12: performing region segmentation on the drivable road surface areas in the M first images using the first semantic segmentation model to obtain a drivable road surface area corresponding to each first image.

[0056] Specifically, for one of the M first images, the road surface in the first image is used as the background, and the first semantic segmentation model is used to perform region segmentation on the first image to obtain the drivable road surface area corresponding to the first image. In the above manner, the drivable road surface area corresponding to each of the M first images is obtained.

[0057] Step A13: Perform lane line recognition on the M first images using a lane line detection model to obtain a first position of the lane line in each first image.

[0058] Specifically, for one of the M first images, the lane line detection model is used to identify the first position of the lane line in the 2D visual image (ie, the first image). In the above manner, the first position of the lane line in each of the M first images is obtained.

[0059] Step A14: According to the first position of the lane line in each first image, the lane line in the first image is marked at a corresponding position of the drivable road surface area of ​​the first image to obtain a background image library.

[0060] Specifically, for one of the M first images, the lane line is marked at the corresponding position of the drivable road surface area of ​​the first image according to the first position of the lane line in the first image, thereby obtaining a background image corresponding to the first image, which includes but is not limited to the image information in the first image, the drivable road surface area in the first image, and the first position of the lane line in the drivable road surface area. The first image corresponding to each background image is the original image of the background image. In the above manner, the background image corresponding to each of the M first images is obtained, and the multiple background images form a background image library.

[0061] It should be noted that during the shooting process, the camera can divide the lane lines into the current lane (i.e., the lane where the vehicle is located), the left lane of the current lane, and the right lane of the current lane, etc. according to the relative position of the vehicle and the road surface, thereby more accurately locating the lane lines and the drivable road surface area; and, during the shooting process, the camera generates driving lanes in a manner of larger near and smaller far (i.e., close to the camera or far from the camera), so the first image shows a visual effect of narrower lane lines in the distance, which increases the sense of reality.

[0062] As an optional specific embodiment, Figure 2 As shown, the step 102 of acquiring the first foreground image may specifically include:

[0063] Step A3: Obtain a foreground image library;

[0064] Step A4: upon receiving the second input, determining the first foreground image from the foreground image library.

[0065] Specifically, the foreground image library includes a plurality of foreground images. If the second input is received, a first foreground image is obtained from the foreground image library, where the first foreground image is one of the foreground images in the foreground image library.

[0066] The second input may be a selection input of the first foreground image in the foreground image library.

[0067] As an optional specific embodiment, Figure 4 As shown, the above step A3 obtains the foreground image library, which may specifically include steps A31 to A35:

[0068] Step A31: Acquire N second images, where N is an integer greater than or equal to 1.

[0069] Specifically, N second images of the road surface are obtained through a high-precision map road network or by photographing with a camera of a collection vehicle.

[0070] Step A32: Extract road surface information of each of the N second images through a second semantic segmentation model, wherein the road surface information includes: first road surface information of the current lane, second road surface information of the left lane of the current lane, and third road surface information of the right lane of the current lane.

[0071] Specifically, for one of the N second images, the road surface information of the second image is extracted through the second semantic segmentation model, and the road surface information includes: the first road surface information of the current lane, the second road surface information of the left lane of the current lane, and the third road surface information of the right lane of the current lane, etc.

[0072] Step A33: combining the first road surface information corresponding to the N second images respectively to form a foreground image library of the current lane.

[0073] Specifically, through the above step A32, the first road surface information, the second road surface information, the third road surface information, etc. corresponding to each of the N second images are obtained, and the first road surface information corresponding to the N second images respectively forms the foreground image library of the current lane.

[0074] Step A34: Combining the second road surface information corresponding to the N second images to form a foreground image library of the left lane.

[0075] Specifically, through the above step A32, the first road surface information, the second road surface information, the third road surface information, etc. corresponding to each of the N second images are obtained, and the second road surface information corresponding to the N second images respectively forms a foreground image library of the left lane.

[0076] Step A35: combining the third road surface information corresponding to the N second images respectively to form a foreground image library of the right lane;

[0077] The foreground image library includes: the foreground image library of the current lane, the foreground image library of the left lane, and the foreground image library of the right lane.

[0078] Specifically, through the above-mentioned step A32, the first road surface information, the second road surface information, the third road surface information, etc. corresponding to each of the N second images are obtained, and the third road surface information corresponding to the N second images respectively forms the foreground image library of the right lane, and the foreground image library of the current lane, the foreground image library of the left lane and the foreground image library of the right lane constitute the foreground image library.

[0079] It should be noted that, in the process of acquiring the second image, the camera can distinguish the foreground in the current lane, the foreground in the left lane, and the foreground in the right lane according to the relative position of the vehicle on the road and the camera installation angle.

[0080] As an optional specific embodiment, the step 103 of fusing the first foreground image with the target lane of the first background image to generate a first target image may specifically include:

[0081] Acquire a first size of the first foreground image and a second size of a target lane of the first background image;

[0082] When the first size is less than or equal to the second size, fusing the first foreground image with the target lane of the first background image to generate the first target image;

[0083] When the first size is larger than the second size, the first foreground image is fused with the drivable road area of ​​the first background image according to the first position of the lane line in the drivable road area to generate the first target image, in which the lane line in the drivable road area is placed above the first foreground image.

[0084] Specifically, after obtaining the first foreground image and the first background image, since the first foreground image is the foreground image of the target lane, it is necessary to obtain the first size of the first foreground image and the second size of the target lane of the first background image, and compare the first size with the second size. If the first size is less than or equal to the second size, it means that the first foreground image can be placed in the target lane of the first background image. The first foreground image and the target lane of the first background image are fused to generate the first target image.

[0085] If the first size is larger than the second size, it means that the size of the first foreground image is larger and can cover multiple lanes including the target lane. The first foreground image can be fused with the drivable road area of ​​the first background image according to the first position of the lane line in the drivable road area to generate a first target image. In the first target image, the lane lines in the drivable road area need to be placed above the first foreground image to ensure that the first target image can retain complete lane line features.

[0086] As an optional specific embodiment, the step 103 of fusing the first foreground image with the target lane of the first background image to generate a first target image may specifically include:

[0087] Determining a target position in a target lane of the first background image;

[0088] The first foreground image is fused with the target position of the first background image to generate the first target image.

[0089] Specifically, when the first foreground image and the first background image are obtained and the first size is less than or equal to the second size, it is necessary to determine the target position in the target lane of the first background image, and fuse the first foreground image with the target position of the first background image, that is, place the first foreground image at the target position of the first background image, thereby obtaining a first target image that is a combination of the first foreground image and the first background image.

[0090] As an optional specific embodiment, the step of determining the target position in the target lane of the first background image may specifically include:

[0091] Performing a sliding window operation on the first foreground image in the first background image, and calculating the overlap value between the first foreground image and the target lane when the first foreground image is respectively at P positions in the first background image, where P is an integer greater than 1;

[0092] Compare the P overlap values ​​with the preset values ​​respectively;

[0093] Determine a position corresponding to an overlap value greater than the preset value as a position to be determined;

[0094] When the number of the to-be-determined positions is 1, determining the to-be-determined position as the target position;

[0095] When the number of the positions to be determined is greater than 1, determining one of the multiple positions to be determined as the target position; or,

[0096] When the number of the positions to be determined is greater than 1, one of the positions for which road surface information does not exist among the positions to be determined is determined as the target position.

[0097] Specifically, the idea of ​​Intersection Over Union (IOU) is adopted to slide the first foreground image from the upper left to the lower right in the first background image. During the sliding window process, the overlap between the first foreground image and the target lane is calculated when the first foreground image is in each of the P positions in the first background image. The overlap is represented by the overlap value to obtain P overlap values. The P overlap values ​​are compared with the preset values ​​respectively, and the position corresponding to the overlap value greater than the preset value among the P overlap values ​​is determined as the position to be determined. If the position to be determined is one, then the position to be determined is the target position. For example: the preset value is 0.8, which means that the current position makes the first foreground image intersect with the target lane more than 80% to be used as the position to be determined.

[0098] If there are multiple positions to be determined, select one of them as the target position; or, if there are multiple positions to be determined, determine whether there is other road surface information at each position to be determined, that is, determine whether there are other arrows, text, tire marks, etc. at each position to be determined, and select one of the positions to be determined that do not have other road surface information as the target position.

[0099] It should be noted that the target location may be selected randomly or according to other rules, which is not specifically limited here.

[0100] The above sliding window process is described below through a specific embodiment:

[0101] like Figure 5 As shown, assuming that the size of the first foreground image is 20*20, a 20*20 matrix with all 1s is used to represent it, the target lane is the left lane, and the preset value is 0.8. Assuming that the size of the first background image is 100*100, the pixel position where the first foreground image appears in the left lane of the first background image is assigned 1, and the rest is assigned 0. In the process of sliding the first foreground image from the upper left corner to the lower right corner (i.e., in the sliding window process), when the first foreground image slides to position A, the left lane of the first foreground image and the first background image overlap. If there are 100 1s in the overlapping part and 300 0s in the other part, the overlap is calculated as: 100 / 400=0.25, where the numerator 100 is the intersection area (i.e., the number of 1s), and the denominator is the area of ​​the first foreground image 20*20=400. Since 0.25 is less than 0.8, position A cannot be used as the target position.

[0102] When the first foreground image slides to position B, the first foreground image 20*20 is a matrix of all 1s, that is, the first foreground image completely overlaps with the first background image, that is, the calculated overlap is: 400 / 400=1. Since 1 is greater than 0.8, position A can be used as the target position.

[0103] In addition, if the first foreground image is the first foreground image to be fused with the target lane of the first background image, then after the fusion of the first foreground image and the target lane of the first background image is completed, if the second foreground image needs to be fused with the target lane of the first background image, the target position of the fusion of the first foreground image needs to be set to 0, so that the second foreground image does not overlap with the first foreground image. When fusing the third foreground image, the target position of the fusion of the first foreground image and the target position of the fusion of the second foreground image need to be set to 0, and so on.

[0104] As an optional specific embodiment, the step 103 of fusing the first foreground image with the target lane of the first background image to generate a first target image may further include:

[0105] The first foreground image is fused with the target lane of the first background image, and image attributes of the first foreground image are adjusted to be consistent with image attributes of the first background image to generate the first target image.

[0106] Specifically, in order to increase the realism of the first target image and ensure the natural fusion of the first foreground image and the first background image, it is necessary to adjust the image attributes of the first foreground image so that the image attributes of the first foreground image are consistent with the image attributes of the first background image, so that the generated first target image can be more realistic and natural. The image attributes include but are not limited to: brightness, color, boundary transition, etc.

[0107] The brightness distribution of the first foreground image and the first background image is adjusted to be consistent, which can be achieved by histogram equalization or contrast adjustment, thereby ensuring that the first foreground image and the first background image are more consistent in brightness.

[0108] If the first foreground image and the first background image do not match in color, a color correction technique, such as color balancing or color transfer, may be used to make the colors of the first foreground image and the first background image more consistent.

[0109] Since the first target image is formed by the fusion of the first foreground image and the first background image, the transparency value of the pixels at the fusion boundary is calculated by the Matting transition method, and the first foreground image and the first background image are synthesized by alpha blending to make the transition between the first foreground image and the first background image smooth. Such a transition method can make there be no obvious boundary between the first foreground image and the first background image, thereby making the first target image more natural.

[0110] Among them, Matting is a method of creating a natural transition between foreground and background by calculating the transparency value of pixels.

[0111] The above image generation process is described below through two embodiments:

[0112] Embodiment 1, as Figure 6 As shown, it is taken as an example that the first size is less than or equal to the second size.

[0113] Step 601: Obtain a background image C from a background image library, where the background image C includes image information, a drivable road surface area, and a first position of a lane line.

[0114] Step 602: Divide the road surface of the background image C into three lanes, namely, the left lane m1, the current lane m2, and the right lane m3, according to the drivable road surface area and the first position of the lane line.

[0115] Step 603: Obtain n1 foreground images of the left lane, n2 foreground images of the current lane, and n3 foreground images of the right lane from the foreground image library.

[0116] Step 604: Perform image transformation on n1 foreground images of the left lane, n2 foreground images of the current lane, and n3 foreground images of the right lane to obtain w1 foreground images of the left lane, w2 foreground images of the current lane, and w3 foreground images of the right lane, wherein w1 is greater than n1, w2 is greater than n2, and w3 is greater than n3.

[0117] Step 605: Through the sliding window operation and IOU technology, m1 is randomly fused with at least one of the foreground images of w1 left lanes, m2 is randomly fused with at least one of the foreground images of w2 left lanes, and m3 is randomly fused with at least one of the foreground images of w3 left lanes.

[0118] Step 606: Adjust the brightness and color of the fused image.

[0119] Step 607: Adjust the boundary of the foreground image by using the Matting technique to obtain the final target image.

[0120] For example: Figure 7 As shown in the figure, "straight", "travel" and "area" are the foreground images of the three left lanes, and "straight", "travel" and "area" are transformed by rotating, enlarging, and bolding, and then merged into the left lane of the background image. "Courtesy", "high" and "speed" are the foreground images of the three current lanes, and "Courtesy", "high" and "speed" are transformed by rotating, enlarging, and bolding, and then merged into the current lane of the background image. "Slow" and "travel" are the foreground images of the two right lanes, and "Slow" and "travel" are transformed by rotating, enlarging, and bolding, and then merged into the right lane of the background image. Thus, the final target image can be obtained, in which the three characters "straight", "travel" and "area" of different sizes, thicknesses and angles are displayed in the left lane of the target image, the three characters "Courtesy", "high" and "speed" of different sizes, thicknesses and angles are displayed in the current lane, and the two characters "Slow" and "travel" of different sizes, thicknesses and angles are displayed in the right lane.

[0121] Embodiment 2: Figure 8 As shown, the first size is larger than the second size as an example.

[0122] Step 701: Obtain a background image D from a background image library, where the background image D includes image information, a drivable road surface area, and a first position of a lane line.

[0123] Step 702: Obtain n4 foreground images from the foreground image library. The foreground images may all be foreground images of the left lane, or may all be foreground images of the current lane, or may all be foreground images of the right lane; or the foreground images may also be a mixture of at least two of the foreground images of the left lane, the foreground images of the current lane, and the foreground images of the right lane.

[0124] Step 703: Perform image transformation on the n4 foreground images to obtain w4 foreground images, where w4 is greater than n4.

[0125] Step 704: Through the sliding window operation and IOU technology, the background image D is randomly fused with at least one of the w4 foreground images, and the lane line of the background image D is placed above the foreground image.

[0126] Step 705: Adjust the brightness and color of the fused image.

[0127] Step 706: Adjust the boundary of the foreground image by using the Matting technique to obtain the final target image.

[0128] For example: Fig. 9 As shown, the no-parking line is the foreground image. If the target lane is the current lane, since the first size of the no-parking line is larger than the second size of the target lane, the lane line is ignored and the no-parking line is merged into the drivable road surface area of ​​the background image, that is, the no-parking line occupies at least two lanes to obtain the final target image. The no-parking line of the target image occupies three lanes, namely the left lane, the current lane and the right lane, and the covered lane line is displayed above the no-parking line to ensure that the target image can retain the complete lane line features.

[0129] To summarize, the above embodiments of the present invention can fuse the visual interference factors (such as text, arrows, tire marks, no-parking lines, etc.) in the foreground image with the background image through the foreground-background fusion technology to generate target images of various visual interference factors, so as to better simulate the visual interference in different environments, so as to train the lane line detection model to better deal with various interference factors and improve the robustness of lane line detection; and the whole process does not require manual labeling of road surface information, which reduces the cost of data acquisition and labeling and improves the efficiency of model training.

[0130] The above describes the image generating method provided by the embodiment of the present invention. The following describes the image generating device provided by the embodiment of the present invention in conjunction with the accompanying drawings.

[0131] like Fig.10 As shown, the embodiment of the present invention further provides an image generating device 800, the device comprising:

[0132] A first acquisition module 801 is used to acquire a first background image, wherein the first background image includes at least: image information, a drivable road surface area, and a first position of a lane line in the drivable road surface area;

[0133] The second acquisition module 802 is used to acquire a first foreground image, wherein the first foreground image at least includes: road surface information in the target lane, and the road surface information includes: road signs and / or tire marks on the road surface;

[0134] The first generating module 803 is used to fuse the first foreground image with the target lane of the first background image to generate a first target image.

[0135] Optionally, the method further comprises:

[0136] A processing module, used for transforming the first foreground image to obtain a second foreground image;

[0137] The second generating module is used to fuse the second foreground image with the target lane of the first background image to generate a second target image.

[0138] Optionally, the first acquisition module 801 is specifically configured to:

[0139] Get the background image library;

[0140] When a first input is received, the first background image is acquired from the background image library.

[0141] Optionally, when acquiring the background image library, the first acquisition module 801 is specifically used to:

[0142] Acquire M first images, where M is an integer greater than or equal to 1;

[0143] Performing region segmentation on the drivable road surface areas in the M first images by using a first semantic segmentation model to obtain a drivable road surface area corresponding to each first image;

[0144] Performing lane line recognition on the M first images using a lane line detection model to obtain a first position of a lane line in each first image;

[0145] According to the first position of the lane line in each first image, the lane line in the first image is marked at a corresponding position of the drivable road surface area of ​​the first image to obtain a background image library.

[0146] Optionally, the second acquisition module 802 is specifically configured to:

[0147] Get the foreground image library;

[0148] Upon receiving a second input, the first foreground image is determined from the foreground image library.

[0149] Optionally, when acquiring the foreground image library, the second acquisition module 802 is specifically used to:

[0150] Acquire N second images, where N is an integer greater than or equal to 1;

[0151] Extracting road surface information of each of the N second images through a second semantic segmentation model, the road surface information including: first road surface information of a current lane, second road surface information of a left lane of the current lane, and third road surface information of a right lane of the current lane;

[0152] Combining the first road surface information respectively corresponding to the N second images to form a foreground image library of the current lane;

[0153] Combining the second road surface information respectively corresponding to the N second images to form a foreground image library of the left lane;

[0154] Combining the third road surface information respectively corresponding to the N second images to form a foreground image library of the right lane;

[0155] The foreground image library includes: the foreground image library of the current lane, the foreground image library of the left lane, and the foreground image library of the right lane.

[0156] Optionally, the first generating module 803 is specifically configured to:

[0157] Acquire a first size of the first foreground image and a second size of a target lane of the first background image;

[0158] When the first size is less than or equal to the second size, fusing the first foreground image with the target lane of the first background image to generate the first target image;

[0159] When the first size is larger than the second size, the first foreground image is fused with the drivable road area of ​​the first background image according to the first position of the lane line in the drivable road area to generate the first target image, in which the lane line in the drivable road area is placed above the first foreground image.

[0160] Optionally, when the first generating module 803 fuses the first foreground image with the target lane of the first background image to generate the first target image, it is specifically used to:

[0161] Determining a target position in a target lane of the first background image;

[0162] The first foreground image is fused with the target position of the first background image to generate the first target image.

[0163] Optionally, when determining the target position in the target lane of the first background image, the first generating module 803 is specifically configured to:

[0164] Performing a sliding window operation on the first foreground image in the first background image, and calculating the overlap value between the first foreground image and the target lane when the first foreground image is respectively at P positions in the first background image, where P is an integer greater than 1;

[0165] Compare the P overlap values ​​with the preset values ​​respectively;

[0166] Determine a position corresponding to an overlap value greater than the preset value as a position to be determined;

[0167] When the number of the positions to be determined is 1, the position to be determined is determined to be the target position.

[0168] Optionally, when determining the target position in the target lane of the first background image, the first generating module 803 is further configured to:

[0169] When the number of the positions to be determined is greater than 1, determining one of the multiple positions to be determined as the target position; or,

[0170] When the number of the positions to be determined is greater than 1, one of the positions for which road surface information does not exist among the positions to be determined is determined as the target position.

[0171] Optionally, the first generating module 803 is specifically configured to:

[0172] The first foreground image is fused with the target lane of the first background image, and image attributes of the first foreground image are adjusted to be consistent with image attributes of the first background image to generate the first target image.

[0173] It should be noted here that the above-mentioned image generating device provided by the embodiment of the present invention can implement all the method steps implemented by the above-mentioned image generating method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0174] like Fig.11 As shown, an embodiment of the present invention further provides an electronic device, including a memory 920, a transceiver 910, and a processor 900:

[0175] Memory 920, used for storing computer programs;

[0176] A transceiver 910, configured to transmit and receive data under the control of the processor;

[0177] The processor 900 is used to read the computer program in the memory and execute the steps of the image generating method described in any of the above embodiments.

[0178] Among them, Fig.11 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 900 and memory represented by memory 920. The bus architecture may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 910 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, which may include a wireless channel, a wired channel, an optical cable, and other transmission media. The processor 900 is responsible for managing the bus architecture and general processing, and the memory 920 may store data used by the processor 900 when performing operations.

[0179] The processor 900 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD). The processor may also adopt a multi-core architecture.

[0180] The processor calls the computer program stored in the memory to execute any image generation method provided in the embodiments of the present application according to the obtained executable instructions. The processor and the memory can also be arranged physically separately.

[0181] It should be noted here that the above-mentioned electronic device provided by the embodiment of the present invention can implement all the method steps implemented by the above-mentioned image generation method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0182] An embodiment of the present invention further provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the above-mentioned image generating method.

[0183] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0184] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0185] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer executable instructions. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These processor executable instructions may also be stored in a processor readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0188] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire a first background image, wherein the first background image includes at least: image information, a drivable road surface area, and a first position of a lane line in the drivable road surface area; Acquire a first foreground image, wherein the first foreground image at least includes: road surface information in the target lane, wherein the road surface information includes: road signs and / or tire marks on the road surface; Determining a target position in a target lane of the first background image; Fusing the first foreground image with the target lane of the first background image to generate a first target image; The step of fusing the first foreground image with the target lane of the first background image to generate a first target image includes: Acquire a first size of the first foreground image and a second size of a target lane of the first background image; When the first size is less than or equal to the second size, fusing the first foreground image with the target lane of the first background image to generate the first target image; When the first size is larger than the second size, the first foreground image is fused with the drivable road area of ​​the first background image according to the first position of the lane line in the drivable road area to generate the first target image, in which the lane line in the drivable road area is placed above the first foreground image.

2. The method according to claim 1, characterized in that The method further comprises: Transforming the first foreground image to obtain a second foreground image; The second foreground image is fused with the target lane of the first background image to generate a second target image.

3. The method according to claim 1, characterized in that The step of obtaining the first background image includes: Get the background image library; When a first input is received, the first background image is acquired from the background image library.

4. The method according to claim 3, characterized in that The step of obtaining a background image library comprises: Acquire M first images, where M is an integer greater than or equal to 1; Performing region segmentation on the drivable road surface areas in the M first images by using a first semantic segmentation model to obtain a drivable road surface area corresponding to each first image; Performing lane line recognition on the M first images using a lane line detection model to obtain a first position of a lane line in each first image; According to the first position of the lane line in each first image, the lane line in the first image is marked at a corresponding position of the drivable road surface area of ​​the first image to obtain a background image library.

5. The method according to claim 1, characterized in that The step of acquiring the first foreground image comprises: Get the foreground image library; Upon receiving a second input, the first foreground image is determined from the foreground image library.

6. The method according to claim 5, characterized in that The step of acquiring a foreground image library comprises: Acquire N second images, where N is an integer greater than or equal to 1; Extracting road surface information of each of the N second images through a second semantic segmentation model, the road surface information including: first road surface information of a current lane, second road surface information of a left lane of the current lane, and third road surface information of a right lane of the current lane; Combining the first road surface information respectively corresponding to the N second images to form a foreground image library of the current lane; Combining the second road surface information respectively corresponding to the N second images to form a foreground image library of the left lane; Combining the third road surface information respectively corresponding to the N second images to form a foreground image library of the right lane; The foreground image library includes: the foreground image library of the current lane, the foreground image library of the left lane, and the foreground image library of the right lane.

7. The method according to claim 1, characterized in that The determining the target position in the target lane of the first background image includes: Performing a sliding window operation on the first foreground image in the first background image, and calculating the overlap value between the first foreground image and the target lane when the first foreground image is respectively at P positions in the first background image, where P is an integer greater than 1; Compare the P overlap values ​​with the preset values ​​respectively; Determine a position corresponding to an overlap value greater than the preset value as a position to be determined; When the number of the positions to be determined is 1, the position to be determined is determined to be the target position.

8. The method according to claim 7, characterized in that The determining of the target position in the target lane of the first background image further includes: When the number of the positions to be determined is greater than 1, determining one of the multiple positions to be determined as the target position; or, When the number of the positions to be determined is greater than 1, one of the positions for which road surface information does not exist among the positions to be determined is determined as the target position.

9. The method according to claim 1, characterized in that: The step of fusing the first foreground image with the target lane of the first background image to generate a first target image includes: The first foreground image is fused with the target lane of the first background image, and image attributes of the first foreground image are adjusted to be consistent with image attributes of the first background image to generate the first target image.

10. An image generating device, characterized in that: The device comprises: A first acquisition module is used to acquire a first background image, wherein the first background image at least includes: image information, a drivable road surface area, and a first position of a lane line in the drivable road surface area; A second acquisition module is used to acquire a first foreground image, wherein the first foreground image at least includes: road surface information in the target lane, and the road surface information includes: road signs and / or tire marks on the road surface; A first generating module, configured to determine a target position in a target lane of the first background image, fuse the first foreground image with the target lane of the first background image, and generate a first target image; The first generating module is specifically used for: Acquire a first size of the first foreground image and a second size of a target lane of the first background image; When the first size is less than or equal to the second size, fusing the first foreground image with the target lane of the first background image to generate the first target image; When the first size is larger than the second size, the first foreground image is fused with the drivable road area of ​​the first background image according to the first position of the lane line in the drivable road area to generate the first target image, in which the lane line in the drivable road area is placed above the first foreground image.

11. An electronic device, characterized in that: Including memory, transceiver, processor: Memory for storing computer programs; a transceiver, for transmitting and receiving data under the control of the processor; A processor, configured to read the computer program in the memory and execute the image generating method according to any one of claims 1 to 9.

12. A processor-readable storage medium, characterized in that: The processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the image generating method according to any one of claims 1 to 9.

Images