Lane line detection simulation data construction method and system and electronic equipment

Abstract

The invention provides a lane line detection simulation data construction method and system and electronic equipment. The method comprises the steps that character data and background image data are acquired; segmenting each background image in the background image data to obtain a drivable area mask; obtaining a lane line mask and a target frame according to a pre-marked lane line true value and a ground target true value in the background image data; dynamically acquiring a coordinate value of each character in the character data in the background image according to the drivable area mask, the lane line mask and the target frame, and pasting the characters on the background image according to the coordinate values to obtain map data; and performing image post-processing on the chartlet data to obtain simulation image data. A mask of a drivable area is obtained in advance through a segmentation model, so that the mapping range of road surface characters is roughly positioned, and the authenticity of a simulation image is preliminarily ensured; and further determining a lane line mask on the basis of the drivable area mask so as to determine the mapping range of the characters, so that the characters do not press lines, and the authenticity of the simulation image is further improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a method, system and electronic equipment for constructing lane line detection simulation data. Background Art

[0002] Lane detection is a crucial component for ensuring stable vehicle operation during autonomous driving. It aims to identify and locate lane markings on the road, providing critical information for applications such as autonomous driving and assisted driving. However, it's easy for detection systems to mistakenly detect road markings as lane markings. This is because road markings and lane markings may share similarities in shape, color, and texture, making it difficult for detection algorithms to accurately distinguish them. Specifically, some road markings may appear straight and continuous, closely resembling the shape of lane markings. Furthermore, in poor lighting conditions or unusual road conditions, the color of road markings may be confused with that of lane markings due to reflections, increasing the risk of misjudgment.

[0003] In related technologies, the problem of lane marking and character misdetection is primarily due to the low proportion of scene samples containing text in the training dataset. Although expanding the training data with character annotations can theoretically improve model robustness, in practice, manual collection not only requires significant human and material resources, but also takes a long time. If automated simulation images are used, characters can easily interfere with lane markings or vehicles, affecting the authenticity of the images. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, system and electronic device for constructing lane line detection simulation data, which solves the problem that characters easily interfere with lane lines or vehicles, affecting the authenticity of the image.

[0005] To this end, in a first aspect, the present invention provides a method for constructing lane line detection simulation data, comprising the following steps:

[0006] Get character data and background image data;

[0007] Segmenting each background image in the background image data to obtain a drivable area mask;

[0008] Obtaining a lane line mask and a target frame according to the lane line true value and the ground target true value pre-labeled in the background image data;

[0009] The coordinate value of each character in the character data in the background image is dynamically obtained according to the drivable area mask, lane line mask and target frame, and the character is pasted on the background image according to the coordinate value to obtain the mapping data.

[0010] Optionally, segmenting each background image in the background image data to obtain a drivable area mask includes:

[0011] Each background image in the background image data is segmented by an image segmentation model to obtain a drivable area mask, and a set of coordinate points of the drivable area is obtained according to the drivable area mask.

[0012] Optionally, the coordinate value of each character in the character data in the background image is dynamically obtained according to the drivable area mask, lane mark mask and target frame, and the character is pasted on the background image according to the coordinate value to obtain the mapping data:

[0013] Determine the initial coordinates of the character in the background image according to the coordinate point set of the drivable area, and paste the character on the background image according to the initial coordinates;

[0014] Scale the characters and adjust their height;

[0015] Perform conflict detection on the characters. If the conflict detection fails, the initial coordinates of the characters in the background image are re-determined; if the conflict detection passes, the mapping is completed.

[0016] Optionally, determining the initial coordinates of the character in the background image according to the coordinate point set of the drivable area, and pasting the character on the background image according to the initial coordinates includes:

[0017] A point is randomly selected from the coordinate point set of the drivable area as the initial coordinate of the character, and the character is pasted on the background image according to the initial coordinate. This is repeated K times, where K is the number of characters that need to be pasted on each background image.

[0018] Optionally, scaling the characters to adjust the height of the characters includes:

[0019] Scale the characters under 3D geometric constraints and adjust the height of the characters.

[0020] Optionally, scaling the characters under 3D geometric constraints to adjust the height of the characters includes:

[0021] Calculate the character height based on the camera focal length, distance, and actual height, including:

[0022] h pixel =f y *(h real / d)

[0023] Among them, h pixel is the height of the character, f y is the focal length of the camera, h real is the actual height and d is the distance.

[0024] Optionally, performing conflict detection on the characters, and if the conflict detection fails, re-determining the initial coordinates of the characters in the background image; if the conflict detection passes, completing the mapping includes:

[0025] Calculate the intersection-over-union (IoU) of the character with the target box on the background image, the lane mask, and the previously successful characters. If any of the IoUs is not 0, conflict detection fails and the character's initial coordinates in the background image are re-determined. If all IoUs are 0, conflict detection passes and mapping is complete.

[0026] Optionally, the method further includes performing image post-processing on the mapping data to obtain simulated image data.

[0027] Optionally, the image post-processing includes boundary smoothing, illumination fusion and color correction.

[0028] In a second aspect, a lane detection simulation data construction device is provided, comprising:

[0029] Data acquisition module, used to acquire character data and background image data;

[0030] A drivable area mask acquisition module is used to segment each background image in the background image data to obtain a drivable area mask;

[0031] A lane mask acquisition module is used to obtain a lane mask and a target frame based on the true values ​​of the lane lines and the true values ​​of the ground objects pre-labeled in the background image data;

[0032] The mapping module is used to dynamically obtain the coordinate value of each character in the character data in the background image based on the drivable area mask, lane line mask and target frame, and paste the character on the background image according to the coordinate value to obtain the mapping data.

[0033] Optionally, an image post-processing module is also included, which is used to perform image post-processing on the mapping data to obtain simulated image data.

[0034] In a third aspect, an electronic device is provided, including a memory and a processor;

[0035] The memory stores computer-executable instructions;

[0036] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the lane detection simulation data construction method.

[0037] Beneficial effects:

[0038] (1) The present disclosure provides a method, system, and electronic device for constructing lane line detection simulation data. A mask of a drivable area is obtained in advance through a segmentation model to roughly locate the mapping range of road surface characters, thereby preliminarily ensuring the authenticity of the simulated image. A lane line mask is further determined based on the mask of the drivable area to determine the mapping range of the characters, so that the characters will not cross the line, thereby further improving the authenticity of the simulated image.

[0039] (2) This disclosure utilizes image post-processing methods such as boundary smoothing, illumination fusion, and color correction to avoid boundary artifacts when characters are pasted onto a background image, eliminate color differences at the edges of characters, remove visible seams, and incorporate illumination variations. Color correction is also used to perform color conversion, further enhancing the authenticity of the simulated image.

[0040] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A method flow chart of a lane detection simulation data construction method disclosed in the present invention;

[0043] Figure 2 A flowchart of a method for constructing lane detection simulation data in the present disclosure;

[0044] Figure 3 A system structure diagram of a lane detection simulation data construction system in the present disclosure;

[0045] Figure 4 This is a schematic structural diagram of an electronic device in the present disclosure.

[0046] In the figure, 101 is a data acquisition module, 102 is a drivable area mask acquisition module, 103 is a lane line mask acquisition module, 104 is a mapping module, 105 is an image post-processing module, 200 is an electronic device, 210 is a processor, 220 is a memory, 230 is a communication component, and 240 is a bus. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0048] In the specification and claims of this application, as well as in the accompanying drawings, the terms "first," "second," "third," "fourth," and the like are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that such terms are interchangeable where appropriate. For example, first information could be referred to as second information, and similarly, second information could be referred to as first information without departing from the scope of this disclosure.

[0049] The word "if" as used herein may be interpreted as "when" or "when" or "in response to determining," depending on the context.

[0050] Furthermore, as used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise.

[0051] It should be further understood that the terms “comprises” and “includes” indicate the existence of features, steps, operations, elements, components, items, types, and / or groups, but do not preclude the existence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, types, and / or groups.

[0052] The terms "or" and "and / or" as used herein are to be interpreted as inclusive, or mean any one or any combination. Thus, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C." An exception to this definition occurs only when a combination of elements, functions, steps, or operations are inherently mutually exclusive in some manner.

[0053] Lane detection is a crucial component for ensuring stable vehicle operation during autonomous driving. It aims to identify and locate lane markings on the road, providing critical information for applications such as autonomous driving and assisted driving. However, it's easy for detection systems to mistakenly detect road markings as lane markings. This is because road markings and lane markings may share similarities in shape, color, and texture, making it difficult for detection algorithms to accurately distinguish them. Specifically, some road markings may appear straight and continuous, closely resembling the shape of lane markings. Furthermore, in poor lighting conditions or unusual road conditions, the color of road markings may be confused with that of lane markings due to reflections, increasing the risk of misjudgment.

[0054] In the related art, the problem of lane line and character misdetection is mainly due to the low proportion of scene samples containing text information in the training data set to the total scene samples. Although the robustness of the model can be improved in theory by expanding the training data with character annotations, the manual collection process in actual applications not only requires a lot of human resources and material costs, but also takes a long time. If the method of automatically generating simulated images is adopted, the characters are likely to interfere with the lane lines or vehicles, affecting the authenticity of the image. To this end, in the first aspect, the present disclosure provides the following Figure 1 A method for constructing lane detection simulation data shown includes the following steps:

[0055] S100, obtaining character data and background image data;

[0056] Character data consists of representative character images, such as text or arrows, annotated with the character annotations to form character data. Based on the character annotations, accurate character masks can be obtained through data processing. The character data is then cropped based on the character mask area to obtain the true character shape.

[0057] The background image data is a real lane line background image. The lane line true value and ground target true value are marked on the background image to provide a basis for the subsequent character mapping range.

[0058] S200, segmenting each background image in the background image data to obtain a drivable area mask;

[0059] Each background image in the background image data is segmented by an image segmentation model to obtain a drivable area mask, and a set of coordinate points of the drivable area is obtained according to the drivable area mask.

[0060] The image segmentation model is used to segment different objects or areas in the image, so that the computer can clearly understand the specific location and range of each object, and then accurately identify each target. The optional image segmentation model is FCN, U-Net, SegNet or Mask R-CNN. The drivable area is the area within the lane line in the background image where the vehicle can drive. The drivable area in the background image is segmented by the image segmentation model to obtain a drivable area mask. In the drivable area mask, the pixel value of the drivable area is 255, and the pixel value of the non-drivable area is 0. By filtering the drivable area mask with non-zero values, the coordinate point set of the drivable area can be obtained. The coordinate point set provides the basis for determining the initial coordinates of subsequent characters.

[0061] S300, obtaining a lane line mask and a target frame according to the lane line true value and the ground target true value pre-labeled in the background image data;

[0062] A lane mask is generated based on the true lane values ​​pre-annotated in the background image data. The true lane values ​​are the positions of the lane lines pre-annotated in the background image. Pixels in the lane mask with lane lines have a value of 255, while pixels without lane lines have a value of 0. By filtering the lane mask for non-zero values, a set of lane line coordinate points is obtained.

[0063] The ground target truth value in the background image marks the ground targets such as vehicles, pedestrians, obstacles and other ground targets with annotation boxes, including the coordinates of the upper left corner, lower right corner, width and height of the annotation box where the target is located, so as to calculate the coordinates of the four vertices of the annotation box.

[0064] S400, dynamically obtaining the coordinate value of each character in the character data in the background image according to the drivable area mask, the lane mark mask, and the target frame, and pasting the character on the background image according to the coordinate value to obtain the mapping data;

[0065] like Figure 2 As shown, including:

[0066] S401, determining the initial coordinates of the character in the background image based on the coordinate point set of the drivable area, and pasting the character on the background image based on the initial coordinates;

[0067] A point is randomly selected from the coordinate point set of the drivable area as the initial coordinate of the character, and the character is pasted on the background image according to the initial coordinate. This is repeated K times, where K is the number of characters that need to be pasted on each background image.

[0068] S402, scaling the characters to adjust their height;

[0069] Scale the characters under 3D geometric constraints and adjust the height of the characters.

[0070] Calculate the character height based on the camera focal length, distance, and actual height, including:

[0071] h pixel =f y *(h real / d)

[0072] Among them, h pixel is the height of the character, f y is the focal length of the camera, h real is the actual height and d is the distance.

[0073] The coverage of characters is determined by scaling the characters to ensure that the character size conforms to the perspective relationship and avoid proportional distortion.

[0074] S403, performing a conflict detection on the characters. If the conflict detection fails, the initial coordinates of the characters in the background image are re-determined; if the conflict detection passes, the mapping is completed.

[0075] Calculate the intersection-of-union (IoU) ratio of the character with the target box on the background image, the lane mask, and the previously successfully mapped character. If any of the IoU ratios is not 0, conflict detection fails and the character's initial coordinates in the background image are re-determined. If all IoU ratios are 0, conflict detection passes and mapping is complete. Previously successfully mapped characters are the frames where characters were successfully mapped.

[0076] For each background image, traverse each character in the character data. If the character passes the conflict detection after being pasted on the background image, continue to traverse the next character. If no suitable pasting point can be found in the background image, skip the character and continue to traverse the next character.

[0077] S500: Perform image post-processing on the mapping data to obtain simulation image data.

[0078] The image post-processing includes boundary smoothing, illumination fusion and color correction.

[0079] Among them, boundary smoothing blurs the image through a Gaussian function, eliminates image noise and reduces the chromatic aberration of high-contrast edges in the image.

[0080] Lighting Blending removes visible seams and adds lighting variation using Poisson blending.

[0081] Boundary smoothing and color correction can be used to smooth the seams between pasted characters and background images, suppressing boundary artifacts.

[0082] Color correction blends the pasted characters with the background image through color conversion. In one embodiment, the HSV space is used to provide color conversion. Taking the saturation channel as an example, the saturation of the pasted characters is changed according to the following ratio:

[0083] s=M s / S

[0084] Among them, R S and M s Represents the average saturation of the background map area and the pasted characters respectively.

[0085] The mapping data are aggregated after image post-processing to obtain simulated image data with lane line true values, ground target true values ​​and characters.

[0086] Mixing the simulated image data generated above with real images and using them together as a training set to train the lane line detection model can alleviate the problem of character misdetection caused by too little data with characters.

[0087] Second, as Figure 3 As shown, a lane detection simulation data construction device is provided, comprising:

[0088] Data acquisition module 101, used to acquire character data and background image data;

[0089] a drivable area mask acquisition module 102 for segmenting each background image in the background image data to obtain a drivable area mask;

[0090] A lane mask acquisition module 103 is configured to obtain a lane mask and a target frame based on the true values ​​of the lanes and the ground objects pre-labeled in the background image data;

[0091] The mapping module 104 is used to dynamically obtain the coordinate value of each character in the character data in the background image based on the drivable area mask, lane mark mask and target frame, and to map the character on the background image according to the coordinate value to obtain mapping data;

[0092] The image post-processing module 105 is used to perform image post-processing on the mapping data to obtain simulated image data.

[0093] Thirdly, as Figure 4 As shown, an electronic device is provided, characterized in that it includes: a memory, a processor;

[0094] The memory stores computer-executable instructions;

[0095] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the above method.

[0096] In one embodiment, the electronic device 200 includes: at least one processor 210 and a memory 220. Optionally, the electronic device 200 further includes a communication component 230. The processor 210, the memory 220, and the communication component 230 are connected via a bus 240.

[0097] During the specific implementation process, at least one processor 210 executes the computer-executable instructions stored in the memory 220, so that the at least one processor 210 performs the above method.

[0098] The specific implementation process of the processor 210 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0099] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0100] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0101] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0102] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A method for constructing lane line detection simulation data, characterized in that: The steps include: Get character data and background image data; Segmenting each background image in the background image data to obtain a drivable area mask; Obtaining a lane line mask and a target frame according to the lane line true value and the ground target true value pre-labeled in the background image data; The coordinate value of each character in the character data in the background image is dynamically obtained according to the drivable area mask, lane line mask and target frame, and the character is pasted on the background image according to the coordinate value to obtain the mapping data.

2. The method for constructing lane line detection simulation data according to claim 1, characterized in that: The step of segmenting each background image in the background image data to obtain a drivable area mask includes: Each background image in the background image data is segmented by an image segmentation model to obtain a drivable area mask, and a set of coordinate points of the drivable area is obtained according to the drivable area mask.

3. The method for constructing lane line detection simulation data according to claim 2, wherein: The coordinate value of each character in the character data in the background image is dynamically obtained based on the drivable area mask, lane line mask and target frame, and the character is pasted on the background image according to the coordinate value to obtain the mapping data: Determine the initial coordinates of the character in the background image according to the coordinate point set of the drivable area, and paste the character on the background image according to the initial coordinates; Scale the characters and adjust their height; Perform conflict detection on the characters. If the conflict detection fails, redetermine the initial coordinates of the characters in the background image. If the conflict detection passes, the mapping is completed.

4. The method for constructing lane line detection simulation data according to claim 3, wherein: The determining of the initial coordinates of the character in the background image according to the coordinate point set of the drivable area, and pasting the character on the background image according to the initial coordinates comprises: A point is randomly selected from the coordinate point set of the drivable area as the initial coordinate of the character, and the character is pasted on the background image according to the initial coordinate. This is repeated K times, where K is the number of characters that need to be pasted on each background image.

5. The method for constructing lane line detection simulation data according to claim 3, wherein: Scaling the characters and adjusting the height of the characters includes: Scale the characters under 3D geometric constraints and adjust the height of the characters.

6. The method for constructing lane line detection simulation data according to claim 5, characterized in that: Scaling the characters under the 3D geometric constraints and adjusting the height of the characters includes: Calculate the character height based on the camera focal length, distance, and actual height, including: h pixel =f y *(h real / d) Among them, h pixel is the height of the character, f y is the focal length of the camera, h real is the actual height and d is the distance.

7. The method for constructing lane line detection simulation data according to claim 3, wherein: The character is subjected to conflict detection, and if the conflict detection fails, the initial coordinates of the character in the background image are re-determined; If the conflict detection passes, the mapping is completed including: Calculate the intersection-over-union (IoU) of the character with the target box on the background image, the lane mask, and the previously successful characters. If any of the IoUs is not 0, conflict detection fails and the character's initial coordinates in the background image are re-determined. If all IoUs are 0, conflict detection passes and mapping is complete.

8. The method for constructing lane line detection simulation data according to claim 1, wherein: The method also includes performing image post-processing on the mapping data to obtain simulation image data.

9. The method for constructing lane line detection simulation data according to claim 8, characterized in that: The image post-processing includes boundary smoothing, illumination fusion and color correction.

10. A lane detection simulation data construction device, characterized in that: include: Data acquisition module, used to acquire character data and background image data; a drivable area mask acquisition module, configured to segment each background image in the background image data to obtain a drivable area mask; A lane mask acquisition module is used to obtain a lane mask and a target frame based on the true values ​​of the lane lines and the true values ​​of the ground objects pre-labeled in the background image data; The mapping module is used to dynamically obtain the coordinate value of each character in the character data in the background image based on the drivable area mask, lane line mask and target frame, and paste the character on the background image according to the coordinate value to obtain the mapping data.

11. The lane detection simulation data construction device according to claim 10, characterized in that: It also includes an image post-processing module for performing image post-processing on the mapping data to obtain simulated image data.

12. An electronic device, characterized in that: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs a method according to any one of claims 1 to 9.

Images