Lane line detection method, model training method and device

Abstract

The invention provides a lane line detection method and device and a model training method and device, and relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, computer vision, automatic driving and electronic navigation. The specific implementation scheme of the lane line detection method comprises the steps of generating target offset information based on image features of a to-be-detected image; wherein the target offset information indicates displacement of a target position for each associated feature relative to a preset position in the feature enhancement process, and each associated feature is associated with the image feature; performing feature enhancement on the image features based on the target offset information to obtain enhanced features; wherein the enhanced feature enhances the deformation feature of the lane line in the to-be-detected image; the enhanced features are detected, and a detection result is generated; wherein the detection result indicates the position and the type of the lane line.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technology, in particular to deep learning, computer vision, autonomous driving, and electronic navigation technology, and specifically to a lane line detection method, a model training method, and a device. Background Art

[0002] With the development of artificial intelligence technology, it has been widely used in many fields such as electronic navigation and autonomous driving.

[0003] On complex roads with a crisscross pattern of lanes of varying types, accurate lane information is a prerequisite for safe driving. Therefore, a method that can accurately detect the location and type of lanes is urgently needed. Summary of the Invention

[0004] The present disclosure provides a lane line detection method, a model training method and a device.

[0005] According to one aspect of the present disclosure, a lane line detection method is provided, comprising: generating target offset information based on image features of an image to be detected; wherein the target offset information indicates the displacement of a target position of each associated feature relative to a predetermined position during a feature enhancement process, wherein each associated feature is associated with an image feature; performing feature enhancement on the image features based on the target offset information to obtain enhanced features; wherein the enhanced features strengthen the deformation features of the lane line in the image to be detected; and detecting the enhanced features to generate a detection result; wherein the detection result indicates the position and type of the lane line.

[0006] According to another aspect of the present disclosure, a model training method is provided, in which an initial model includes: a feature enhancement module and a detection module, and the model training method includes: using the feature enhancement module to generate target sample offset information based on sample image features of a sample image; and based on the target sample offset information, feature enhancing the sample image features to obtain enhanced sample features; wherein the enhanced sample features strengthen the deformation features of the sample lane lines in the sample image, and the target sample offset information indicates the displacement of the sample target position of each sample-associated feature relative to the sample predetermined position during the feature enhancement process; each sample-associated feature is associated with the sample image feature; using the detection module to detect the enhanced sample features to generate a detection result; wherein the detection result indicates the sample position and sample type of the sample lane line; and based on the loss value between the detection result and the sample label, training the initial model to generate a model for detecting lane lines; wherein the sample label indicates the actual position and actual type of the sample lane lines in the sample image.

[0007] According to another aspect of the present disclosure, a lane line detection device is provided, including: a first generating module, an obtaining module and a second generating module.

[0008] The first generation module is used for generating target offset information based on the image features of the image to be detected; wherein the target offset information indicates the displacement of the target position of each associated feature relative to the predetermined position during the feature enhancement process, and each associated feature is associated with the image feature.

[0009] The acquisition module is used to enhance the image features based on the target offset information to obtain enhanced features; wherein the enhanced features strengthen the deformation characteristics of the lane lines in the image to be detected.

[0010] The second generation module is used to detect the enhanced features and generate a detection result; wherein the detection result indicates the position and type of the lane line.

[0011] According to another aspect of the present disclosure, a model training device is provided, wherein the initial model includes: a feature enhancement module and a detection module, and the model training device includes: a feature enhancement module, a detection module and a training module.

[0012] A feature enhancement module is used to generate target sample offset information based on the sample image features of the sample image; and based on the target sample offset information, the sample image features are feature enhanced to obtain enhanced sample features; wherein the enhanced sample features strengthen the deformation features of the sample lane lines in the sample image, and the target sample offset information indicates the displacement of the sample target position relative to the sample predetermined position for each sample-associated feature during the feature enhancement process; each sample-associated feature is associated with the sample image feature.

[0013] The detection module is used to detect the enhanced sample features and generate a detection result; wherein the detection result indicates the sample position and sample type of the sample lane line.

[0014] A training module is used to train the initial model based on the loss value between the detection results and the sample labels to generate a model for detecting lane lines; wherein the sample labels indicate the actual position and actual type of the sample lane lines in the sample image.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described above.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described above.

[0017] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method described above when executed by a processor.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0020] Figure 1 Schematically illustrates an exemplary system architecture to which a lane detection method or a model training method and apparatus according to an embodiment of the present disclosure may be applied;

[0021] Figure 2 The following schematically shows a flow chart of a lane line detection method according to an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of the offset position of an associated feature associated with an image feature perceived based on target offset information according to an embodiment of the present disclosure is shown;

[0023] Figure 4 The following schematically shows a neural network model architecture diagram for detecting lane lines according to an embodiment of the present disclosure;

[0024] Figure 5 Schematically shows a network architecture diagram of an LDA module according to an embodiment of the present disclosure;

[0025] Figure 6 The following schematically shows a flow chart of a model training method according to an embodiment of the present disclosure;

[0026] Figure 7 A block diagram of a lane detection device according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 8 A block diagram of a model training device according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 9 A block diagram of an electronic device suitable for implementing a lane line detection method or a model training method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] As road environments become increasingly complex, the types of road markings are proliferating. Some neural network models used to detect lane markings typically use a fixed sampling pattern for feature extraction. Because lane markings in images of road environments captured by image acquisition devices can be curved, occluded, or deformed, features extracted using a fixed sampling pattern are difficult to adaptively adjust sampling positions, reducing lane detection accuracy.

[0031] In light of this, embodiments of the present disclosure provide a lane detection method that, based on the image features of an image to be detected, generates target offset information indicating the displacement of the target position of each associated feature relative to a predetermined position during feature enhancement. Based on this target offset information, the image features are enhanced, thereby strengthening the deformation characteristics of the lane lines in the image to be detected, improving the ability to perceive lane deformation characteristics, and further enhancing the accuracy of lane detection.

[0032] Figure 1 An exemplary system architecture to which the lane line detection method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0033] It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure. This does not mean that the embodiments of the present disclosure cannot be applied to other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the lane detection method and apparatus can be applied may include a terminal device, but the terminal device may implement the lane detection method and apparatus provided by the embodiments of the present disclosure without interacting with a server.

[0034] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a terminal device 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired and / or wireless communication links, etc.

[0035] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Terminal device 101 may be installed with various communication client applications, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only). Terminal device 101 may be installed in an autonomous vehicle or any other electronic device capable of interacting with an autonomous vehicle.

[0036] The terminal device 101 may be any electronic device having a display screen and supporting web browsing, including but not limited to a vehicle-mounted terminal, a smart phone, a tablet computer, a laptop computer, a desktop computer, and the like.

[0037] Server 103 may be a server that provides various services, such as a background management server (for example only) that supports content browsed by a user using terminal device 101. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal device.

[0038] It should be noted that the lane line detection method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101. Accordingly, the lane line detection apparatus provided in the embodiment of the present disclosure can also be provided in the terminal device 101.

[0039] Alternatively, the lane line detection method provided in the embodiment of the present disclosure may also be generally executed by the server 103. Accordingly, the lane line detection device provided in the embodiment of the present disclosure may generally be set in the server 103. The lane line detection method provided in the embodiment of the present disclosure may also be executed by a server or server cluster that is different from the server 103 and can communicate with the terminal device 101 and / or the server 103. Accordingly, the lane line detection device provided in the embodiment of the present disclosure may also be set in a server or server cluster that is different from the server 103 and can communicate with the terminal device 101 and / or the server 103.

[0040] For example, when a user is driving a vehicle, the terminal device 101 may acquire an image to be detected 110 and then send the image to be detected to the server 103. The server 103 may detect the image to be detected, generate a detection result, and feed the detection result back to the terminal device 101. Alternatively, a server or server cluster capable of communicating with the terminal device 101 and / or the server 103 may detect the image to be detected and ultimately generate a detection result including a lane position 112 and a lane type 111.

[0041] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0042] Figure 2 The flowchart of the lane line detection method according to an embodiment of the present disclosure is schematically shown.

[0043] like Figure 2 As shown, the method 200 includes operations S210 to S230.

[0044] In operation S210 , target offset information is generated based on image features of an image to be detected.

[0045] In operation S220 , feature enhancement is performed on the image features based on the target offset information to obtain enhanced features.

[0046] In operation S230 , the enhanced features are detected to generate a detection result.

[0047] According to an embodiment of the present disclosure, the image to be detected may be captured using an image acquisition device, which may be configured on an autonomous vehicle or in other road environment monitoring equipment. The image to be detected may include multiple objects such as roads, lane lines, vehicles ahead, sky, and trees. The image features of the image to be detected may be extracted using a feature extraction module in a trained neural network model. For example, the feature extraction module may be a CNN (Convolutional Neural Network). The image to be detected is input into the feature extraction module to obtain an initial feature map of the image to be detected. The initial feature map may include multiple initial features. Among the multiple initial features, the initial feature for the lane line may be used as an image feature. In the initial feature map, features adjacent to the image feature may be used as associated features of the image feature. In one example, one or more initial features in the initial feature map whose distance from the image feature is less than or equal to a preset distance threshold may be used as one or more associated features of the image feature.

[0048] In the embodiment of the present disclosure, each associated feature is associated with an image feature, and the target offset information indicates the displacement of the target position of each associated feature relative to a predetermined position during the feature enhancement process.

[0049] In some embodiments, the predetermined position may be a fixed sampling position of a convolution kernel in a fixed sampling mode in a convolutional neural network. Because the image to be detected may be deformed or occluded, lane line information may not be obtained based on the features at the predetermined position.

[0050] In the disclosed embodiment, the associated features corresponding to the offset target position can all capture lane line information, so as to perform feature enhancement on the image features based on the target offset information to obtain enhanced features.

[0051] According to an embodiment of the present disclosure, the enhanced features strengthen the deformation features of the lane lines in the image to be detected.

[0052] For example, lane line deformation features may include, but are not limited to, horizontal shape features and vertical line type features. Shape features can be used to characterize the horizontal shape of lane lines, such as curves, straight lines, and dual lanes. Line type features can be used to characterize the vertical line type of lane lines, such as solid lines and dashed lines.

[0053] Since the enhanced features strengthen the deformation characteristics of the lane lines in the image to be detected, detection is performed on the basis of fully perceiving the deformation information of the lane lines in the image to be detected, and the detection results generated for indicating the position and type of the lane lines are more accurate.

[0054] The embodiment of the present disclosure provides a lane line detection method, which generates target offset information indicating the displacement of the target position of each associated feature relative to the predetermined position during the feature enhancement process based on the image features of the image to be detected. Based on the target offset information, the image features are enhanced, the deformation features of the lane lines in the image to be detected are strengthened, the perception ability of the lane line deformation features is improved, and the accuracy of lane line detection is further improved. Figure 3 The association features and image features of the present disclosure are described.

[0055] Figure 3 The figure schematically shows the offset position of the associated feature associated with the image feature perceived based on the target offset information according to an embodiment of the present disclosure.

[0056] like Figure 3 As shown, the predetermined positions of the image features and the associated features are shown in a nine-square grid in the image to be detected 310. For example, the image feature is located at the position marked "4" in the center of the nine-square grid, and the associated features are located at the positions marked "0, 1, 2, 3, 5, 7, 8, 9" around the center of the nine-square grid.

[0057] Since the associated features located at the predetermined position do not capture the lane line information, in the embodiment of the present disclosure, the associated features containing the lane line information can be captured by dynamically generating offset information.

[0058] like Figure 3As shown, through dynamic offset, the predetermined positions that failed to capture the lane line information are all offset, resulting in image 320. For example, the predetermined positions of the markers "0, 1, 2" are offset to the upper left, the predetermined position of the marker "3" is offset upward, the predetermined position of the markers "5, 7, 8" is offset to the lower right, and the predetermined position of the marker "9" can capture the lane line information, so no offset occurs.

[0059] It is understandable that the displacement of each predetermined position relative to each target position may be different, and the target offset information will be adaptively generated according to the position of the lane line in the image to be detected. This ability to adaptively generate target offset information based on the input image to be detected is based on the loss value between the detection results output by the neural network model and the sample label learned during the training process of the neural network model.

[0060] The following combination Figure 4 and Figure 5 The lane detection method described above is explained in detail.

[0061] Figure 4 The figure schematically shows a neural network model architecture diagram for detecting lane lines according to an embodiment of the present disclosure.

[0062] like Figure 4 As shown, the neural network model architecture 400 may include: Stem (Spatio-Temporal Embedding Module) 410, FPN (Feature Pyramid Networks) 420 and a detection module 430.

[0063] In the disclosed embodiment, FPN 420 may include three stages, each of which may include a Residual Network (ResNet) and a Lane-aware Deformable Aggregation (LDA). The number of channels in the network structure of each stage may be different.

[0064] For example, the first stage, Stage 1421, may include 64-channel ResNet 14211 and 64-channel LDA 14212. The second stage, Stage 2422, may include 128-channel ResNet 24221 and 128-channel LDA 24222. The third stage, Stage 3423, may include 256-channel ResNet 34231 and 256-channel LDA 34232.

[0065] In the embodiment of the present disclosure, the image to be detected 110 is input into Stem410 and outputs image features through spatial sampling and temporal modeling. Then, the image features can be processed using 64 independent convolution kernels using 64 channels of ResNet14211 to obtain image features at a scale of 1 / 4. Similarly, the image features can be processed using 128 independent convolution kernels using 128 channels of ResNet24221 to obtain image features at a scale of 1 / 8. The image features can also be processed using 256 independent convolution kernels using 256 channels of ResNet34231 to obtain image features at a scale of 1 / 16.

[0066] Then, the 1 / 4 scale image features can be input into the 64-channel LDA14212, the 1 / 8 scale image features can be input into the 128-channel LDA24222, and the 1 / 16 scale image features can be input into the 256-channel LDA34232. After feature enhancement, the features are added horizontally to generate fused features.

[0067] Finally, the fused features are input into the detection module 430 , which outputs the lane type 111 and the lane position 112 .

[0068] The following combination Figure 5 The network architecture of the LDA module in the embodiment of the present disclosure is described in detail to further illustrate the lane line detection method described above.

[0069] Figure 5 The network architecture diagram of the LDA module according to an embodiment of the present disclosure is schematically shown.

[0070] In an embodiment of the present disclosure, generating target offset information based on image features of an image to be detected may include the following operations: performing convolution on the image features along a first direction to obtain a first feature; performing convolution on the first feature along a second direction to obtain a second feature; and generating target offset information based on the second feature.

[0071] In some embodiments, the first direction and the second direction are perpendicular to each other. For example, the first direction may be a horizontal direction, and the second direction may be a vertical direction.

[0072] like Figure 5 As shown, first, the image features can be input into the two-dimensional convolution layer Conv 2D 3×1 (Convolution two-Dimensional) 501 with a 3×1 convolution kernel, and convolution is performed along the vertical direction to obtain the first feature V1(x). For example, after obtaining the convolution result along the vertical direction, the convolution result can be normalized and activated in sequence, as shown in formula (1):

[0073] (1)

[0074] Where x represents the image feature; ReLU (Rectified Linear Unit) represents the rectified linear unit function; BN (Batch Normalization) represents the normalization function; Conv() represents two-dimensional convolution, and 3×1 represents the size of the convolution kernel.

[0075] Then, the first feature can be input into the two-dimensional convolution layer Conv 2D 1×3 502 of the 1×3 convolution kernel, and convolution is performed along the horizontal direction to obtain the second feature V out (x). For example, after obtaining the convolution result along the horizontal direction, the convolution result can be normalized and activated in sequence, as shown in formula (2):

[0076] (2)

[0077] Among them, 1×3 represents the size of the convolution kernel.

[0078] Finally, the second feature is provided to a two-dimensional convolution layer Conv 2D 3×3 503 with a 3×3 convolution kernel to generate target offset information.

[0079] It's understandable that the texture features of lane markings are distributed vertically relative to the vehicle's travel direction. Therefore, performing convolution along the vertical direction first captures the associated features of lane markings along a longer distance in the vehicle's travel direction. Performing convolution along the horizontal direction captures the associated features of lane markings in that direction, further improving the accuracy of representing the lane markings' geometric deformation features and enabling adaptive adjustment of the offset information of the associated features.

[0080] In some embodiments, generating target offset information based on the second feature may include the following operations: performing convolution processing on the second feature to obtain initial offset information; and scaling the initial offset information according to a predetermined scaling factor to obtain target offset information.

[0081] like Figure 5 As shown, the second feature is input into the two-dimensional convolution layer Conv 2D 3×3 503 with a 3×3 convolution kernel to generate the initial offset information. As shown in formula (3):

[0082] (3)

[0083] Among them, offsets raw Indicates the initial offset information. Conv offset () represents a two-dimensional convolution with a convolution kernel of 3x3.

[0084] To further improve the stability of the model, the generated offset information can be constrained to a predetermined range, for example: [-1, 1].

[0085] Therefore, the nonlinear activation function tanh can be used to constrain the target offset information to a predetermined range. As shown in formula (4):

[0086] (4)

[0087] Among them, offsets modulated represents the target offset information, and s represents the predetermined scaling factor.

[0088] In an embodiment of the present disclosure, the initial value of the predetermined scaling factor can be 1.5. The predetermined scaling factor is a learnable variable. When the initial offset information of certain feature points is scaled based on the initial value during the detection process, and the scaled offset information is still outside the predetermined range, the model can automatically adjust the value of the predetermined scaling factor so that the target offset information is within the predetermined range, further improving the stability of the model.

[0089] In an embodiment of the present disclosure, feature enhancement of image features is performed based on target offset information to obtain enhanced features, which may include the following operations: determining the target position of each associated feature based on the target offset information and the predetermined position of each associated feature; and feature enhancement of image features based on the target position of each associated feature to generate enhanced features.

[0090] like Figure 5 As shown, the image features and target offset information can be input into Deformable Conv 2D 504 (Deformable Convolution two-Dimensional, two-dimensional deformable convolution layer), and the enhanced features are output. As shown in formula (5):

[0091] (5)

[0092] in, Represents the weight of the convolution kernel; R represents the receptive field area; x represents the image feature; p0 represents the pixel position of the image feature; p n Indicates the intended location of the associated feature; Indicates target offset information.

[0093] Based on the target offset information, the position of the associated features can be adaptively adjusted during the feature enhancement process, and the deformation features such as the structure and line shape of the lane line can be captured, further improving the accuracy of lane line detection.

[0094] In some embodiments, feature enhancement of image features based on the associated features of each target position to generate enhanced features may include the following operations: determining the weight of each associated feature according to the target position of each associated feature; and feature enhancement of image features based on each weight and each associated feature to generate enhanced features.

[0095] For example, a mapping relationship between target positions and weights may be pre-configured, and the weights of the associated features may be determined based on the mapping relationship according to the target positions of the associated features.

[0096] In some embodiments, the mapping relationship between the target position and the weight can also be adaptively updated.

[0097] For example, during model training, the weight distribution strategy between different positions can be learned for different lane line types.

[0098] For example, we can add the weight coefficient based on formula (5): , to distinguish the degree of interest in the associated features of different target locations. As shown in formula (6):

[0099] (6)

[0100] in, Represents the weight coefficient of the associated feature.

[0101] During lane line detection, the weight distribution ratio of each target position is adaptively updated based on the extracted image features of the image to be detected, further improving the detection accuracy.

[0102] In some embodiments, the enhanced features may be fused based on an interactive attention mechanism, and then the fused features may be detected to obtain the position and type of the lane line.

[0103] In a real-time detection scenario, in order to further reduce the computational overhead of the feature fusion process, the enhanced features are detected and the detection results are generated, which may also include the following operations: pooling the enhanced features in the first direction to obtain a third feature; pooling the enhanced features in the second direction to obtain a fourth feature; fusing the enhanced features, the third features, and the fourth features to generate a fused feature; and detecting the fused feature to generate a detection result.

[0104] In the embodiment of the present disclosure, the first direction and the second direction are perpendicular to each other. For example, the first direction may be a vertical direction, and the second direction may be a horizontal direction.

[0105] like Figure 5As shown in the figure, first, the enhanced features are divided into two paths for pooling. For example, the size of the enhanced features can be expressed as B×C×H×W, where B represents the batch size, C represents the number of channels, H represents the height, and W represents the width.

[0106] Then, the enhanced features are input into the vertical pooling module Pooling12D 506, and a vertical pooling operation is performed on the enhanced features, as shown in formula (7):

[0107] (7)

[0108] Among them, X represents the enhanced features, .

[0109] Then, the channel compression operation is performed, as shown in formula (8):

[0110] (8)

[0111] in, Represents the third feature; Conv 1×1 Indicates a convolution operation with a convolution kernel of 1×1; Indicates the number of intermediate channels.

[0112] In the embodiment of the present disclosure, the enhanced features can be input to the horizontal pooling module Pooling2 2D507 in parallel to perform a pooling operation in the horizontal direction, as shown in formula (9):

[0113] (9)

[0114] Then, the channel compression operation is performed, as shown in formula (10):

[0115] (10)

[0116] in, Indicates the fourth feature.

[0117] In some embodiments, when the enhanced feature, the third feature, and the fourth feature are fused, the enhanced feature, the third feature, and the fourth feature may be concatenated to generate a fused feature, which is then detected to generate the location and type of the lane line.

[0118] For example, you can input the fused features into the regression detection module to generate lane positions, or you can input the fused features into the classification detection module to generate lane types.

[0119] Pooling the enhanced features along the first direction preserves fine-grained lateral lane position information, facilitating lateral lane positioning. Pooling the enhanced features along the second direction preserves longitudinal lane continuity, facilitating longitudinal tracking of curved lanes. By performing pooling operations on the enhanced features in different directions, the interference of background information in the image to be detected, such as guardrail shadows and pavement cracks, on lane detection accuracy can be effectively reduced.

[0120] In order to further capture the long-distance dependency in the vertical direction and the contextual association in the horizontal direction of the lane line deformation features, in the embodiment of the present disclosure, the enhanced features, the third features and the fourth features are fused to generate fused features, which may include the following operations: fusing the third features and the fourth features to generate a processed feature map; analyzing the correlation between the processed features in the processed feature map to generate a spatial attention distribution map; and fusing the enhanced features based on the spatial attention distribution map to generate a fused feature.

[0121] like Figure 5 As shown, first, the third feature and the fourth feature can be input into Matmul 508 to perform a matrix multiplication operation, as shown in formula (11):

[0122] (11)

[0123] in, represents matrix multiplication, Represents the processed feature map.

[0124] Then, the processed feature map can be input into the two-dimensional convolutional network Conv 2D 1×1509 with a convolution kernel of 1×1, and then input into Sigmoid 1×1 510 for attention weight calculation, as shown in formula (12):

[0125] (12)

[0126] in, Represents the sigmoid function; Represents the spatial attention distribution map.

[0127] Then, the spatial attention distribution map and the enhanced features can be input into the two-dimensional convolutional network Conv 2D 1×1 511 with a convolution kernel of 1×1, and element-by-element multiplication operation can be performed to generate fusion features. As shown in formula (13):

[0128] (13)

[0129] in, Represents element-wise (Hadamard) product.

[0130] Generating a spatial attention distribution map based on the cross-attention mechanism is conducive to capturing the long-distance vertical dependencies and horizontal contextual associations of lane line deformation features, thereby highlighting the feature information of key lane line areas in the image to be detected, such as lane starting points and curved turning points, further improving the accuracy of lane line detection.

[0131] In the embodiment of the present disclosure, the lane detection method described above is implemented based on a trained neural network model. Figure 6 The training method of the neural network model for lane line detection is explained in detail.

[0132] Figure 6 The flowchart of the model training method according to an embodiment of the present disclosure is schematically shown.

[0133] like Figure 6 As shown, the model training method 600 may include operations S610 to S630.

[0134] In operation S610, a feature enhancement module is used to generate target sample offset information based on sample image features of a sample image; and based on the target sample offset information, feature enhancement is performed on the sample image features to obtain enhanced sample features.

[0135] In operation S620, the enhanced sample features are detected using a detection module to generate a detection result.

[0136] In operation S630 , the initial model is trained based on the loss value between the detection result and the sample label to generate a model for detecting lane lines.

[0137] According to the embodiments of the present disclosure, the enhanced sample features strengthen the deformation features of the sample lane lines in the sample image, such as the shape and line type of the sample lane lines.

[0138] According to an embodiment of the present disclosure, the target sample offset information indicates the displacement of the sample target position relative to the sample predetermined position for each sample-associated feature during the feature enhancement process; each sample-associated feature is associated with a sample image feature.

[0139] According to an embodiment of the present disclosure, the initial model may include a feature enhancement module and a detection module. The feature enhancement module may include Figure 4 The Stem 410 and FPN 420 shown in FIG. 4 are the same as the network structures described in the lane detection method described above, and are not described in detail here.

[0140] In some embodiments, the detection module can be configured according to actual needs, for example, it can include a regression detection unit for detecting the position of the lane line, and can also include a classification detection unit for detecting the type of the lane line.

[0141] In the embodiment of the present disclosure, the detection result indicates the sample position and sample type of the sample lane line, which can be understood as the sample position and sample type of the sample lane line output by the model.

[0142] In some embodiments, a loss function, such as a cross-entropy loss function, can be selected for model training based on actual needs. The loss function is used to calculate the loss between the detection result and the sample label. The sample label indicates the actual position and type of the sample lane line in the sample image. The actual position and type of the sample lane line can represent the position and type of the annotation for the position and type of the sample lane in the sample image.

[0143] In some embodiments, convergence conditions may be pre-configured, for example, the loss value is less than a predetermined threshold or the number of training times is greater than a predetermined training times threshold.

[0144] For example, when the loss value between the detection result and the sample label does not meet the predetermined convergence condition, the model parameters of the initial model are adjusted and iterative training is performed until the loss value between the detection result and the sample label output by the model meets the predetermined convergence condition.

[0145] In some embodiments, when adjusting the model parameters of the initial model, the parameters of the feature enhancement module and the detection module can be adjusted simultaneously. Alternatively, the parameters of the detection module can be fixed first, and then the parameters of the feature enhancement module can be adjusted. After adjusting the parameters of the feature enhancement module multiple times and the loss value tends to converge, the parameters of the detection module can be fine-tuned until the predetermined convergence condition is met, further improving training efficiency.

[0146] During the model training process, the model can learn how to adaptively adjust the offset information based on the input sample image, which helps the model capture the deformation characteristics of the lane lines in the sample image and further improves the generalization ability of the model.

[0147] Figure 7 A block diagram of a lane detection device according to an embodiment of the present disclosure is schematically shown.

[0148] like Figure 7 As shown, the lane line detection device 700 may include: a first generating module 710 , an obtaining module 720 and a second generating module 730 .

[0149] The first generation module 710 is used to generate target offset information based on the image features of the image to be detected; wherein the offset information indicates the displacement of the target position of each associated feature relative to the predetermined position during the feature enhancement process, and each associated feature is associated with the image feature.

[0150] The acquisition module 720 is used to perform feature enhancement on the image features based on the target offset information to obtain enhanced features; wherein the enhanced features strengthen the deformation features of the lane lines in the image to be detected.

[0151] The second generation module 730 is used to detect the enhanced features and generate a detection result; wherein the detection result indicates the position and type of the lane line.

[0152] According to an embodiment of the present disclosure, the first generation module 710 includes: a first convolution submodule, a second convolution submodule and a first generation submodule.

[0153] The first convolution submodule is used to convolve the image features along a first direction to obtain a first feature.

[0154] The second convolution submodule is used to convolve the first feature along a second direction to obtain a second feature; the first direction and the second direction are perpendicular to each other.

[0155] The first generating submodule is used to generate target offset information based on the second feature.

[0156] According to an embodiment of the present disclosure, the first generation submodule includes: a first convolution unit and a scaling unit.

[0157] The first convolution unit is used to perform convolution processing on the second feature to obtain initial offset information.

[0158] The scaling unit is configured to scale the initial offset information according to a predetermined scaling factor to obtain target offset information.

[0159] According to an embodiment of the present disclosure, the obtaining module 720 includes: a determining submodule and a second generating submodule.

[0160] The determination submodule is configured to determine a target position of each associated feature based on the target offset information and the predetermined position of each associated feature.

[0161] The second generating submodule is used to perform feature enhancement on the image features based on the target position of each associated feature to generate enhanced features.

[0162] According to an embodiment of the present disclosure, the second generating submodule includes: a determining unit and a generating unit.

[0163] The determining unit is used to determine the weight of each associated feature according to the target position of each associated feature.

[0164] The generating unit is used to perform feature enhancement on the image features based on the weights and the associated features to generate enhanced features.

[0165] According to an embodiment of the present disclosure, the second generation module 730 includes: a first pooling submodule, a second pooling submodule, a fusion submodule and a detection submodule.

[0166] The first pooling submodule is used to pool the enhanced features in the first direction to obtain the third features.

[0167] The second pooling submodule is used to pool the enhanced features in a second direction to obtain a fourth feature; wherein the first direction and the second direction are perpendicular to each other.

[0168] The fusion submodule is used to fuse the enhanced features, the third features and the fourth features to generate fused features.

[0169] The detection submodule is used to detect the fused features and generate detection results.

[0170] According to an embodiment of the present disclosure, the fusion submodule includes: a first fusion unit, an analysis unit and a second fusion unit.

[0171] The first fusion unit is used to fuse the third feature and the fourth feature to generate a processed feature map; the scale of the processed feature map is the same as the scale of the image to be detected.

[0172] The analysis unit is used to analyze the correlation between the processed features in the processed feature map to generate a spatial attention distribution map.

[0173] The second fusion unit is used to fuse the enhanced features based on the spatial attention distribution map to generate fused features.

[0174] Figure 8 A block diagram of a model training device according to an embodiment of the present disclosure is schematically shown.

[0175] like Figure 8 As shown, the model training device 800 may include: a feature enhancement module 810, a detection module 820 and a training module 830.

[0176] The feature enhancement module 810 is used to generate target sample offset information based on the sample image features of the sample image; and based on the target sample offset information, perform feature enhancement on the sample image features to obtain enhanced sample features; wherein the enhanced sample features strengthen the deformation features of the sample lane lines in the sample image, and the target sample offset information indicates the displacement of the sample target position relative to the sample predetermined position for each sample-associated feature during the feature enhancement process; each sample-associated feature is associated with the sample image feature.

[0177] A detection module 820 is configured to detect the enhanced sample features and generate a detection result, wherein the detection result indicates the sample location and sample type of the sample lane line;

[0178] The training module 830 is used to train the initial model based on the loss value between the detection results and the sample labels to generate a model for detecting lane lines; wherein the sample labels indicate the actual position and actual type of the sample lane lines in the sample image.

[0179] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0180] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0181] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described above.

[0182] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method described above.

[0183] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0184] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. Computing unit 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to bus 904.

[0185] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0186] The computing unit 901 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the lane detection method or the model training method. For example, in some embodiments, the lane detection method or the model training method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the lane detection method or the model training method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the lane detection method or the model training method in any other appropriate manner (eg, by means of firmware).

[0187] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0188] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0189] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0190] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0191] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0192] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0193] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0194] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A lane line detection method, comprising: generating target offset information based on image features of the image to be detected, wherein the target offset information indicates a displacement of a target position relative to a predetermined position for each associated feature during a feature enhancement process, wherein each associated feature is associated with the image feature; Based on the target offset information, the image features are enhanced to obtain enhanced features; wherein the enhanced features strengthen the deformation features of the lane lines in the image to be detected; and The enhanced features are detected to generate a detection result; wherein the detection result indicates the position and type of the lane line.

2. The method according to claim 1, wherein The generating target offset information based on the image features of the image to be detected includes: Convolving the image feature along a first direction to obtain a first feature; Convolving the first feature along a second direction to obtain a second feature, wherein the first direction and the second direction are perpendicular to each other; and The target offset information is generated based on the second feature.

3. The method according to claim 2, wherein: The generating the target offset information based on the second feature includes: performing convolution processing on the second feature to obtain initial offset information; and The initial offset information is scaled according to a predetermined scaling factor to obtain the target offset information.

4. The method according to claim 1, wherein The step of enhancing the image features based on the target offset information to obtain enhanced features includes: Determining a target position of each associated feature based on the target offset information and the predetermined position of each associated feature; and Based on the target position of each associated feature, the image feature is enhanced to generate the enhanced feature.

5. The method according to claim 4, wherein The step of enhancing the image features based on the target positions of the associated features to generate the enhanced features includes: Determining the weight of each associated feature according to the target position of each associated feature; and Based on the weights and the associated features, feature enhancement is performed on the image features to generate the enhanced features.

6. The method according to claim 1, wherein The detecting the enhanced features and generating a detection result includes: Pooling the enhanced features in the first direction to obtain a third feature; Pooling the enhanced features in a second direction to obtain a fourth feature; wherein the first direction and the second direction are perpendicular to each other; fusing the enhanced feature, the third feature, and the fourth feature to generate a fused feature; and The fusion feature is detected to generate the detection result.

7. The method according to claim 6, wherein: The fusing the enhanced feature, the third feature, and the fourth feature to generate a fused feature includes: fusing the third feature and the fourth feature to generate a processed feature map; the scale of the processed feature map is the same as the scale of the image to be detected; Analyzing the correlation between the processed features in the processed feature map to generate a spatial attention distribution map; and The enhanced features are fused based on the spatial attention distribution map to generate the fused features.

8. A model training method, wherein the initial model includes: A feature enhancement module and a detection module, the method includes: Utilizing the feature enhancement module, target sample offset information is generated based on sample image features of the sample image; and based on the target sample offset information, feature enhancement is performed on the sample image features to obtain enhanced sample features; wherein the enhanced sample features enhance the deformation features of the sample lane lines in the sample image, and the target sample offset information indicates the displacement of the sample target position relative to the predetermined position of the sample for each sample-associated feature during the feature enhancement process; and each sample-associated feature is associated with the sample image feature; Utilizing the detection module, detecting the enhanced sample features to generate a detection result; wherein the detection result indicates the sample position and sample type of the sample lane line; and The initial model is trained based on the loss value between the detection result and the sample label to generate a model for detecting lane lines; wherein the sample label indicates the actual position and actual type of the sample lane line in the sample image.

9. A lane detection device, comprising: a first generating module, configured to generate target offset information based on image features of the image to be detected, wherein the offset information indicates a displacement of a target position relative to a predetermined position for each associated feature during a feature enhancement process, wherein each associated feature is associated with the image feature; an acquisition module, configured to perform feature enhancement on the image feature based on the target offset information to obtain an enhanced feature; wherein the enhanced feature strengthens the deformation feature of the lane line in the image to be detected; and The second generation module is used to detect the enhanced features and generate a detection result; wherein the detection result indicates the position and type of the lane line.

10. A model training device, wherein the initial model comprises: Feature enhancement module and detection module, the device includes: The feature enhancement module is configured to generate target sample offset information based on sample image features of the sample image; and based on the target sample offset information, perform feature enhancement on the sample image features to obtain enhanced sample features; wherein the enhanced sample features enhance the deformation features of the sample lane lines in the sample image, and the target sample offset information indicates the displacement of the sample target position relative to the sample predetermined position for each sample-associated feature during the feature enhancement process; and each sample-associated feature is associated with the sample image feature; The detection module is configured to detect the enhanced sample features and generate a detection result; wherein the detection result indicates the sample position and sample type of the sample lane line; and A training module is used to train the initial model based on the loss value between the detection results and the sample labels to generate a model for detecting lane lines; wherein the sample labels indicate the actual position and actual type of the sample lane lines in the sample image.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images