Image processing method, lane line detection method and related equipment

Abstract

An embodiment of the present application discloses an image processing method, which can be applied to scenarios including lane line detection, such as adaptive cruise control, lane departure warning, and lane keeping assist. The method includes: extracting features from the image to be detected to obtain a first feature; processing the detection frame information of the image to be detected to obtain a second feature, the detection frame information including the position of the detection frame of at least one object in the image to be detected in the image to be detected; inputting the first feature and the second feature into a first neural network based on a transformer structure to obtain the lane line in the image to be detected. By applying a neural network with a transformer structure to the lane line detection task, the global information of the image to be detected can be obtained, thereby effectively modeling the long-range connection between lane lines. And by adding detection frame information, the perception ability of the image scene is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to an image processing method, a lane line detection method, and related equipment. Background Art

[0002] Intelligent driving technologies (such as autonomous driving and assisted driving) rely on the collaborative efforts of artificial intelligence, visual computing, radar, monitoring devices, and global positioning systems to enable vehicles to operate autonomously without the need for active human intervention. Lane detection is one of the most important technologies in intelligent driving and is crucial for other technologies used in intelligent driving systems, such as adaptive cruise control, lane departure warning, and road condition understanding. Lane detection technology aims to predict each lane line in an image captured by a camera, thereby assisting the vehicle in staying in the correct lane.

[0003] With the development of deep learning technology, lane line detection based on image segmentation has begun to emerge. The lane line detection model based on image segmentation first predicts the segmentation result of the entire image, and then outputs the lane line detection result after clustering.

[0004] However, most lane detection methods based on deep learning technology are based on convolutional neural networks, such as spatial convolutional neural networks (SCNNs). Due to the limitations of the receptive field of convolutional neural networks, they cannot well perceive the global information of the image and thus cannot accurately predict the position of lane lines. Especially in scenes with many vehicles blocking the view, the model is prone to misdetection. Summary of the Invention

[0005] The embodiments of the present application provide an image processing method, a lane line detection method, and related devices, which can improve the accuracy of lane line detection in an image.

[0006] The first aspect of the embodiment of the present application provides an image processing method, which can be applied to intelligent driving scenarios. For example: adaptive cruise control, lane departure warning (LDW), lane keeping assist (LKA), etc., which include scenarios of lane line detection. The method can be executed by an image processing device (such as a terminal device or a server), or by a component of an image processing device (such as a processor, a chip, or a chip system, etc.). The method is implemented by a target neural network containing a transformer structure, and the method includes: extracting features from the image to be detected to obtain a first feature; processing the detection frame information of the image to be detected to obtain a second feature, and the detection frame information includes the position of the detection frame of the object in the image to be detected in the image to be detected; inputting the first feature and the second feature into a first neural network based on a transformer structure to obtain the lane line in the image to be detected.

[0007] In the embodiments of this application, by applying the transformer architecture to lane detection, global information about the image to be detected can be acquired, effectively modeling the long-range connections between lane lines. Furthermore, by adding detection box information about objects in the image during lane detection, the perception of the image scene can be improved, reducing misjudgments in scenarios where lane lines are obscured by vehicles.

[0008] Optionally, in a possible implementation of the first aspect, the above-mentioned step of processing the detection frame information of the image to be detected to obtain the second feature includes processing at least one third feature and the detection frame information to obtain the second feature, and the at least one third feature is an intermediate feature obtained in the process of obtaining the first feature.

[0009] In this possible implementation, the acquired second feature contains not only the detection frame information but also the image features, providing more details for subsequent lane line determination.

[0010] Optionally, in a possible implementation of the first aspect, the above-mentioned second feature includes the position feature and semantic feature of the detection frame corresponding to the object in the image to be detected, and the detection frame information also includes: the category and confidence of the detection frame; processing at least one third feature and the detection frame information to obtain the second feature includes: obtaining the semantic feature based on at least one third feature, position and confidence; obtaining the position feature based on the position and category.

[0011] In this possible implementation, the second feature not only considers the position of the detection frame, but also the category and confidence of the detection frame, making the subsequently determined lane line more accurate.

[0012] Optionally, in a possible implementation of the first aspect, the above steps: obtaining semantic features based on at least one third feature, position and confidence, including: extracting region of interest ROI features from at least one third feature based on position; multiplying the ROI features and the confidence, and inputting the obtained features into a fully connected layer to obtain semantic features; obtaining position features based on position and category, including: obtaining a vector of the category, and splicing it with the vector corresponding to the position, and inputting the obtained features into a fully connected layer to obtain position features.

[0013] In this possible implementation, by determining the semantic features related to the detection frame in the image features and introducing position features containing the detection frame position information, the information contained in the second feature is made more comprehensive, thereby improving the accuracy of lane line prediction.

[0014] Optionally, in a possible implementation of the first aspect, the first neural network based on the transformer structure includes an encoder, a decoder and a feedforward neural network; the first feature and the second feature are input into the first neural network based on the transformer structure to obtain the lane line in the image to be detected, including: obtaining a fourth feature based on the first feature, the second feature and the encoder; inputting the fourth feature, the second feature and the query feature into the decoder to obtain a fifth feature; and inputting the fifth feature into the feedforward neural network to obtain multiple point sets.

[0015] In this possible implementation, applying the transformer architecture to lane detection can, on the one hand, capture global information about the image being detected, effectively modeling the long-range connections between lane lines. Furthermore, by adding a second feature containing detection bounding box information during point set determination, subsequent lane line determination based on the point set becomes more accurate.

[0016] Optionally, in a possible implementation of the first aspect, the above steps also include: obtaining a first row feature and a first column feature based on the first feature, the first row feature being obtained by flattening the matrix corresponding to the first feature along the row direction, and the first column feature being obtained by flattening the matrix along the column direction; inputting the first feature and the second feature into the encoder to obtain a fourth feature, including: inputting the first feature, the second feature, the first row feature and the first column feature into the decoder to obtain the fourth feature.

[0017] In this possible implementation, by introducing the first row features and the first column features that can adapt to the lane line shape to mine contextual information, the ability to construct long lane line features can be improved, thereby achieving better lane line detection results.

[0018] Optionally, in a possible implementation of the first aspect, the above steps of: inputting the first feature, the second feature, the first row feature, and the first column feature into the encoder to obtain the fourth feature include: performing self-attention calculation on the first feature to obtain a first output; performing cross-attention calculation on the first feature and the second feature to obtain a second output; performing self-attention calculation and splicing processing on the first row feature and the first column feature to obtain row and column outputs; and obtaining the fourth feature based on the first output, the second output, and the row and column outputs.

[0019] In this possible implementation, row and column outputs are also considered in the process of acquiring the fourth feature. By introducing row and column outputs that can adapt to the lane line shape and mine contextual information, the ability to construct long lane line features can be improved, thereby achieving better lane line detection effects.

[0020] Optionally, in a possible implementation of the first aspect, the above step of obtaining the fourth feature based on the first output, the second output, and the row and column outputs includes: adding the first output and the second output to obtain the fifth output; and concatenating the fifth output and the row and column outputs to obtain the fourth feature.

[0021] This possible implementation refines the process of the fourth feature, which is obtained by concatenating the row and column outputs with the sum of the first and second outputs. By introducing row and column outputs that can adapt to lane shape and extract contextual information, the ability to construct features for long lane lines can be improved, thereby achieving better lane detection results.

[0022] Optionally, in a possible implementation of the first aspect, the above steps of: inputting the first feature and the second feature into the encoder to obtain the fourth feature, include: performing self-attention calculation on the first feature to obtain a first output; performing cross-attention calculation on the first feature and the second feature to obtain a second output; and adding the first output and the second output to obtain the fourth feature.

[0023] In this possible implementation, the fourth feature not only contains the first output calculated based on the first feature through the self-attention mechanism, but also contains the second output calculated based on the cross-attention of the first and second features, thereby improving the expressive power of the fourth feature.

[0024] Optionally, in a possible implementation of the first aspect, the above step of: inputting the fourth feature, the second feature and the query feature into the decoder to obtain the fifth feature includes: performing cross-attention calculation on the query feature and the fourth feature to obtain a third output; processing the query feature and the second feature to obtain a fourth output; and adding the third output and the fourth output to obtain the fifth feature.

[0025] In this possible implementation, the fifth feature obtained through cross-attention calculation takes into account more information with predicted images, improves the expressive power of the fifth feature, and makes the subsequent lane lines determined based on the point set more accurate.

[0026] Optionally, in a possible implementation of the first aspect, the above-mentioned step of: extracting features from the image to be detected to obtain the first feature includes: performing feature fusion and dimensionality reduction processing on the features output by different layers in the backbone network to obtain the first feature, and the input of the backbone network is the image to be detected.

[0027] In this possible implementation, features from different layers are combined. Due to the varying performance of features extracted by different neural network layers, lower-layer features have higher resolution and contain more location and detail information. However, due to fewer convolutions, they are less semantic and contain more noise. Higher-layer features have stronger semantic information but lower resolution and poorer perception of detail. Therefore, by fusing features extracted from different neural network layers, the resulting first feature has multi-layered features.

[0028] The second aspect of the embodiment of the present application provides a lane line detection method, which can be applied to intelligent driving scenarios. For example: adaptive cruise control, lane departure warning, lane keeping assist and other scenarios that include lane line detection. The method can be executed by a detection device (such as a vehicle or a device in a vehicle), or by a component of the detection device (such as a processor, a chip, or a chip system, etc.). The method includes: acquiring an image to be detected; processing the image to be detected to obtain multiple point sets, each point set in the multiple point sets represents a lane line in the image to be detected; wherein, processing a first neural network based on a transformer structure and detection frame information to predict the point set of the lane line in the image, and the detection frame information includes the position of the detection frame of at least one object in the image to be detected in the image to be detected.

[0029] In the embodiments of this application, by applying the transformer architecture to lane detection, global information about the image to be detected can be acquired, effectively modeling the long-range connections between lane lines. Furthermore, by adding detection box information about objects in the image during lane detection, the target neural network's perception of the image scene can be enhanced, reducing misjudgments in scenarios where lane lines are obscured by vehicles.

[0030] Optionally, in a possible implementation manner of the second aspect, the above-mentioned detection box information further includes: a category and a confidence level of the detection box.

[0031] In this possible implementation, by introducing the category and confidence of the detection box, the detection box information for subsequent predicted lane line reference can be increased, making the subsequent lane line determined based on the point set more accurate.

[0032] Optionally, in a possible implementation manner of the second aspect, the above steps further include: displaying lane lines.

[0033] In this possible implementation, by displaying lane lines, users can pay attention to the lane line conditions of the current road, especially in scenarios where lane lines are obscured, helping users accurately determine lane lines and reduce the risks caused by blurred lane lines.

[0034] Optionally, in a possible implementation manner of the second aspect, the above steps further include: modeling at least one object to obtain a virtual object; fusing multiple point sets with the virtual object based on position to obtain a target image; and displaying the target image.

[0035] In this possible implementation, a target image is generated by modeling a virtual object and fusing it with multiple point sets based on its position. This allows the user to understand surrounding objects and lane lines, helping them accurately identify surrounding objects and lane lines, reducing the risk of blurred lane lines.

[0036] The third aspect of the embodiment of the present application provides an image processing method, which can be applied to intelligent driving scenarios. For example: adaptive cruise control, lane departure warning, lane keeping assist, and other scenarios that include lane line detection. The method can be executed by an image processing device (such as a terminal device or a server), or by a component of an image processing device (such as a processor, a chip, or a chip system, etc.). The method includes: obtaining a training image; inputting the training image into a target neural network to obtain a first point set of the training image, the first point set representing the predicted lane line in the training image; the target neural network is used to: extract features from the training image to obtain a first feature; process the detection frame information of the training image to obtain a second feature, the detection frame information includes the position of the detection frame of the object in the training image in the training image; based on the first feature and the second feature, a first point set is obtained, and the target neural network is used to predict the point set of the lane line in the image based on the transformer structure; the target neural network is trained according to the first point set and the real point set of the actual lane line in the training image to obtain a trained target neural network.

[0037] In the embodiments of this application, by applying the transformer architecture to lane detection, global information about the image to be detected can be acquired, effectively modeling the long-range connections between lane lines. Furthermore, by adding detection box information about objects in the image during lane detection, the target neural network's perception of the image scene can be enhanced, reducing misjudgments in scenarios where lane lines are obscured by vehicles.

[0038] The fourth aspect of the embodiment of the present application provides an image processing device, which can be applied to intelligent driving scenarios. For example: scenarios including lane line detection, such as adaptive cruise control, lane departure warning, and lane keeping assist. The image processing device includes: an extraction unit for extracting features from the image to be detected to obtain a first feature; a processing unit for processing the detection frame information of the image to be detected to obtain a second feature, and the detection frame information includes the position of the detection frame of at least one object in the image to be detected in the image to be detected; a determination unit for inputting the first feature and the second feature into a first neural network based on a transformer structure to obtain the lane line in the image to be detected.

[0039] Optionally, in a possible implementation of the fourth aspect, the above-mentioned processing unit is specifically used to process at least one third feature and detection frame information to obtain a second feature, and at least one third feature is an intermediate feature obtained in the process of obtaining the first feature.

[0040] Optionally, in a possible implementation of the fourth aspect, the above-mentioned second feature includes the position feature and semantic feature of the detection frame corresponding to the object in the image to be detected, and the detection frame information also includes: the category and confidence of the detection frame; a processing unit, specifically used to obtain the semantic feature based on at least one third feature, position and confidence; a processing unit, specifically used to obtain the position feature based on the position and category.

[0041] Optionally, in a possible implementation of the fourth aspect, the above-mentioned processing unit is specifically used to extract the region of interest ROI feature from at least one third feature based on the position; the processing unit is specifically used to multiply the ROI feature with the confidence, and input the obtained feature into the fully connected layer to obtain the semantic feature; the processing unit is specifically used to obtain the category vector, and splice it with the vector corresponding to the position, and input the obtained feature into the fully connected layer to obtain the position feature.

[0042] Optionally, in a possible implementation of the fourth aspect, the above-mentioned first neural network based on the transformer structure includes an encoder, a decoder and a feedforward neural network; the determination unit is specifically used to input the first feature and the second feature into the encoder to obtain a fourth feature; the determination unit is specifically used to input the fourth feature, the second feature and the query feature into the decoder to obtain a fifth feature; the determination unit is specifically used to input the fifth feature into the feedforward neural network to obtain multiple point sets, each of the multiple point sets represents a lane line in the image to be detected.

[0043] Optionally, in a possible implementation of the fourth aspect, the above-mentioned image processing device also includes: an acquisition unit, used to obtain a first row feature and a first column feature based on the first feature, the first row feature is obtained by flattening the matrix corresponding to the first feature along the row direction, and the first column feature is obtained by flattening the matrix along the column direction; a determination unit, specifically used to input the first feature, the second feature, the first row feature and the first column feature into a decoder to obtain a fourth feature.

[0044] Optionally, in a possible implementation of the fourth aspect, the above-mentioned determination unit is specifically used to perform self-attention calculation on the first feature to obtain a first output; the determination unit is specifically used to perform cross-attention calculation on the first feature and the second feature to obtain a second output; the determination unit is specifically used to perform self-attention calculation and splicing processing on the first row feature and the first column feature to obtain row and column outputs; the determination unit is specifically used to obtain the fourth feature based on the first output, the second output and the row and column outputs.

[0045] Optionally, in a possible implementation of the fourth aspect, the above-mentioned determination unit is specifically used to add the first output and the second output to obtain the fifth output; the determination unit is specifically used to splice the fifth output with the row and column outputs to obtain the fourth feature.

[0046] Optionally, in a possible implementation of the fourth aspect, the above-mentioned determination unit is specifically used to perform self-attention calculation on the first feature to obtain a first output; the determination unit is specifically used to perform cross-attention calculation on the first feature and the second feature to obtain a second output; the determination unit is specifically used to add the first output and the second output to obtain a fourth feature.

[0047] Optionally, in a possible implementation of the fourth aspect, the above-mentioned determination unit is specifically used to perform cross-attention calculation on the query feature and the fourth feature to obtain a third output; the determination unit is specifically used to process the query feature and the second feature to obtain a fourth output; the determination unit is specifically used to add the third output and the fourth output to obtain a fifth feature.

[0048] Optionally, in a possible implementation of the fourth aspect, the above-mentioned extraction unit is specifically used to perform feature fusion and dimensionality reduction processing on the features output by different layers in the backbone network to obtain the first feature, and the input of the backbone network is the image to be detected.

[0049] The fifth aspect of the embodiment of the present application provides a detection device, which can be applied to intelligent driving scenarios. For example: scenarios including lane line detection such as adaptive cruise control, lane departure warning, and lane keeping assist. The detection device is applied to a vehicle, and the detection device includes: an acquisition unit for acquiring an image to be detected; a processing unit for processing the image to be detected to obtain multiple point sets, each of the multiple point sets represents a lane line in the image to be detected; wherein the point set of the lane line in the image is predicted by the first neural network based on the transformer structure and the detection frame information, and the detection frame information includes the position of the detection frame of at least one object in the image to be detected in the image to be detected.

[0050] Optionally, in a possible implementation manner of the fifth aspect, the above-mentioned detection box information also includes: a category and confidence level of the detection box.

[0051] Optionally, in a possible implementation of the fifth aspect, the above-mentioned detection device further includes: a display unit, configured to display lane lines.

[0052] Optionally, in a possible implementation of the fifth aspect, the above-mentioned processing unit is also used to model at least one object to obtain a virtual object; the processing unit is also used to fuse multiple point sets with the virtual object based on position to obtain a target image; and the display unit is also used to display the target image.

[0053] The sixth aspect of the embodiment of the present application provides an image processing device, which can be applied to intelligent driving scenarios. For example: adaptive cruise control, lane departure warning, lane keeping assist and other scenarios that include lane line detection. The image processing device includes: an acquisition unit for acquiring a training image; a processing unit for inputting the training image into a target neural network to obtain a first point set of the training image, the first point set representing the predicted lane line in the training image; the target neural network is used to: extract features from the training image to obtain a first feature; process the detection frame information of the training image to obtain a second feature, the detection frame information includes the position of the detection frame of the object in the training image in the training image; based on the first feature and the second feature, a first point set is obtained, and the target neural network is used to predict the point set of the lane line in the image based on the transformer structure; a training unit is used to train the target neural network according to the first point set and the real point set of the actual lane line in the training image to obtain a trained target neural network.

[0054] In a seventh aspect, the present application provides an image processing device, comprising: a processor, the processor being coupled to a memory, the memory being used to store programs or instructions, and when the programs or instructions are executed by the processor, the image processing device implements the method of the aforementioned first aspect or any possible implementation of the first aspect, or implements the method of the aforementioned third aspect or any possible implementation of the third aspect.

[0055] In an eighth aspect, the present application provides a detection device, comprising: a processor, the processor being coupled to a memory, the memory being used to store programs or instructions, and when the programs or instructions are executed by the processor, the detection device implements the method in the above-mentioned second aspect or any possible implementation of the second aspect.

[0056] In a ninth aspect, the present application provides a computer-readable medium having a computer program or instruction stored thereon. When the computer program or instruction runs on a computer, the computer is caused to execute the method in the aforementioned first aspect or any possible implementation of the first aspect, or the computer is caused to execute the method in the aforementioned second aspect or any possible implementation of the second aspect, or the computer is caused to execute the method in the aforementioned third aspect or any possible implementation of the third aspect.

[0057] In the tenth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method in the aforementioned first aspect or any possible implementation of the first aspect, or enables the computer to execute the method in the aforementioned second aspect or any possible implementation of the second aspect, or enables the computer to execute the method in the aforementioned third aspect or any possible implementation of the third aspect.

[0058] Among them, the technical effects brought about by the fourth, seventh, eighth, ninth, and tenth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here.

[0059] Among them, the technical effects brought about by the fifth, seventh, eighth, ninth, and tenth aspects or any possible implementation methods thereof can be referred to the technical effects brought about by the second aspect or different possible implementation methods of the second aspect, and will not be repeated here.

[0060] Among them, the technical effects brought about by the sixth, seventh, eighth, ninth, and tenth aspects or any possible implementation methods thereof can be referred to the technical effects brought about by the third aspect or different possible implementation methods of the third aspect, and will not be repeated here.

[0061] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: First, by applying the transformer structure to the lane detection task, global information of the image to be detected can be obtained, thereby effectively modeling the long-range connections between lane lines. Second, by adding detection frame information of objects in the image during lane detection, the perception ability of the image scene can be improved, reducing misjudgments in scenarios where lane lines are obscured by vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of the system architecture provided in an embodiment of the present application;

[0063] Figure 2 A schematic diagram of a chip hardware structure provided in an embodiment of the present application;

[0064] Figure 3a A schematic diagram of the structure of the image processing system provided in an embodiment of the present application;

[0065] Figure 3b Another structural diagram of the image processing system provided in an embodiment of the present application;

[0066] Figure 4 A schematic structural diagram of a vehicle provided in an embodiment of the present application;

[0067] Figure 5 A flowchart of an image processing method provided in an embodiment of the present application;

[0068] Figure 6 A schematic diagram of a process for obtaining the second feature in an embodiment of the present application;

[0069] Figure 7 A schematic diagram of the structure of a first neural network provided in an embodiment of the present application;

[0070] Figure 8 A schematic diagram of the transformer structure provided in an embodiment of the present application;

[0071] Figure 9 A schematic diagram of a process for obtaining the fourth feature in an embodiment of the present application;

[0072] Figure 10 A schematic diagram of a process for obtaining a fourth output in an embodiment of the present application;

[0073] Figure 11 Another structural diagram of the first neural network provided in an embodiment of the present application;

[0074] Figure 12Another structural diagram of the transformer structure provided in an embodiment of the present application;

[0075] Figure 13 A schematic diagram of the structure of the row-column attention module provided in an embodiment of the present application;

[0076] Figure 14a An example diagram of a process for determining multiple point sets provided in an embodiment of the present application;

[0077] Figure 14b An example diagram of multiple point sets provided in an embodiment of the present application;

[0078] Figure 14c An example diagram of an image to be detected including multiple point sets provided in an embodiment of the present application;

[0079] Figure 14d An example diagram corresponding to lane line detection provided in an embodiment of the present application;

[0080] Figure 15 Another flowchart of the image processing method provided in an embodiment of the present application;

[0081] Figure 16 A schematic diagram of the structure of the target neural network provided in an embodiment of the present application;

[0082] Figure 17 Another schematic diagram of the structure of the target neural network provided in an embodiment of the present application;

[0083] Figure 18 A schematic flow chart of a lane detection method provided in an embodiment of the present application;

[0084] Figure 19 An example diagram of a target image provided in an embodiment of the present application;

[0085] Figure 20 A flowchart of the model training method provided in an embodiment of the present application;

[0086] Figure 21 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0087] Figure 22 A schematic diagram of the structure of the detection device provided in an embodiment of the present application;

[0088] Figure 23 Another structural diagram of the image processing device provided in an embodiment of the present application;

[0089] Figure 24 Another structural diagram of the image processing device provided in an embodiment of the present application;

[0090] Figure 25 Another structural schematic diagram of the detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] The embodiments of the present application provide an image processing method, a lane line detection method, and related devices, which can improve the accuracy of lane line detection in an image.

[0092] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0093] The first step in intelligent driving is collecting and processing environmental information. Lane markings, as one of the most important indicators of the road surface, effectively guide intelligent vehicles within restricted road areas. Therefore, accurately detecting lane markings in real time is a crucial step in the design of intelligent vehicle systems. This assists with route planning, provides lane deviation warnings, and provides a reference for precise navigation. Lane detection technology aims to accurately identify lane markings on the road by analyzing images captured by on-board cameras during driving, thereby assisting vehicles in staying in the correct lane.

[0094] With the development of deep learning technology, lane detection based on image segmentation and detection has begun to emerge. The image segmentation-based lane detection model first predicts the segmentation results of the entire image and then outputs the lane detection results through clustering. Detection-based lane detection, on the other hand, generates multiple anchor points and predicts the lane line offsets relative to these anchor points to predict a large number of candidate lane lines. It then performs post-processing using non-maximum suppression to obtain the final lane detection results.

[0095] Most lane detection methods based on deep learning are based on convolutional neural networks, such as spatial convolutional neural networks (SCNNs). SCNNs are a lane detection solution based on image segmentation. This solution uses a convolutional neural network to segment the image to be detected, predicting a class for each pixel. This solution generalizes the traditional deep convolutional architecture into a slice-by-slice convolutional architecture, performing convolutions in different directions to transfer information between rows and columns within the image. Specifically, while traditional convolutions operate on a feature of dimension HxWxC, this solution first divides the HxWxC feature into H slices of WxC vertically, then convolves these slices from bottom to top and top to bottom. The HxWxC feature is then divided horizontally into W slices of HxC horizontally, and then convolved from left to right and right to left. Finally, the convolution results obtained in these four directions are concatenated and passed through a fully connected layer to output a segmentation map of the image. This achieves lane detection.

[0096] However, due to the limited receptive field of convolutional neural networks, they cannot effectively perceive the global information of an image. This makes them difficult to predict objects with long tails (e.g., long, narrow shapes) such as lane markings. Furthermore, in scenes with numerous vehicles obstructing the view, the model cannot accurately predict the position of lane markings, making it prone to misjudgment.

[0097] In order to solve the above technical problems, the embodiments of the present application provide an image processing method, a lane line detection method and related equipment. On the one hand, by applying the transformer structure to the lane line detection task, the long-range connection between lane lines can be effectively modeled. On the other hand, by adding the detection frame position information of the object in the image during the lane line detection process, the perception ability of the scene can be improved. Reduce misjudgment in scenes where the lane line is blocked by the vehicle. The image processing method and related equipment of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0098] To facilitate understanding, the following first introduces the relevant terms and concepts mainly involved in the embodiments of this application.

[0099] 1. Neural Networks

[0100] A neural network can be composed of neural units, which can be represented by X s The output of the operation unit with the intercept 1 as input can be:

[0101]

[0102] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For X s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a Relu function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0103] The operation of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the operation of each layer in a neural network can be understood as transforming the input space (a set of input vectors) into the output space (i.e., from the row space to the column space of a matrix) through five operations. These operations include: 1. Dimensionality increase / decrease; 2. Scaling / reduction; 3. Rotation; 4. Translation; and 5. "Bending." Operations 1, 2, and 3 are performed by Wx, 4 by +b, and 5 by a(). The word "space" is used here because the objects being classified are not individual things, but rather a class of things, and space refers to the collection of all individuals within that class. W is the weight vector, each value in which represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from input space to output space described above. That is, the weights W of each layer control how the space is transformed. The goal of training a neural network is to ultimately obtain the weight matrix for all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control spatial transformations, and more specifically, about learning the weight matrix.

[0104] 2. Convolutional Neural Networks

[0105] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. A CNN consists of a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be thought of as convolving the same trainable filter with an input image or feature map. A convolutional layer is the layer of neurons in a CNN that performs convolution on the input signal. Within a convolutional layer, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each composed of a rectangular arrangement of neurons. Neurons within the same feature plane share weights, which are referred to as the convolution kernel. Shared weights can be understood as the way image information is extracted, regardless of position. The underlying principle is that the statistical information of one part of the image is the same as that of another. This means that image information learned in one part can also be applied to other parts. Therefore, the same learned image information can be used at all positions in the image. In the same convolutional layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0106] Convolution kernels can be initialized as matrices of random size, and during the training process of the convolutional neural network, the convolution kernels can be learned to obtain reasonable weights. In addition, the direct benefit of shared weights is that they reduce the number of connections between the layers of the convolutional neural network, while also reducing the risk of overfitting.

[0107] 3. Transformer

[0108] The transformer structure is a feature extraction network that includes an encoder and a decoder (similar to a convolutional neural network).

[0109] Encoder: Performs feature learning in the global receptive field through self-attention, such as pixel features.

[0110] Decoder: Learns the features of the required modules, such as the features of the output box, through self-attention and cross-attention.

[0111] The following describes attention (also called attention mechanism):

[0112] The attention mechanism can quickly extract important features from sparse data. The attention mechanism occurs between the encoder and decoder, or rather, between the input sentence and the generated sentence. In contrast, the self-attention mechanism in the self-attention model occurs within the input sequence or the output sequence, extracting connections between distant words within the same sentence, such as syntactic features (phrase structure). The self-attention mechanism, through QKV, provides an effective modeling approach for capturing global contextual information. Assume the input is Q (query), and the context is stored in the form of key-value pairs (K, V). The attention mechanism is essentially a mapping function from the query to a series of key-value pairs (key, value). The attention function can be described as mapping a query to a series of (key-value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in the form of (K, V), attention performs addressing by calculating the similarity between Q and K. The calculated similarity between Q and K reflects the importance of the extracted V value, i.e., the weight. The weighted summation then yields the final feature value.

[0113] The calculation of attention is mainly divided into three steps. The first step is to calculate the similarity between the query and each key to obtain the weight. Common similarity functions include dot product, concatenation, perceptron, etc. Then the second step is generally to use a softmax function (on the one hand, it can be normalized to obtain a probability distribution where the sum of all weight coefficients is 1. On the other hand, the characteristics of the softmax function can be used to highlight the weights of important elements) to normalize these weights. Finally, the weights and the corresponding key values ​​are weighted and summed to obtain the final eigenvalue. The specific calculation formula can be as follows:

[0114]

[0115] Where d is the dimension of the QK matrix.

[0116] Attention includes self-attention and cross-attention. Self-attention can be considered a special type of attention, where the QKV input is consistent. Cross-attention, on the other hand, uses inconsistent QKV input. Attention uses the similarity between features (e.g., inner product) as weights to integrate the query features as the updated value of the current feature. Self-attention extracts attention based on the attention of the feature map itself.

[0117] For convolution, the kernel setting limits the size of the receptive field, often requiring multiple layers of stacking to focus on the entire feature map. The advantage of self-attention is its global focus, enabling it to capture the global spatial information of the feature map through simple queries and assignments. A special feature of self-attention in the query key value (QKV) model is that the inputs to the QKV are consistent. The QKV model will be described later.

[0118] 4. Feedforward Neural Network

[0119] A feedforward neural network (FNN) is one of the earliest simple artificial neural networks invented. In a FNN, neurons belong to different layers. Neurons in each layer receive signals from neurons in the previous layer and generate output signals to the next layer. Layer 0 is called the input layer, the last layer is called the output layer, and the remaining intermediate layers are called hidden layers. There is no feedback in the entire network; signals propagate in a single direction, from the input layer to the output layer.

[0120] 5. Multilayer perceptron (MLP)

[0121] A multilayer perceptron, also known as a multilayer perceptron, is a feedforward artificial neural network model that maps inputs to a single output.

[0122] 6. Loss Function

[0123] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value, and this adjustment is continued until the neural network can predict the desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value." This is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0124] 7. Feature Fusion

[0125] Different features extracted by the neural network are used to generate new features through some method, so that the new features are more effective for classification, recognition or detection. There are generally two ways of feature fusion: concat and add. Among them, concat is a serial feature fusion method, that is, directly connecting two features. If the dimensions of the two input features x and y are p and q, the dimension of the output feature z is p+q; add is a parallel fusion strategy, which is to combine two feature vectors. For the input features x and y, a new feature z with the same number of channels is obtained. In other words, add increases the amount of information under the features describing the image, but the dimension describing the image itself does not increase, only the amount of information under each dimension increases; while concat is a merging of the number of channels, that is, the features describing the image itself increase, while the information under each feature does not increase.

[0126] 8. Dimensionality reduction

[0127] Dimensionality reduction is the process of converting high-dimensional data into low-dimensional data. In this embodiment, dimensionality reduction primarily targets the feature matrix. Specifically, the feature matrix can be reduced in dimensionality through a linear transformation layer. Dimensionality reduction of the feature matrix can also be understood as reducing the dimensionality of the vector space corresponding to the feature matrix.

[0128] 9. Area of ​​interest.

[0129] Region of interest (ROI): In machine vision and image processing, the area to be processed is outlined in the image being processed using a box, circle, ellipse, irregular polygon, etc.

[0130] The following describes the system architecture provided by the embodiments of the present application.

[0131] See attached Figure 1, an embodiment of the present invention provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to collect training data. In the embodiment of the present application, the training data includes: training images. Optionally, the training data may also include the first feature of the training image and the detection box information corresponding to the object in the training image. The training data is stored in the database 130, and the training device 120 obtains the target model / rule 101 based on the training data maintained in the database 130. The following will describe in more detail how the training device 120 obtains the target model / rule 101 based on the training data. The target model / rule 101 can be used to implement the image processing method provided by the embodiment of the present application. The target model / rule 101 in the embodiment of the present application can specifically be a target neural network. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily come from the collection of the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 entirely based on the training data maintained by the database 130. It is also possible to obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0132] The target model / rule 101 obtained by training the training device 120 can be applied to different systems or devices, such as Figure 1 The execution device 110 shown in the figure can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) device / virtual reality (VR) device, a vehicle terminal, etc. Of course, the execution device 110 can also be a server or a cloud. Figure 1 In the embodiment, the execution device 110 is configured with an I / O interface 112 for exchanging data with external devices. A user may input data into the I / O interface 112 via a client device 140. In this embodiment, the input data may include an image to be detected. Furthermore, the input data may be user-entered, uploaded by the user via a camera, or sourced from a database, though this is not specifically limited here.

[0133] The preprocessing module 113 is used to preprocess the input data received by the I / O interface 112. In the embodiment of the present application, the preprocessing module 113 can be used to obtain features of the image to be detected. Optionally, the preprocessing module 113 can also be used to obtain detection box information corresponding to the object in the image to be detected.

[0134] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0135] Finally, the I / O interface 112 returns the processing result, such as the point set obtained above or an image including the point set, to the client device 140 , thereby providing it to the user.

[0136] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0137] In the attached Figure 1 In the case shown in FIG, the user can manually input data, which can be operated through the interface provided by I / O interface 112. In another case, client device 140 can automatically send input data to I / O interface 112. If the automatic transmission of input data by client device 140 requires user authorization, the user can set the corresponding permissions in client device 140. The user can view the results output by execution device 110 on client device 140, which can be presented in the form of display, sound, action, etc. Client device 140 can also serve as a data acquisition terminal, collecting input data input into I / O interface 112 and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, the collection can also be performed without client device 140, and instead the input data input into I / O interface 112 and output results from I / O interface 112 as new sample data can be directly stored in database 130 by I / O interface 112.

[0138] It is worth noting that the Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in the attached Figure 1 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0139] like Figure 1 As shown, the target model / rule 101 is obtained through training by the training device 120. The target model / rule 101 in the embodiment of the present application can specifically be a target neural network.

[0140] The following describes a chip hardware structure provided by an embodiment of the present application.

[0141] Figure 2 The embodiment of the present invention provides a chip hardware structure, which includes a neural network processor 20. The chip can be set as follows Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101.

[0142] The neural network processor 20 can be a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU), any processor suitable for large-scale XOR operation processing. Taking the NPU as an example: the neural network processor 20 is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU is the operation circuit 203, and the controller 204 controls the operation circuit 203 to extract data from the memory (weight memory or input memory) and perform operations.

[0143] In some implementations, arithmetic circuit 203 includes multiple processing engines (PEs). In some implementations, arithmetic circuit 203 is a two-dimensional systolic array. Arithmetic circuit 203 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 203 is a general-purpose matrix processor.

[0144] For example, assume there are input matrix A, weight matrix B, and output matrix C. Operation circuit 203 retrieves the corresponding data of matrix B from weight memory 202 and caches it on each PE in the operation circuit. The operation circuit retrieves the data of matrix A from input memory 201 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 208.

[0145] The vector calculation unit 207 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 207 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0146] In some implementations, the vector calculation unit 207 can store the processed output vector to the unified buffer 206. For example, the vector calculation unit 207 can apply a nonlinear function to the output of the operation circuit 203, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 207 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 203, for example, for use in a subsequent layer in a neural network.

[0147] The unified memory 206 is used to store input data and output data.

[0148] The weight data is directly transferred from the external memory to the input memory 201 and / or the unified memory 206 through the direct memory access controller 205 (DMAC), the weight data in the external memory is stored in the weight memory 202, and the data in the unified memory 206 is stored in the external memory.

[0149] The bus interface unit (BIU) 210 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 209 through the bus.

[0150] An instruction fetch buffer 209 connected to the controller 204 is used to store instructions used by the controller 204 .

[0151] The controller 204 is used to call the instructions cached in the memory 209 to control the working process of the computing accelerator.

[0152] Generally, the unified memory 206, the input memory 201, the weight memory 202 and the instruction fetch memory 209 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0153] Next, several application scenarios of this application are introduced.

[0154] Figure 3a A schematic diagram of the structure of an image processing system provided in an embodiment of the present application, wherein the image processing system includes a user device ( Figure 3a The user device (using a vehicle as an example) and an image processing device. It is understood that, in addition to being a vehicle, the user device can also be a mobile phone, an in-vehicle terminal, an aircraft terminal, a VR / AR device, an intelligent robot, and other intelligent terminals. The user device is the initiator of image processing and, as the initiator of the image processing request, is typically initiated by a user through the user device.

[0155] The image processing device described above can be a device or server with image processing capabilities, such as a cloud server, network server, application server, or management server. The image processing device receives image processing requests from smart terminals via an interactive interface and then uses its data storage and image processing processor to perform image processing using methods such as machine learning, deep learning, search, reasoning, and decision-making. The memory in an image processing device is a general term that includes local storage and a database that stores historical data. The database can be located on the image processing device or on another network server.

[0156] exist Figure 3a In the image processing system shown, a user device can receive user instructions. For example, the user device can obtain an image input / selected by the user (or an image captured by the user device through a camera), and then initiate a request to the image processing device, causing the image processing device to execute an image processing application (e.g., lane line detection in the image, etc.) on the image obtained by the user device, thereby obtaining a corresponding processing result for the image. For example, the user device can obtain an image input by the user, and then initiate an image detection request to the image processing device, causing the image processing device to detect the image, thereby obtaining a detection result of the image (i.e., a set of lane line points), and display the image detection result for the user to view and use.

[0157] exist Figure 3a In the embodiment, the image processing device can execute the image processing method of the embodiment of the present application.

[0158] Figure 3b Another structural diagram of the image processing system provided in the embodiment of the present application is shown in FIG. Figure 3b In the user equipment ( Figure 3b In this example, a vehicle is used as an image processing device. The user device can directly obtain images and process them directly by the hardware of the user device itself. The specific process is the same as Figure 3a Similarly, please refer to the above description and will not be repeated here.

[0159] Optionally, in Figure 3b In the image processing system shown, the user device can receive user instructions. For example, the user device can obtain an image selected by the user in the user device, and then the user device itself executes an image processing application (for example, lane line detection in the image, etc.) on the image, thereby obtaining the corresponding processing result for the image, and displays the processing result for the user to view and use.

[0160] Optionally, in Figure 3b In the image processing system shown, the user device can collect images of the road where the user device is located in real time or periodically, and then the user device itself performs image processing applications on the image (for example, lane line detection in the image, etc.), thereby obtaining corresponding processing results for the image, and implementing intelligent driving functions based on the processing results, such as: adaptive cruise control, lane departure warning (LDW), lane keeping assist (LKA), etc.

[0161] exist Figure 3b In the embodiment, the user equipment itself can execute the image processing method of the embodiment of the present application.

[0162] above Figure 3a and Figure 3b The user equipment in the example may be Figure 1 The client device 140 or the execution device 110 in Figure 3a The image processing device in the embodiment can be Figure 1 The execution device 110 in the embodiment, wherein the data storage system 250 can store the data to be processed by the execution device 210, and the data storage system 250 can be integrated on the execution device 210, or can be set on the cloud or other network servers.

[0163] Figure 3a and Figure 3bThe processor in the image processing apparatus can perform data training / machine learning / deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned from the data to perform image processing applications on the image, thereby obtaining corresponding processing results.

[0164] The following describes the vehicle architecture in the above scenario. Figure 4 , Figure 4 A schematic structural diagram of a vehicle provided in an embodiment of the present application.

[0165] The vehicle may include various subsystems, such as a travel system 402, a sensor system 404, a control system 406, one or more peripheral devices 408, a power source 410, and a user interface 416. Alternatively, the vehicle may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of the vehicle may be interconnected via wired or wireless (e.g., Bluetooth).

[0166] The travel system 402 may include components that provide powered movement for the vehicle. In one embodiment, the travel system 402 may include an engine 418 , an energy source 419 , a transmission 420 , and wheels 421 .

[0167] Among them, the engine 418 can be an internal combustion engine, an electric motor, an air compression engine, or a combination of other types of engines, for example, a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine. The engine 418 converts the energy source 419 into mechanical energy. Examples of energy sources 419 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. The energy source 419 can also provide energy for other systems of the vehicle. The transmission 420 can transmit the mechanical power from the engine 418 to the wheels 421. The transmission 420 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 420 may also include other devices, such as a clutch. Among them, the drive shaft may include one or more shafts that can be coupled to the wheels 421.

[0168] The sensor system 404 may include several sensors that sense information about the vehicle's location. For example, the sensor system 404 may include a positioning system 422 (e.g., a global positioning system, a BeiDou system, or other positioning systems), an inertial measurement unit (IMU) 424, a radar 426, a laser rangefinder 428, and a camera 430. The sensor system 404 may also include sensors for the monitored vehicle's internal systems (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). The sensory data from one or more of these sensors can be used to detect objects and their corresponding characteristics (e.g., position, shape, direction, speed, etc.). This detection and recognition is a key function for the safe operation of autonomous vehicles.

[0169] Among them, the positioning system 422 can be used to estimate the geographic location of the vehicle, such as the latitude and longitude information of the vehicle's location. The IMU 424 is used to sense the position and orientation changes of the vehicle based on inertial acceleration. In one embodiment, the IMU 424 can be a combination of an accelerometer and a gyroscope. The radar 426 can use radio signals to sense objects in the vehicle's surrounding environment, and can specifically be manifested as a millimeter wave radar or a lidar. In some embodiments, in addition to sensing objects, the radar 426 can also be used to sense the speed and / or direction of travel of objects. The laser rangefinder 428 can use lasers to sense objects in the environment in which the vehicle is located. In some embodiments, the laser rangefinder 428 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. The camera 430 can be used to capture multiple images of the vehicle's surrounding environment. The camera 430 can be a still camera or a video camera.

[0170] The control system 406 controls the operation of the vehicle and its components and may include various components, including a steering system 432 , an accelerator 434 , a brake unit 436 , an electronic control unit (ECU) 438 , and a body control module (BCM) 440 .

[0171] The steering system 432 is operable to adjust the vehicle's forward direction. For example, in one embodiment, it may be a steering wheel system. The throttle 434 is used to control the operating rate of the engine 418 and, in turn, the vehicle's speed. The brake unit 436 is used to control vehicle deceleration. The brake unit 436 may use friction to slow the wheels 421. In other embodiments, the brake unit 436 may convert the kinetic energy of the wheels 421 into electrical current. The brake unit 436 may also take other forms to slow the rotational speed of the wheels 421 and thus control the vehicle's speed. The vehicle electronic control unit 438 may be implemented as a single ECU or multiple ECUs on the vehicle, configured to communicate with the peripheral devices 408 and the sensor system 404. The vehicle ECU 438 may include at least one processor 4381 and memory 4382 (read-only memory, ROM). The at least one processor may be implemented or executed using one or more general-purpose processors, content-addressable memory, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, any suitable programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. In particular, the at least one processor may be implemented as one or more microprocessors, controllers, microcontroller units (MCUs), or state machines. Furthermore, the at least one processor may be implemented as a combination of computing devices, such as a digital signal processor or a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other combination of such configurations. ROM may provide storage for data, including, in this application, storage of addresses, routes, and driving directions.

[0172] BCM140 can provide ECU438 with information such as vehicle engine status, speed, gear position, steering wheel angle, etc.

[0173] The vehicle interacts with external sensors, other vehicles, other computer systems, or users via peripherals 408. Peripherals 408 may include a wireless communication system 446, a navigation system 448, a microphone 450, and / or a speaker 452. In some embodiments, peripherals 408 provide a means for the vehicle user to interact with user interface 416. For example, navigation system 448 may be implemented as part of an in-vehicle entertainment system, an in-vehicle display system, an in-vehicle instrument cluster, or the like. In one embodiment, navigation system 448 is implemented to include or collaborate with sensor system 404, which derives the vehicle's current geographic location in real time or substantially real time. Navigation system 448 is configured to provide navigation data to the vehicle's driver. The navigation data may include the vehicle's location data, suggested route planning driving instructions, and visible map information for the vehicle operator. Navigation system 448 may present this location data to the vehicle's driver via a display element or other presentation device. The current location of the vehicle may be described by one or more of the following: triangulated location, latitude / longitude location, x and y coordinates, or any other symbol or measurement indicating the geographic location of the vehicle.

[0174] The user interface 416 can also operate the navigation system 448 to receive user input. The navigation system 448 can be operated via a touch screen. The navigation system 448 provides route planning and navigation capabilities when the user enters geographic location values ​​for the starting and ending points. In other cases, the peripheral device 408 can provide a means for the vehicle to communicate with other devices located within the vehicle. For example, the microphone 450 can receive audio (e.g., voice commands or other audio input) from the user of the vehicle. Similarly, the speaker 452 can output audio to the user of the vehicle. The wireless communication system 446 can wirelessly communicate with one or more devices directly or via a communication network. For example, the wireless communication system 446 can use 3G cellular communication, such as code division multiple access (CDMA), EVDO, global system for mobile communications (GSM) / general packet radio service (GPRS), or 4G cellular communication, such as long term evolution (LTE), or 5G cellular communication. The wireless communication system 446 can use WiFi to communicate with a wireless local area network (WLAN). In some embodiments, wireless communication system 446 may utilize infrared links, Bluetooth, or ZigBee to communicate directly with devices. Other wireless protocols, such as various vehicle communication systems, may be used. For example, wireless communication system 446 may include one or more dedicated short range communications (DSRC) devices that may enable public and / or private data communications between vehicles and / or roadside stations.

[0175] Power source 410 can provide power to various components of the vehicle. In one embodiment, power source 410 can be a rechargeable lithium-ion or lead-acid battery. One or more battery packs of such batteries can be configured as a power source to provide power to various components of the vehicle. In some embodiments, power source 410 and energy source 419 can be implemented together, such as in some all-electric vehicles.

[0176] Alternatively, one or more of the above components may be installed or associated separately from the vehicle. For example, the memory 4382 may be partially or completely separate from the vehicle. The above components may be communicatively coupled together in a wired and / or wireless manner.

[0177] Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 4It should not be understood as limiting the embodiments of the present application.

[0178] The above-mentioned vehicles can be cars, trucks, motorcycles, buses, boats, lawn mowers, recreational vehicles, amusement park vehicles, construction equipment, trams, golf carts, and carts, etc., and the embodiments of the present application do not make special limitations.

[0179] The image processing method provided by the embodiment of the present application is described below. The method can be executed by an image processing device, or by a component of the image processing device (such as a processor, a chip, or a chip system, etc.). The image processing device can be a cloud device (such as the aforementioned Figure 3a as shown), or a vehicle (e.g. Figure 4 Vehicle shown) or terminal equipment (such as vehicle-mounted terminal, aircraft terminal, etc.) (as mentioned above Figure 3b Of course, the method can also be performed by a system consisting of a cloud device and a vehicle (as described above). Figure 3a Optionally, the method may be processed by a CPU in an image processing device, or may be processed jointly by a CPU and a GPU, or may use other processors suitable for neural network computing without using a GPU, and this application does not impose any restrictions.

[0180] The application scenario of this method (or understood as the application scenario of the first neural network or the target neural network) can be used in intelligent driving scenarios. For example: adaptive cruise control, lane departure warning (LDW), lane keeping assist (LKA), etc., which include lane line detection. In the intelligent driving scenario, the image processing method provided in the embodiment of the present application can obtain the image to be detected through the sensor on the vehicle (such as a camera), and obtain the lane line in the image to be detected, thereby realizing the above-mentioned adaptive cruise control, LDW or LKA, etc.

[0181] In the embodiment of the present application, depending on whether the image processing device is a cloud device or a user device, the image processing method provided in the embodiment of the present application may include two cases, which are described below respectively.

[0182] In the first case, the image processing device is a user device. Here, we only take the user device as an example (as mentioned above) Figure 3b It is understandable that, in addition to being a vehicle, the user device may also be a mobile phone, a vehicle-mounted terminal, an aircraft terminal, a VR / AR device, an intelligent robot, or other intelligent terminal, without limitation here.

[0183] See also Figure 5, a flow chart of an image processing method provided by an embodiment of the present application, the method is implemented by a target neural network, and the method may include steps 501 to 504. Steps 501 to 504 are described in detail below.

[0184] Step 501: Acquire an image to be detected.

[0185] In the embodiment of the present application, there are many ways for the image processing device to obtain the image to be detected. It can be a method of collecting the image to be detected by the image processing device, a method of receiving the image to be detected sent by other devices, or a method of selecting training data from a database, etc. The specific methods are not limited here.

[0186] Optionally, the image to be detected includes at least one of a car, a person, an object, a tree, a sign, etc.

[0187] For example, in the field of intelligent driving, the image processing device may be a vehicle. Sensors on the vehicle (e.g., cameras) capture images. It is understood that the sensors on the vehicle may capture images in real time or periodically, for example, every 0.5 seconds, without limitation herein.

[0188] Step 502: extract features from the image to be detected to obtain a first feature.

[0189] After the image processing device acquires the image to be detected, it can obtain the first feature of the image to be detected. Specifically, feature extraction is performed on the image to be detected to obtain the first feature. It is understood that the features mentioned in the embodiments of the present application can be expressed in a matrix or vector format.

[0190] Optionally, the image processing device may extract features from the image to be detected through a backbone network to obtain the first feature. The backbone network may be a convolutional neural network, a graph convolutional network (GCN), a recurrent neural network, or other network capable of extracting image features, the specific details of which are not limited herein.

[0191] Furthermore, to obtain multi-level features of the image to be detected, the image processing device can perform feature fusion and dimensionality reduction on the features output by different layers in the backbone network to obtain the first feature. The features output by different layers can also be understood as intermediate features (also referred to as at least one third feature) in the process of calculating the first feature. The number of third features is related to the number of layers in the backbone network. For example, the number of third features is the same as the number of layers in the backbone network, or the number of third features is the number of layers in the backbone network minus 1.

[0192] In this way, due to the different performance of features extracted from different layers of the neural network, low-level features have higher resolution and contain more position and detail information, but due to fewer convolutions, their semantics are lower and there is more noise. High-level features have stronger semantic information, but low resolution and poor perception of details. Therefore, the features extracted from different layers of the backbone network are fused to obtain the fused features (denoted as H f ), the fused features have multi-level features. Further, the fused features are processed by dimensionality reduction to obtain the first feature (denoted as H′ f ). Therefore, the first feature also has multi-level features. Among them, the above H f ∈R h×w×d , h is H f The number of rows, w is H f The number of columns, d is H f For example, through a linear transformation layer, H f The dimension d is reduced to d′, that is, H′ f ∈R h×w×d′ .

[0193] Exemplarily, the above-mentioned backbone network adopts a 50-layer residual convolutional neural network (Residual Neural Network-50, ResNet50).

[0194] Step 503: Process the detection frame information of the image to be detected to obtain a second feature.

[0195] After acquiring the image to be detected, the image processing device can first obtain the detection frame information of the image to be detected based on the human-vehicle detection model. Specifically, the image to be detected is input into the human-vehicle detection model to obtain the detection frame information, and the detection frame information includes the position of the detection frame of at least one object in the image to be detected in the image to be detected. Among them, the human-vehicle detection model can be a region convolutional neural network (R-CNN), a fast region convolutional neural network (fast R-CNN) or a faster region convolutional neural network (faster R-CNN), etc., which are not specifically limited here. The above-mentioned objects may include at least one of the cars, people, objects, trees, signs, etc. in the image to be detected, which are not specifically limited here. It can be understood that the position of the detection frame can be a normalized position.

[0196] It can be understood that if more detection frame information of objects in the image to be detected is obtained, the expression ability of the obtained second feature will be stronger.

[0197] Optionally, the detection box information may further include the category and confidence level of the detection box.

[0198] After the image processing device obtains the detection frame information, it can process the detection frame information to obtain the second feature. The second feature can also be understood as the detection frame feature of the image to be detected. The second feature includes the position feature and semantic feature of the detection frame corresponding to the object in the image to be detected. Among them, the position feature can be recorded as Z b , the semantic feature can be recorded as Z r .

[0199] Optionally, at least one third feature and the detection frame information are processed to obtain a second feature. The at least one third feature is an intermediate feature obtained during the process of obtaining the first feature (such as the intermediate feature in step 502). Specifically, the detection frame information and the intermediate feature are input into a preprocessing module to obtain a positional feature and a semantic feature.

[0200] Optionally, if the backbone network adopts a feature pyramid network (FPN) structure, the second feature can be obtained by processing at least one third feature and the detection box information. If the backbone network does not adopt the FPN structure, the second feature can be obtained by using the first feature before dimensionality reduction and the detection box information.

[0201] In the embodiments of the present application, the specific process of obtaining the second feature (which can also be understood as the function of the preprocessing module) varies based on the different detection frame information, which are described below:

[0202] 1. The detection frame information only includes the location of the detection frame.

[0203] The above process of obtaining semantic features may include: scaling the detection frame according to the position of the detection frame and the sampling rate between different layers in the backbone network. Using the scaled detection frame, the ROI feature is extracted from the feature layer corresponding to the sampling rate of the intermediate feature. The ROI feature is processed (for example, input to the fully connected layer, or input to the single-layer perceptron and activation layer) to obtain the semantic feature of the detection frame: Z r ∈R M×d′ , where M is the number of detection boxes in the image to be detected.

[0204] The above process of obtaining position features may include: processing the vector corresponding to the position of the detection frame (for example, inputting the processing of the fully connected layer, or inputting the processing of the single-layer perceptron and the activation layer) to obtain the position feature of the detection frame: Z b ∈R M ×d′ .

[0205] For example, assuming the backbone network is a 5-layer neural network with a downsampling rate of 8 in the third layer, we will reduce the original detection box by a factor of 8. Generally speaking, the larger the detection box area, the smaller the feature layer (the later the layer) is used to extract the ROI features.

[0206] 2. The detection box information includes the location and confidence of the detection box.

[0207] The above process of obtaining semantic features may include: scaling the detection frame according to the position of the detection frame and the sampling rate between different layers in the backbone network. Using the scaled detection frame, the ROI feature is extracted from the feature layer of the intermediate feature corresponding to the sampling rate. The confidence of the detection frame is used as a coefficient and multiplied with the extracted ROI feature, and the multiplied feature is processed (for example, input to the fully connected layer, or input to the single-layer perceptron and activation layer) to obtain the semantic feature of the detection frame: Z r ∈R M×d′ , where M is the number of detection boxes in the image to be detected.

[0208] The above process of obtaining position features may include: processing the vector corresponding to the position of the detection frame (for example, inputting the processing of the fully connected layer, or inputting the processing of the single-layer perceptron and the activation layer) to obtain the position feature of the detection frame: Z b ∈R M ×d′ Among them, the category of the detection box can be encoded using a one-hot encoding method to obtain a category vector.

[0209] 3. The detection box information includes the location, confidence and category of the detection box.

[0210] The above process of obtaining semantic features may include: scaling the detection frame according to the position of the detection frame and the sampling rate between different layers in the backbone network. Using the scaled detection frame, the ROI feature is extracted from the feature layer of the first feature corresponding to the sampling rate. The confidence of the detection frame is used as a coefficient and multiplied with the extracted ROI feature, and the multiplied feature is processed (for example, input to the fully connected layer, or input to the single-layer perceptron and activation layer) to obtain the semantic feature of the detection frame: Z r ∈R M×d′ , where M is the number of detection boxes in the image to be detected.

[0211] The process of obtaining the position feature may include: transforming the category of the detection frame into a category vector. Then, the category vector is concatenated with the vector corresponding to the position of the detection frame, and processed (for example, input to a fully connected layer, or input to a single-layer perceptron and activation layer) to obtain the position feature of the detection frame: Z b ∈R M×d′Among them, the category of the detection box can be encoded using a one-hot encoding method to obtain a category vector.

[0212] It can be understood that the above-mentioned several situations of detection frame information and several specific processes for obtaining the second feature are just examples. In actual applications, there may be other situations for the detection frame (for example: the detection frame information includes the position and category of the detection frame), and there may be other ways to obtain the second feature, which are not limited here.

[0213] For example, the process of obtaining the second feature can be as follows: Figure 6 The steps performed by the detection preprocessing module refer to the above description of the process of obtaining the second feature, which will not be repeated here.

[0214] In step 504 , the first feature and the second feature are input into a first neural network based on a transformer structure to obtain lane lines in the image to be detected.

[0215] After acquiring the first and second features, the image processing device can input the first and second features into a first neural network based on a transformer structure to obtain lane markings in the image to be detected. Specifically, multiple point sets can be acquired first, and then lane markings can be determined based on the multiple point sets. Each point set in the multiple point sets represents a lane marking in the image to be detected.

[0216] Optionally, the first neural network based on the transformer structure includes an encoder, a decoder, and a feedforward neural network. Acquiring multiple point sets may include the following steps: inputting the first and second features into the encoder to obtain a fourth feature; inputting the fourth, second, and query features into the decoder to obtain a fifth feature; and inputting the fifth feature into the feedforward neural network to obtain multiple point sets. This will be described later with reference to the accompanying figures and in different scenarios, and will be expanded upon here.

[0217] Optionally, the first feature and the second feature can be input into a trained first neural network to obtain multiple point sets. The trained first neural network is obtained by using training data as input to the first neural network and training the first neural network with the goal of ensuring that the value of a first loss function is less than a first threshold. The training data includes the first feature of the training image, the positional features and semantic features of the detection box corresponding to the object in the training image, and the first loss function is used to represent the difference between the point set output by the first neural network during training and the first point set. The first point set is a real point set of actual lane lines in the training image.

[0218] Furthermore, the first neural network includes a transformer structure and a feedforward neural network. The first and second features can be processed using the transformer structure to obtain a fifth feature. The fifth feature is then input into the feedforward neural network to obtain multiple point sets. It is understood that the feedforward neural network here can also be replaced by a fully connected layer, a convolutional neural network, or other structures, and the specifics are not limited here.

[0219] In the embodiment of the present application, the transformer structure is different based on the different inputs of the first neural network. It can also be understood that the steps for obtaining the fifth feature are different, which are described below.

[0220] The first neural network is as follows Figure 7 As shown, the transformer structure is as follows Figure 8 shown.

[0221] In one possible implementation, in order to more intuitively see the process of obtaining the fifth feature based on the first feature and the second feature, you can refer to Figure 7 The first neural network includes a transformer structure and a feedforward neural network. The first feature and the second feature are input into an encoder of the transformer structure to obtain a fourth feature. The query feature, the second feature, and the fourth feature are input into a decoder of the transformer structure to obtain a fifth feature.

[0222] The transformer structure in this case can be as follows Figure 8 As shown, the encoder of the transformer structure includes a first self-attention module and a first attention module, and the decoder of the transformer structure includes a second attention module and a third attention module.

[0223] Optionally, the decoder may further include a second self-attention module ( Figure 8 (not shown) is used to calculate query features. Specifically, self-attention calculation is performed on the query vector to obtain query features. The query vector is initialized to a random value and trained to a fixed value during the training process. This fixed value is used during inference, that is, the query vector is a fixed value obtained through training during the random value training process.

[0224] In this structure, the first feature (H′) is f ) performs self-attention calculation and obtains the first output (O f ). The first feature (H′) is analyzed by the first attention module. f ) and the second characteristic (Z r With Z b ) performs cross attention calculation and obtains the second output (Op2b ). Based on the first output (O f ) and the second output (O p2b ) to obtain the fourth feature. The query feature (Q q ) and the fourth feature to perform cross attention calculation to obtain the third output. q ) and the second characteristic (Z r With Z b ) to obtain a fourth output. The third output and the fourth output are added to obtain a fifth feature. The query feature is obtained by performing self-attention calculation on the query vector.

[0225] Optionally, the first feature (H′) is f ) performs self-attention calculation and obtains the first output (O f ) can be specifically: Since it is a self-attention calculation, the input of QKV is consistent (that is, both are H′ f ). That is, through the first feature (H′ f ) is processed by three linear processes to obtain QKV, and O is calculated based on QKV. f For the description of self-attention, please refer to the aforementioned description of the self-attention mechanism, which will not be repeated here. In addition, it can be understood that in the process of calculating self-attention, the position matrix of the first feature can be introduced, which is described in the subsequent formula 1 and will not be expanded here.

[0226] Optionally, the above-mentioned f With O p2b The specific steps of obtaining the fourth feature may be: adding the first output and the second output to obtain the fourth feature.

[0227] Further, such as Figure 9 As shown, the above is based on the first output (O f ) and the second output (O p2b The step of obtaining the fourth feature may specifically include: adding the first output and the second output, adding the result of the addition to the first feature, and normalizing the result to obtain an output. This output is then input into a feedforward neural network to obtain an output of the feedforward neural network. The output obtained by the addition and normalization is then added to the output of the feedforward neural network and normalized to obtain the fourth feature.

[0228] Optionally, the first attention module is used to f 、Z r and Z b The specific steps for cross attention calculation can be: H′ f As Q, Z bAs K, Z r And as V, cross attention calculation is performed to obtain the second output (O p2b ).

[0229] Optionally, the second attention module is used to q The specific steps of calculating the cross attention with the fourth feature can be: q As Q, the fourth feature is used as K and V for cross attention calculation to obtain the third output.

[0230] Further, such as Figure 10 As shown, the above step of processing the query feature and the second feature to obtain the fourth output can be specifically: using the third attention module to process Q q 、Z r and Z b Perform cross attention calculation to obtain the sixth output. Specifically, Q q As Q, Z b As K, Z r The query feature and the sixth output are added together as V to obtain a sixth output. The query feature and the sixth output are added together, and the result of the addition is added and normalized with the query vector to obtain an output. On the other hand, this output is input into a feedforward neural network to obtain an output of the feedforward neural network. The output obtained by the above addition and normalization is added and normalized with the output of the feedforward neural network to obtain a fourth output.

[0231] It should be noted that in this embodiment, the feature used as Q in the attention calculation process can be introduced into the position matrix of the feature (Q q ). The position matrix can also be obtained by static position coding or dynamic position coding. For example, the position matrix can be calculated based on the absolute position of the feature map corresponding to the first feature, which is not limited here.

[0232] For example, the first output ( f ), the second output (O p2b ) is calculated as follows:

[0233] Formula 1:

[0234] Formula 2:

[0235] Among them, taking Formula 1 and Formula 2 as examples, E f The first feature (H′ f ) position matrix, the following formula and formula 4 are used as an example to illustrate the calculation of the position matrix by the sine and cosine method:

[0236] Formula 3:

[0237] Formula 4:

[0238] Even numbers are calculated using Formula 3, while odd numbers are calculated using Formula 4. i represents the row position of the element in the position matrix, 2j / 2j+1 represents the column position of the element in the position matrix, and d represents the dimension of the position matrix. To more clearly understand the application of Formulas 3 and 4, assume that if an element is in row 2 and column 3, then the position vector of that element is , which can be calculated using Formula 4, where i = 2, j = 1, and d = 3.

[0239] It is understandable that the above formulas 1, 2, 3 and 4 are only examples. In practical applications, there may be other forms of formulas, which are not specifically limited here.

[0240] The second type, the first neural network is as follows Figure 11 As shown, the transformer structure is as follows Figure 12 shown.

[0241] In another possible implementation, see Figure 11 ,in, Figure 11 and Figure 7 The difference is that Figure 7 The input of the encoder includes the first feature and the second feature, Figure 11 The input of the encoder includes the first feature, the first row feature, the first column feature, and the second feature. Figure 11 The input ratio of the encoder Figure 7 The input of the encoder has the first row and first column features.

[0242] The transformer structure in this case is as follows Figure 12 As shown, the encoder of the transformer structure has Figure 8 In addition to the structure shown, it also includes row and column attention modules. Figure 12 The encoder of the transformer structure shown in the figure includes a row-column attention module, a first self-attention module, and a first attention module, while the decoder includes a second self-attention module, a second attention module, and a third attention module. The row-column attention module includes a row attention module and a column attention module.

[0243] In this structure, the first row feature (H′) is r ) performs self-attention calculation to obtain row output. The first column feature (H′ c) performs self-attention calculation to obtain column output. Based on the row output and column output, the row and column outputs are obtained. The first feature (H′) is calculated by the first self-attention module. f ) performs self-attention calculation and obtains the first output (O f ). The first feature (H′) is analyzed by the first attention module. f ) and the second characteristic (Z r With Z b ) performs cross attention calculation and obtains the second output (O p2b ). Based on row and column output, first output (O f ) and the second output (O p2b ) to obtain the fourth feature. The query vector is self-attentioned by the second self-attention module to obtain the query feature (Q q ). The query feature (Q q ) and the fourth feature to perform cross attention calculation to obtain the third output. The query feature (Q q ) and the second characteristic (Z r With Z b ) to obtain a fourth output. The third output and the fourth output are added together to obtain a fifth feature.

[0244] It is understandable that some of the above steps and related structures can refer to the aforementioned Figure 8 Similar descriptions to those in the illustrated embodiment will not be repeated here.

[0245] Alternatively, as Figure 13 As shown, it is the specific structure of the row and column attention module. The above-mentioned step of obtaining row and column outputs based on row output and column output can be specifically: the row output and the first row feature are added and normalized (abbreviated as addition & normalization) to obtain the output. On the one hand, the output is input into a feedforward neural network (abbreviated as feedforward network) to obtain the output result of the feedforward network. The output obtained by the above-mentioned addition and normalization is added and normalized to the output result of the feedforward network to obtain the output of the row. Similarly, the column output and the first column feature are added and normalized to obtain the output. On the one hand, the output is input into the feedforward network to obtain the output result of the feedforward network. The output obtained by the above-mentioned addition and normalization is added and normalized to the output result of the feedforward network to obtain the output of the column. The output of the row and the output of the column are then spliced ​​to obtain the row and column outputs.

[0246] Optionally, the first row feature, first column feature, row output, and column output are described. After obtaining the first feature, the first feature can be flattened in row dimension to obtain H r ∈R h×1×wd, and after processing (for example: input to the fully connected layer processing and dimensionality reduction processing, or input to the single-layer perceptron and activation layer processing and dimensionality reduction processing) to obtain the first row of features: H′ r ∈R h×1×d′ The above-mentioned flattening of the row dimension can also be understood as flattening or compressing the matrix corresponding to the first feature along the row direction to obtain H r Similarly, flatten the first feature by column dimension to obtain H c ∈R 1×w×hd , and after processing (for example: input to the fully connected layer processing and dimensionality reduction processing, or input to the single-layer perceptron and activation layer processing and dimensionality reduction processing) to obtain the first column of features: H′ c ∈R 1×w×d′ .

[0247] Optionally, the above-mentioned output based on rows and columns, the first output ( f ) and the second output (O p2b ) The steps of obtaining the fourth feature may specifically be: adding the first output and the second output to obtain the fifth output; and concatenating the fifth output with the row and column outputs to obtain the fourth feature.

[0248] For example, the above line output ( row ), column output (Oc olumn ) is calculated as follows:

[0249] Formula 5:

[0250] Formula 6:

[0251] Among them, E r The first row of features (H′ r )’s position matrix, E c is the first column feature (H′ c The position matrix may be obtained by static position coding or dynamic position coding, which is not limited here.

[0252] It is understandable that the above formulas 5 and 6 are just examples. In practical applications, there may be other forms of formulas, which are not specifically limited here.

[0253] It should be noted that the above-mentioned several cases of transformer structure, or the methods of obtaining the fifth feature are only examples. In actual applications, the transformer structure can also be other cases, or there can be other ways to obtain the fifth feature, which are not limited here.

[0254] After obtaining the fifth feature through any of the above methods, the image processing device can input the fifth feature into a feedforward neural network to obtain multiple point sets. Lane lines in the image to be detected are determined based on the multiple point sets. It is understood that the feedforward neural network described above can also be replaced by a fully connected layer, a convolutional neural network, or other structures, and the specifics are not limited here.

[0255] In order to understand the process of obtaining point sets more intuitively, Figure 14a For example, Figure 14a The lane line l shown in is (X, s, e), where X is the set of X coordinates corresponding to the intersection of the equally spaced Y-direction straight lines (e.g., 72) and the lane lines, the starting point Y coordinate is s, and the ending point Y coordinate is e. It can be understood that Figure 14a The number of center lane lines and the number of straight lines in the Y direction are just examples and are not specifically limited here.

[0256] In one possible implementation, multiple point sets can be presented as arrays. In another possible implementation, multiple point sets can also be presented as images. For example: Figure 14b Multiple point sets are shown. Multiple point sets are overlapped and fused with the image to be detected to obtain the image to be detected with multiple point sets, for example Figure 14c This embodiment does not limit the presentation method of multiple point sets.

[0257] To more intuitively see the contribution of the first row and first column features to lane line detection, please refer to Figure 14d ,It can be seen that by introducing the first row and first column features that can adapt to the lane line shape to mine context information, the network's ability to construct long lane line features can be improved, thereby achieving better lane line detection results.

[0258] In the embodiment of the present application, on the one hand, by applying the transformer structure to the lane line detection task, the global information of the image to be detected can be obtained, thereby effectively modeling the long-range connection between the lane lines. On the other hand, by adding the detection box position information of the object in the image as input to the lane line detection network, the scene perception ability of the network can be improved. Reduce the misjudgment of the model in the scene where the lane line is blocked by the vehicle. On the other hand, by introducing a row and column self-attention module in the transformer encoder that can adapt to the lane line shape to mine contextual information, the network's ability to construct long lane line features can be improved, thereby achieving better lane line detection results. On the other hand, the various modules in the existing autonomous driving system are often independent of each other. For example, the lane line detection model and the human-vehicle model are independent of each other and predicted separately. The target neural network in the image processing method provided in this embodiment uses the detection box information obtained based on the human-vehicle detection model to predict the lane line in the first neural network, which can improve the accuracy of lane line detection.

[0259] In the second case, the image processing device is a cloud server (as mentioned above Figure 3a It is understandable that in this case, the image processing device may also be a device or server with image processing capabilities, such as a network server, an application server, and a management server. The user device is a vehicle, and the specific description is not limited here.

[0260] See also Figure 15 , a flowchart of an image processing method provided by an embodiment of the present application, the method may include steps 1501 to 1505. Steps 1501 to 1505 are described in detail below.

[0261] Step 1501: The vehicle obtains an image to be detected.

[0262] Alternatively, the vehicle may collect the image to be detected based on a sensor on the vehicle (such as a camera or a camera). Of course, the sensor on the vehicle may also collect images periodically.

[0263] It is understandable that the vehicle may also be acquired by receiving images to be detected sent by other devices, which is not specifically limited here.

[0264] In step 1502, the vehicle sends an image to be detected to the server. Correspondingly, the server receives the image to be detected sent by the vehicle.

[0265] After the vehicle obtains the image to be detected, it sends the image to be detected to the server. Correspondingly, the server receives the image to be detected sent by the vehicle.

[0266] In step 1503, the server inputs the image to be detected into the trained target neural network to obtain multiple point sets.

[0267] After the server receives the image to be detected sent by the vehicle, it can input the image to be detected into the trained target neural network to obtain multiple point sets.

[0268] The trained target neural network uses the training image as input and is trained with the goal of ensuring the value of the target loss function is less than a target threshold. The target function represents the difference between the point set output by the target neural network during training and the target point set, which is the actual lane line point set in the training image. The target loss function and target threshold can be set based on actual needs and are not specified here.

[0269] The target neural network in this embodiment may include the aforementioned Figure 5 The backbone network, preprocessing module and first neural network in the embodiment shown. Figure 5 There are two cases for the structure of the first neural network in the illustrated embodiment. Therefore, there are also two cases for the target neural network in this embodiment, which are described below respectively.

[0270] In one possible implementation, the target neural network can be structured as follows: Figure 16 As shown. The target neural network in this case is equivalent to the aforementioned Figure 6 The backbone network shown, Figure 6 The preprocessing module shown, Figures 7 to 10 The corresponding first neural network. The specific description of the neural network and related processes can refer to the aforementioned Figures 6 to 10 The corresponding description will not be repeated here.

[0271] In another possible implementation, the target neural network can be structured as follows: Figure 17 As shown. The target neural network in this case is equivalent to the aforementioned Figure 6 The backbone network shown, Figure 6 The preprocessing module shown, Figures 11 to 13 The corresponding first neural network. The specific description of the neural network and related processes can refer to the aforementioned Figure 6 、 Figures 11 to 13 The corresponding description will not be repeated here.

[0272] In step 1504, the server sends multiple point sets to the vehicle. Correspondingly, the vehicle receives the multiple point sets sent by the server.

[0273] After the server obtains the multiple point sets, the server sends the multiple point sets to the vehicle.

[0274] Step 1505: Implement intelligent driving functions based on multiple point sets.

[0275] After acquiring multiple point sets, each of which represents a lane line in the image to be detected, the vehicle can determine the lane line in the image to be detected and implement intelligent driving functions such as adaptive cruise control, lane departure warning, and lane keeping assist based on the lane line.

[0276] In addition, the description of determining the lane line in the image to be predicted by multiple point sets can refer to the aforementioned Figure 5 The description of step 504 in the illustrated embodiment is similar and will not be repeated here.

[0277] It can be understood that the steps of this embodiment can be executed periodically, that is, the lane lines on the road surface can be accurately identified based on the images to be detected collected by the on-board camera during driving, thereby realizing functions related to lane lines in intelligent driving, such as: adaptive cruise control, lane departure warning, lane keeping assist, etc.

[0278] In this embodiment, on the one hand, by applying the transformer structure to the lane detection task, global information of the image to be detected can be obtained, thereby effectively modeling the long-range connections between lane lines. On the other hand, by adding the detection box position information of the object in the image as input to the lane detection network, the network's scene perception ability can be improved. This reduces the model's misjudgment in scenarios where the lane line is obscured by the vehicle. On the other hand, by introducing a row and column self-attention module in the transformer encoder that can adapt to the lane line shape and mine contextual information, the network's ability to construct long lane line features can be improved, thereby achieving better lane line detection results. On the other hand, by deploying the target neural network in the cloud and predicting the lane line point set, the vehicle's computing power overhead can be saved. On the other hand, the various modules in existing autonomous driving systems are often independent of each other. For example, the lane line detection model and the human-vehicle model are independent of each other and predicted separately. The target neural network in the image processing method provided in this embodiment improves the accuracy of lane line detection by utilizing the detection box information obtained based on the human-vehicle detection model in the first neural network to predict the lane line.

[0279] In order to more intuitively see the performance of the target neural network provided in the embodiments of the present application, the target neural network (hereinafter referred to as Laneformer) is tested on the CULane and TuSimple datasets with other existing networks. Other existing networks include: spatial convolutional neural network (SCNN), ENeTSAD, PointLane, efficient residual factorized network (ERFNet), CurveLaneS, CurveLaneM, CurveLaneL, LaneATT.

[0280] Among them, CULane is a large-scale lane detection dataset collected in Beijing, China through on-board cameras, and the size of the collected images is 1640×590. The dataset was collected in various locations and contains samples of many complex urban scenes. The CULane dataset contains 88,880 training images, 9,675 verification images, and 34,680 test images. Among them, the test set is also divided into nine categories, one category is regular images, and the other eight categories are challenging special categories (including shadow scenes, highlight scenes, night scenes, curve scenes, scenes without lane lines, etc.). TuSimple is an autonomous driving dataset collected by TuSimple. This dataset focuses on highway scenes, so all images are collected on highways, and the size of the collected images is 1280×720. The TuSimple dataset contains 3,626 images for training and 2,782 images for testing.

[0281] For the backbone network of LaneATT, three residual structures (ResNet18, ResNet34, ResNet122) are adopted. They are respectively denoted as: LaneATT (ResNet18), LaneATT (ResNet34), LaneATT (ResNet122). The backbone network of the Laneformer provided in the embodiment of the present application adopts three residual structures (ResNet18, ResNet34, ResNet50), which are respectively denoted as: Laneformer (ResNet18), Laneformer (ResNet34), Laneformer (ResNet50). And the network in the Laneformer under the ResNet50 structure that does not include the detection attention module (i.e., the first attention module and the third attention module) is denoted as: Laneformer (ResNet50)*.

[0282] The detection accuracy of different lane detection methods on CULane is shown in Table 1:

[0283] Table 1

[0284]

[0285]

[0286] As shown in Table 1, the Laneformer model, using ResNet50 as the backbone network, achieves the current state-of-the-art result on the entire CULane test set, with a score of 77.06%. In addition to achieving optimal results on the entire test set, Laneformer also achieves optimal results on several challenging scene categories, such as night scenes (Night), bright light scenes (Dazzle), and intersection scenes (Cross) (only a subset is shown in Table 1). The Laneformer model performs particularly well on the intersection scene category, with the number of misdetected images two orders of magnitude lower than other models. Since intersection scenes do not have annotated lane lines in the dataset, the performance metric for intersection scenes is FP. While the other models achieve FP values ​​in the thousands for intersection scenes, the Laneformer model proposed in this work achieves an FP value of 19. Table 1 indicates that this improvement comes from the addition of the detection attention module. In the Laneformer (ResNet50)* model without the detection attention module, the FP for intersection scenarios is low, but still in the thousands. However, with the addition of the detection attention module, this metric drops sharply to tens. This indicates that in complex intersection scenarios with complex pedestrian and vehicle traffic, the detection attention module can significantly reduce the model's misprediction rate by perceiving the surrounding scene and objects.

[0287] The detection accuracy of different lane detection methods on TuSimple is shown in Table 2:

[0288] Table 2

[0289] Model Accuracy (%) False positive rate (%) False negative rate (%) SCNN 96.53 6.17 1.8 LSTR 96.18 2.91 3.38 EnetSAD 96.64 6.02 2.05 LineCNN 96.87 4.41 3.36 PolyLaneNet 93.36 9.42 9.33 PointLaneNet 96.34 4.67 5.18 LaneATT(ResNet18) 95.57 3.56 3.01 LaneATT(ResNet34) 95.63 3.53 2.92 LaneATT(ResNet122) 96.1 5.64 2.17 Laneformer(ResNet50)* 96.72 3.46 2.52 Laneformer(ResNet18) 96.54 4.35 2.36 Laneformer (ResNet34) 96.56 5.39 3.37 Laneformer (ResNet50) 96.8 5.6 1.99

[0290] Table 2 shows that the Laneformer model, using ResNet50 as its backbone network, achieved 96.8% accuracy, a false positive rate of 5.6%, and a false negative rate of 1.99% on the TuSimple dataset. In terms of the most important metric, accuracy, Laneformer was only 0.07% lower than the leading LineCNN and 0.6% higher than LSTR, which also uses a self-attention transformer network. Furthermore, unlike its performance on the CULane dataset, using smaller backbone networks such as ResNet18 and ResNet34 also yielded very competitive results on the TuSimple dataset, with the performance differences due to different backbone networks being almost negligible. Furthermore, a model using only row-column attention modules (i.e., Laneformer (ResNet50)*) also achieved very good results on the TuSimple dataset.

[0291] To more clearly demonstrate the individual contributions of each module within the target neural network, the following performance tests are conducted on the CULane dataset using different approaches. These tests include the effects of using only the row-column attention module, as well as the use of different submodules within the detection attention module. The results are also analyzed, including the impact of using the bounding box, confidence score, and category of the vehicle and person detection boxes as input to the detection preprocessing module.

[0292] The test results are shown in Table 3:

[0293] Table 3

[0294] Model F1(％) Accuracy (%) Recall rate (%) Frame rate per second Parameter quantity (millions) Baseline (ResNet50) 75.45 81.65 70.11 61 31.02 + Row and column attention 76.04 82.92 70.22 58 43.02 +Detection frame location information 76.08 85.3 68.66 57 45.38 +Confidence of the detection box 76.25 83.56 70.12 54 45.38 +Detection box category 77.06 84.05 71.14 53 45.38

[0295] Among them, the first model (i.e. Baseline) can be understood as Figure 16 The target neural network shown is the network after removing the first and third attention modules. The second model (+row and column attention) can be understood as the first model plus the row and column attention module, the third model (+ detection box position information) can be understood as the second model plus the detection box position information, the fourth model (+ detection box confidence) can be understood as the third model plus the detection box confidence, and the fifth model (+ detection box category) can be understood as the fourth model plus the detection box category. The fifth model can be regarded as the aforementioned Figure 17 The target neural network shown.

[0296] The Laneformer model proposed in this paper builds on the Transformer model by adding row-column attention modules and detection attention modules (including the first and third attention modules). The detection attention module is divided into three types: simply incorporating detection box information, adding detection box confidence, and adding predicted categories. Therefore, this section experimentally explores the impact of each module on the model. Table 3 shows that in a simple Transformer model without row-column and detection attention modules, the baseline F1 score reaches 75.45%. After adding the row-column attention modules, the model's performance improves to 76.04%. Furthermore, simply incorporating detection box information from the person and vehicle detection modules improves model performance. Furthermore, incorporating detection box confidence into the detection information allows the model to achieve an F1 score of 76.25%. Incorporating detection box category information also yields the optimal model (Table 3), achieving an F1 score of 77.06%. This demonstrates that both row-column and detection attention modules can improve model performance. In addition, it can be observed that the addition of the detection attention module can significantly improve the accuracy of the model, while the impact on the recall rate is relatively weak.

[0297] The above describes the image processing method provided by the embodiment of the present application. The following describes the lane line detection method provided by the embodiment of the present application. The method can be executed by a detection device or by a component of the detection device (such as a processor, a chip, or a chip system). The detection device can be a terminal device (such as a vehicle terminal, an aircraft terminal, etc.) (such as the aforementioned Figure 3b Optionally, the method may be processed by a CPU in the detection device, or may be processed jointly by a CPU and a GPU, or may use other processors suitable for neural network calculations without using a GPU, and this application does not impose any restrictions.

[0298] The application scenario of this method (or understood as the application scenario of the first neural network) can be used in intelligent driving scenarios. For example: adaptive cruise control, lane departure warning (LDW), lane keeping assist (LKA), etc., which include lane line detection. In the intelligent driving scenario, the lane line detection method provided in the embodiment of the present application can obtain the image to be detected through the sensor on the vehicle (such as a camera), and obtain the lane line in the image to be detected, thereby realizing the above-mentioned adaptive cruise control, LDW or LKA, etc.

[0299] See also Figure 18, a flowchart of a lane line detection method provided by an embodiment of the present application, the method is applied to a vehicle, and the method may include steps 1801 to 1806. Steps 1801 to 1806 are described in detail below.

[0300] Step 1801: Acquire an image to be detected.

[0301] This step is similar to the above Figure 5 Step 501 in the illustrated embodiment is similar and will not be described again here.

[0302] For example, continuing the above example, the image to be detected is as follows Figure 6 The image to be detected is consistent with .

[0303] Step 1802: Process the image to be detected to obtain multiple point sets.

[0304] After acquiring the image to be inspected, the inspection device can process the image to obtain multiple point sets. Each of the multiple point sets represents a lane line in the image to be inspected. The point sets of the lane line in the image are predicted using the first neural network based on the transformer structure and the detection frame information. The detection frame information includes the position of the detection frame of at least one object in the image to be inspected in the image to be inspected.

[0305] It is understandable that the steps for predicting the lane line point set in the image based on the transformer-based neural network and the detection box information can refer to the aforementioned Figures 5 to 17 The description in the illustrated embodiment is similar and will not be repeated here.

[0306] Step 1803, displaying lane lines. This step is optional.

[0307] Optionally, after determining the multiple point sets, the detection device may display the lane lines represented by the multiple point sets.

[0308] For example, continuing the above example, the lane lines are as follows Figure 14b shown.

[0309] Step 1804: Model at least one object to obtain a virtual object. This step is optional.

[0310] Optionally, at least one object may be modeled to obtain a virtual object, which may be two-dimensional or multi-dimensional, and the specifics are not limited here.

[0311] Step 1805 , fusing multiple point sets with the virtual object based on positions to obtain a target image. This step is optional.

[0312] Optionally, after obtaining the multiple point sets and the virtual object, the multiple point sets and the virtual object may be fused based on the positions of the multiple point sets in the predicted image to obtain a target image.

[0313] For example, the target image is Figure 19 As shown, it can be understood that Figure 19 The virtual image in the figure is just a two-dimensional example and does not limit the virtual object.

[0314] Step 1806: display the target image. This step is optional.

[0315] Optionally, after the detection device acquires the target image, it can display the target image to the user so that the user driving the vehicle can clearly see the surrounding vehicles and lane lines, thereby improving the driving safety of the vehicle.

[0316] It is understandable that the above steps 1801 to 1806 can be executed periodically, that is, the target image can be displayed to the user in real time, so that the user can determine the surrounding objects and lane lines in real time, thereby improving the user's driving experience.

[0317] In one possible implementation, the lane detection method provided in an embodiment of the present application includes steps 1801 and 1802. In another possible implementation, the lane detection method provided in an embodiment of the present application includes steps 1801 to 1803. In another possible implementation, the lane detection method provided in an embodiment of the present application includes steps 1801 to 1805.

[0318] In the embodiments of this application, by applying the transformer architecture to lane detection, global information about the image to be detected can be acquired, effectively modeling the long-range connections between lane lines. Furthermore, by adding detection box information about objects in the image during lane detection, the target neural network's perception of the image scene can be enhanced, reducing misjudgments in scenarios where lane lines are obscured by vehicles.

[0319] The above describes the image processing method and lane line detection method provided by the embodiment of the present application. The following describes the training process of the target neural network provided by the embodiment of the present application. The training method of the target neural network can be executed by a training device for the target neural network. The training device for the target neural network can be an image processing device (such as a cloud service device or a user device, etc., which has sufficient computing power to execute the training method of the target neural network), or a system composed of a cloud service device and a user device. For example, the training method can be performed by Figure 1 Training equipment 120, Figure 2 The neural network processor 20 in is executed.

[0320] Optionally, the training method can be processed by the CPU, or by the CPU and GPU together, or other processors suitable for neural network calculations can be used without the GPU, and this application does not impose any restrictions.

[0321] See also Figure 20 , a model training method for a target neural network provided in an embodiment of the present application. The model training method includes steps 2001 to 2004.

[0322] Step 2001: Obtain a training image.

[0323] The training device can collect training images through sensors (such as cameras, radars, etc.), obtain training images from a database, or receive training images sent by other devices. The method of obtaining training images is not limited here.

[0324] When the target neural network needs to be trained, the training device can obtain a batch of training samples, that is, training images for training, wherein the actual point set of the lane lines in the training images is known.

[0325] Step 2002: Input the training image into the target neural network to obtain the first point set.

[0326] After obtaining the training image, the training image can be input into the target neural network to implement the following steps through the target neural network: obtaining a first feature of the training image; obtaining a second feature based on the first feature, the second feature including the position feature and semantic feature of the detection box corresponding to the object in the training image; obtaining a first point set based on the first feature and the second feature, the first point set being used to represent the lane lines in the training image.

[0327] Optionally, the above-mentioned method of obtaining the first point set based on the first feature and the second feature specifically includes the following steps: performing self-attention calculation on the first feature to obtain a first output; performing cross-attention calculation on the first feature and the second feature to obtain a second output; obtaining a fourth feature based on the first output and the second output; performing cross-attention calculation on the query feature and the fourth feature to obtain a third output, and the query feature is calculated by the query vector based on the self-attention mechanism; processing the query feature and the second feature to obtain a fourth output; performing addition processing on the third output and the fourth output to obtain a fifth feature; and obtaining the first point set based on the fifth feature.

[0328] For the process of obtaining the first feature, the second feature, the fourth feature, the fifth feature and the point set, please refer to the above Figure 5 The description of steps 502 to 504 in the illustrated embodiment will not be repeated here.

[0329] Step 2003: Obtain a target loss based on the first point set and a real point set of actual lane lines in the training image. The target loss is used to indicate the difference between the first point set and the real point set.

[0330] After obtaining the first point set, the first point set and the true point set can be calculated using a preset target loss function to obtain a target loss, where the target loss is used to indicate the difference between the first point set and the true point set.

[0331] It should be noted that if the number of lane lines corresponding to the first point set is greater than the number of lane lines corresponding to the true point set, the true point set can be expanded and the lane lines in the expanded point set can be set to the non-lane line category. In this case, the target loss is used to indicate the difference between the expanded true point set and the true first point set.

[0332] Step 2004: Update the parameters of the target neural network based on the target loss until the training conditions are met to obtain a trained target neural network.

[0333] After obtaining the target loss, the parameters of the target neural network can be updated based on the target loss, and the target neural network with updated parameters can be trained using the next batch of training samples (i.e., re-execute steps 2002 to 2004) until the model training conditions are met (for example, the target loss reaches convergence, etc.), and a trained target neural network can be obtained.

[0334] In addition, the query vectors involved in the training process are random. In the process of continuously updating the target neural network parameters, the query vectors are also trained to obtain the target query vector. The target query vector can be understood as the query vector used in the inference process, that is, the target query vector is Figure 5 Query vector in the illustrated embodiment.

[0335] The target neural network trained in this embodiment has the ability to predict lane lines using images. During the detection process, by applying the transformer structure to the lane line detection task, global information of the image to be detected can be obtained, thereby effectively modeling the long-range connections between lane lines. On the other hand, by adding the detection box position information of the object in the image as input to the lane line detection network, the scene perception ability of the target neural network can be improved. This reduces the model's misjudgment in scenarios where the lane line is obscured by the vehicle. On the other hand, by introducing a row and column self-attention module in the transformer encoder that can adapt to the lane line shape and mine contextual information, the network's ability to construct long lane line features can be improved, thereby achieving better lane line detection results. On the other hand, the various modules in existing autonomous driving systems are often independent of each other. For example, the lane line detection model and the human-vehicle model are independent of each other and are predicted separately. In this embodiment, the training of the target neural network is obtained by utilizing the detection box information obtained based on the human-vehicle detection model into the first neural network, which can improve the accuracy of the target neural network for lane line detection.

[0336] The image processing method in the embodiment of the present application is described above. The image processing device in the embodiment of the present application is described below. Figure 21 , an embodiment of the image processing device in the embodiment of the present application includes:

[0337] An extraction unit 2101 is configured to extract features from the image to be detected to obtain a first feature;

[0338] A processing unit 2102 is configured to process the detection frame information of the image to be detected to obtain a second feature, where the detection frame information includes a position of the detection frame of the object in the image to be detected in the image to be detected;

[0339] The determination unit 2103 is used to input the first feature and the second feature into a first neural network based on a transformer structure to obtain the lane line in the image to be detected.

[0340] Optionally, the image processing device in this embodiment may further include: an acquisition unit 2104, used to acquire a first row feature and a first column feature based on the first feature, the first row feature is obtained by flattening the matrix corresponding to the first feature along the row direction, and the first column feature is obtained by flattening the matrix along the column direction.

[0341] In this embodiment, the operations performed by each unit in the image processing device are the same as those described above. Figures 5 to 17 The description in the illustrated embodiment is similar and will not be repeated here.

[0342] In this embodiment, by applying the transformer architecture to lane detection, global information about the image being detected can be captured, effectively modeling the long-range connections between lane lines. Furthermore, by adding detection box information about objects in the image during lane detection, scene perception is enhanced, reducing misjudgments caused by lane lines being obscured by vehicles.

[0343] See also Figure 22 , an embodiment of the detection device in the embodiment of the present application includes:

[0344] An acquisition unit 2201 is used to acquire an image to be detected;

[0345] A processing unit 2202 is configured to process the image to be detected to obtain multiple point sets, each of which represents a lane line in the image to be detected; wherein the point set of the lane line in the image is predicted by a first neural network based on a transformer structure and detection frame information, and the detection frame information includes the position of a detection frame of at least one object in the image to be detected in the image to be detected.

[0346] Optionally, the detection device in this embodiment may further include: a display unit 2203, configured to display lane lines.

[0347] In this embodiment, the operations performed by each unit in the detection device are the same as those described above. Figure 18 The description in the illustrated embodiment is similar and will not be repeated here.

[0348] In this embodiment, by applying the transformer architecture to the lane detection task, global information about the image being detected can be acquired, effectively modeling the long-range connections between lane lines. Furthermore, by adding detection bounding boxes to the objects in the image during lane detection, the target neural network's perception of the image scene can be enhanced, reducing misjudgments caused by lane lines being obscured by vehicles.

[0349] See also Figure 23 Another embodiment of the image processing device in the embodiment of the present application includes:

[0350] An acquisition unit 2301 is used to acquire a training image;

[0351] Processing unit 2302 is configured to input a training image into a target neural network to obtain a first point set of the training image, where the first point set represents predicted lane lines in the training image. The target neural network is configured to: extract features from the training image to obtain a first feature; process detection frame information of the training image to obtain a second feature, where the detection frame information includes the position of the detection frame of an object in the training image in the training image; obtain a first point set based on the first feature and the second feature, and the target neural network is configured to predict the point set of lane lines in the image based on a transformer structure.

[0352] The training unit 2303 is used to train the target neural network according to the first point set and the real point set of the actual lane line in the training image to obtain a trained target neural network.

[0353] In this embodiment, the operations performed by each unit in the image processing device are the same as those described above. Figure 20 The description in the illustrated embodiment is similar and will not be repeated here.

[0354] In this embodiment, by applying the transformer architecture to the lane detection task, global information about the image being detected can be acquired, effectively modeling the long-range connections between lane lines. Furthermore, by adding detection bounding boxes to the objects in the image during lane detection, the target neural network's perception of the image scene can be enhanced, reducing misjudgments caused by lane lines being obscured by vehicles.

[0355] See Figure 24 , a schematic diagram of the structure of another image processing device provided in this application. The image processing device may include a processor 2401, a memory 2402, and a communication interface 2403. The processor 2401, the memory 2402, and the communication interface 2403 are interconnected via a circuit. The memory 2402 stores program instructions and data.

[0356] The memory 2402 stores the aforementioned Figures 5 to 17 、 Figure 20 In the corresponding embodiment shown, the steps executed by the device correspond to program instructions and data.

[0357] Processor 2401, configured to execute the aforementioned Figures 5 to 17 、 Figure 20 The steps shown in any of the illustrated embodiments are performed by the apparatus.

[0358] The communication interface 2403 can be used to receive and send data, and to perform the aforementioned Figures 5 to 17 、 Figure 20 The steps related to obtaining, sending, and receiving in any of the embodiments shown.

[0359] In one implementation, the image processing device may include a Figure 24 More or fewer components are merely illustrative in this application and are not limiting.

[0360] See Figure 25 , a schematic diagram of the structure of another detection device provided in this application. The detection device may include a processor 2501, a memory 2502, and a communication interface 2503. The processor 2501, the memory 2502, and the communication interface 2503 are interconnected via a circuit. The memory 2502 stores program instructions and data.

[0361] The memory 2502 stores the aforementioned Figure 18 In the corresponding embodiment shown, the steps executed by the detection device correspond to program instructions and data.

[0362] Processor 2501, configured to execute the aforementioned Figure 18 The steps performed by the detection device in any of the illustrated embodiments.

[0363] The communication interface 2503 can be used to receive and send data, and to perform the aforementioned Figure 18 The steps related to obtaining, sending, and receiving in any of the embodiments shown.

[0364] In one implementation, the detection device may include a Figure 25 More or fewer components are merely illustrative in this application and are not limiting.

[0365] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0366] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0367] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in whole or in part through software, hardware, firmware, or any combination thereof.

[0368] When software is used to implement the integrated unit, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, hard disk, tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0369] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

Claims

1. An image processing method, characterized in that: The method comprises: Perform feature extraction on the image to be detected to obtain the first feature; performing feature extraction on the detection frame information of the image to be detected to obtain a second feature, wherein the detection frame information includes a position of a detection frame of at least one object in the image to be detected in the image to be detected; the second feature is a detection frame feature of the image to be detected; The first feature and the second feature are input into a first neural network based on a transformer structure to obtain the lane lines in the image to be detected.

2. The method according to claim 1, characterized in that The extracting the feature of the detection frame information of the image to be detected to obtain the second feature includes: Feature extraction is performed on at least one third feature and the detection frame information to obtain the second feature, where the at least one third feature is an intermediate feature obtained in the process of obtaining the first feature.

3. The method according to claim 2, characterized in that The second feature includes a position feature and a semantic feature of the detection frame, and the detection frame information further includes: a category and a confidence level of the detection frame; The processing of at least one third feature and the detection frame information to obtain the second feature includes: Acquire the semantic feature based on the at least one third feature, the position, and the confidence; The location feature is obtained based on the location and the category.

4. The method according to claim 3, characterized in that The acquiring the semantic feature based on the at least one third feature, the position, and the confidence level includes: extracting a region of interest (ROI) feature from the at least one third feature based on the position; Multiplying the ROI feature by the confidence level, and inputting the obtained feature into a fully connected layer to obtain the semantic feature; The acquiring the location feature based on the location and the category includes: The vector of the category is obtained and concatenated with the vector corresponding to the position, and the concatenated feature is input into a fully connected layer to obtain the position feature.

5. The method according to any one of claims 1 to 4, characterized in that The first neural network based on the transformer structure includes an encoder, a decoder and a feedforward neural network; Inputting the first feature and the second feature into a first neural network based on a transformer structure to obtain lane lines in the image to be detected includes: Inputting the first feature and the second feature into the encoder to obtain a fourth feature; Inputting the fourth feature, the second feature, and the query feature into the decoder to obtain a fifth feature; The fifth feature is input into the feedforward neural network to obtain multiple point sets, each of the multiple point sets represents a lane line in the image to be detected.

6. The method according to claim 5, characterized in that The method further comprises: Acquire a first row feature and a first column feature based on the first feature, where the first row feature is obtained by flattening the matrix corresponding to the first feature along the row direction, and the first column feature is obtained by flattening the matrix along the column direction; Inputting the first feature and the second feature into the encoder to obtain a fourth feature includes: The first feature, the second feature, the first row feature, and the first column feature are input into the encoder to obtain the fourth feature.

7. The method according to claim 6, characterized in that The step of inputting the first feature, the second feature, the first row feature, and the first column feature into the encoder to obtain the fourth feature includes: Performing self-attention calculation on the first feature to obtain a first output; Performing a cross-attention calculation on the first feature and the second feature to obtain a second output; Performing self-attention calculation and concatenation on the first row features and the first column features to obtain row and column outputs; The fourth feature is obtained based on the first output, the second output, and the row and column outputs.

8. The method according to claim 7, characterized in that The obtaining the fourth feature based on the first output, the second output, and the row and column outputs includes: Adding the first output and the second output to obtain a fifth output; The fifth output and the row and column outputs are concatenated to obtain the fourth feature.

9. The method according to claim 5, characterized in that Inputting the first feature and the second feature into the encoder to obtain a fourth feature includes: Performing self-attention calculation on the first feature to obtain a first output; Performing a cross-attention calculation on the first feature and the second feature to obtain a second output; The first output and the second output are added to obtain the fourth feature.

10. The method according to claim 5, characterized in that The step of inputting the fourth feature, the second feature, and the query feature into the decoder to obtain a fifth feature includes: performing a cross attention calculation on the query feature and the fourth feature to obtain a third output; Processing the query feature and the second feature to obtain a fourth output; The third output and the fourth output are added together to obtain the fifth feature.

11. The method according to any one of claims 1 to 4, characterized in that The feature extraction of the image to be detected to obtain the first feature includes: The features output by different layers in the backbone network are subjected to feature fusion and dimensionality reduction processing to obtain the first feature, where the input of the backbone network is the image to be detected.

12. A lane line detection method, characterized in that: The method is applied to a vehicle, and comprises: Obtain the image to be detected; The image to be detected is processed to obtain multiple point sets, each of the multiple point sets represents a lane line in the image to be detected; wherein the processing is based on a first neural network of a transformer structure and detection frame information to predict the point set of the lane line in the image, and the detection frame information includes the position of the detection frame of at least one object in the image to be detected in the image to be detected.

13. The method according to claim 12, characterized in that The detection frame information also includes: the category and confidence of the detection frame.

14. The method according to claim 12 or 13, characterized in that The method further comprises: The lane markings are displayed.

15. The method according to claim 12 or 13, characterized in that The method further comprises: Modeling the at least one object to obtain a virtual object; fusing the plurality of point sets with the virtual object based on the positions to obtain a target image; The target image is displayed.

16. An image processing device, characterized in that: The image processing device comprises: An extraction unit, configured to extract features from the image to be detected to obtain a first feature; a processing unit, configured to perform feature extraction on the detection frame information of the image to be detected to obtain a second feature, wherein the detection frame information includes a position of a detection frame of at least one object in the image to be detected in the image to be detected; and the second feature is a detection frame feature of the image to be detected; A determination unit is used to input the first feature and the second feature into a first neural network based on a transformer structure to obtain the lane line in the image to be detected.

17. The image processing device according to claim 16, wherein The processing unit is specifically used to perform feature extraction on at least one third feature and the detection frame information to obtain the second feature, and the at least one third feature is an intermediate feature obtained in the process of obtaining the first feature.

18. The image processing device according to claim 17, wherein The second feature includes a position feature and a semantic feature of the detection frame, and the detection frame information further includes: a category and a confidence level of the detection frame; The processing unit is specifically configured to obtain the semantic feature based on the at least one third feature, the position, and the confidence level; The processing unit is specifically configured to obtain the location feature based on the location and the category.

19. The image processing device according to claim 18, wherein The processing unit is specifically configured to extract a region of interest (ROI) feature from the at least one third feature based on the position; The processing unit is specifically configured to perform multiplication processing on the ROI feature and the confidence, and input the obtained feature into a fully connected layer to obtain the semantic feature; The processing unit is specifically configured to obtain the vector of the category, concatenate it with the vector corresponding to the position, and input the concatenated feature into a fully connected layer to obtain the position feature.

20. The image processing apparatus according to any one of claims 16 to 19, characterized in that: The first neural network based on the transformer structure includes an encoder, a decoder and a feedforward neural network; The determining unit is specifically configured to input the first feature and the second feature into the encoder to obtain a fourth feature; The determining unit is specifically configured to input the fourth feature, the second feature, and the query feature into the decoder to obtain a fifth feature; The determining unit is specifically configured to input the fifth feature into the feedforward neural network to obtain a plurality of point sets, each of the plurality of point sets representing a lane line in the image to be detected.

21. The image processing device according to claim 20, wherein The image processing device further includes: an acquiring unit, configured to acquire a first row feature and a first column feature based on the first feature, wherein the first row feature is obtained by flattening a matrix corresponding to the first feature along a row direction, and the first column feature is obtained by flattening the matrix along a column direction; The determining unit is specifically configured to input the first feature, the second feature, the first row feature, and the first column feature into the encoder to obtain the fourth feature.

22. The image processing device according to claim 21, wherein The determining unit is specifically configured to perform self-attention calculation on the first feature to obtain a first output; The determining unit is specifically configured to perform a cross-attention calculation on the first feature and the second feature to obtain a second output; The determining unit is specifically configured to perform self-attention calculation and splicing processing on the first row feature and the first column feature to obtain row and column outputs; The determining unit is specifically configured to obtain the fourth feature based on the first output, the second output, and the row and column outputs.

23. The image processing device according to claim 22, wherein The determining unit is specifically configured to add the first output and the second output to obtain a fifth output; The determining unit is specifically configured to perform splicing processing on the fifth output and the row and column outputs to obtain the fourth feature.

24. The image processing apparatus according to claim 20, wherein The determining unit is specifically configured to perform self-attention calculation on the first feature to obtain a first output; The determining unit is specifically configured to perform a cross-attention calculation on the first feature and the second feature to obtain a second output; The determining unit is specifically configured to perform addition processing on the first output and the second output to obtain the fourth feature.

25. The image processing apparatus according to claim 20, wherein The determining unit is specifically configured to perform a cross-attention calculation on the query feature and the fourth feature to obtain a third output; The determining unit is specifically configured to process the query feature and the second feature to obtain a fourth output; The determining unit is specifically configured to perform an addition process on the third output and the fourth output to obtain the fifth feature.

26. The image processing apparatus according to any one of claims 16 to 19, characterized in that: The extraction unit is specifically used to perform feature fusion and dimensionality reduction processing on the features output by different layers in the backbone network to obtain the first feature, and the input of the backbone network is the image to be detected.

27. A detection device, characterized in that: The detection device is applied to a vehicle, and the detection device includes: An acquisition unit, configured to acquire an image to be detected; A processing unit is configured to process the image to be detected to obtain a plurality of point sets, each of the plurality of point sets representing a lane line in the image to be detected; wherein the processing is based on a first neural network of a transformer structure and detection frame information to predict the point set of the lane line in the image, and the detection frame information includes a position of a detection frame of at least one object in the image to be detected in the image to be detected.

28. The detection device according to claim 27, characterized in that The detection frame information also includes: the category and confidence of the detection frame.

29. The detection device according to claim 27 or 28, characterized in that The detection device also includes: A display unit is used to display the lane line.

30. The detection device according to claim 29, characterized in that The processing unit is further configured to model the at least one object to obtain a virtual object; The processing unit is further configured to fuse the plurality of point sets with the virtual object based on the positions to obtain a target image; The display unit is further configured to display the target image.

31. An image processing device, characterized in that include: A processor is coupled to a memory, wherein the memory is used to store programs or instructions, and when the programs or instructions are executed by the processor, the image processing device performs the method according to any one of claims 1 to 11.

32. A detection device, characterized in that: The detection device is applied to a vehicle, and includes: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, and when the program or instructions are executed by the processor, the detection device executes the method as described in any one of claims 12 to 15.

33. A computer storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, causes the electronic device to execute the method according to any one of claims 1 to 11, or causes the electronic device to execute the method according to any one of claims 12 to 15.

34. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to perform the method according to any one of claims 1 to 11, or the computer is enabled to perform the method according to any one of claims 12 to 15.

Images