Image prediction method, apparatus and electronic device

Abstract

The application discloses an image prediction method and device and electronic equipment. The method comprises the following steps: obtaining a target image; and predicting the target image by using a target neural network to obtain a target prediction result, wherein the target prediction result is used for indicating whether a lane line exists in the target image, and in the case that the lane line exists in the target image, the type of the lane line contained in the target image; and the target neural network comprises multiple groups of target convolution layers, and the number of channels contained in each group of convolution layers in the multiple groups of target convolution layers is the same. The application solves the technical problems of redundancy and time consumption of a lane line detection process in the field of visual perception of automatic driving in the prior art.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicles, and more specifically, to an image prediction method, apparatus, and electronic device. Background Technology

[0002] In autonomous vehicles, lane detection is an important functional module in the autonomous driving system. By collecting information about the surrounding environment through the perception module, and in combination with the map module but not entirely relying on high-precision maps, the lane detection results are also important input information for vehicle motion planning, enabling autonomous vehicles to navigate smoothly in various complex road scenarios.

[0003] In the current perception modules of autonomous driving, three methods can be used to perceive and detect lane lines: semantic segmentation-based methods, anchor-based methods, and curve fitting formula methods. However, semantic segmentation-based methods, such as SCNN (Spatial Convolutional Neural Networks), often require a huge backbone network, resulting in slow detection speeds. Anchor-based methods and curve fitting formulas are highly dependent on prior information, and changes in parameters such as camera pitch angle often lead to a rapid decline in detection performance.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides an image prediction method, apparatus, and electronic device to at least solve the technical problems of redundant and time-consuming lane detection processes in the field of visual perception for autonomous driving in the prior art.

[0006] According to one aspect of the present invention, an image prediction method is provided, comprising: acquiring a target image; predicting the target image using a target neural network to obtain a target prediction result, wherein the target prediction result is used to indicate whether lane lines exist in the target image, and, if lane lines exist in the target image, the type of lane lines contained in the target image; wherein the target neural network includes multiple sets of target convolutional layers, and each set of convolutional layers contains the same number of channels.

[0007] Optionally, the target neural network further includes: a first sampling branch and a second sampling branch, wherein predicting the target image using the target neural network to obtain a target prediction result includes: extracting features from the target image using multiple sets of target convolutional layers to obtain a target feature vector; predicting the target feature vector using the first sampling branch and the second sampling branch respectively to obtain a first prediction result and a second prediction result, wherein the first prediction result is used to indicate whether the target image contains lane lines, and the second prediction result is used to indicate the type of lane lines contained in the target image when lane lines are contained; and obtaining a target prediction result based on the first prediction result and the second prediction result.

[0008] Optionally, the method further includes: receiving a preset total number of channels; grouping the target convolutional layer based on the preset total number of channels to obtain multiple groups of target convolutional layers.

[0009] Optionally, the second sampling branch includes: a first sub-sampling layer and multiple second sub-sampling layers. The second sampling branch is used to predict the target feature vector to obtain a second prediction result, including: using the first sub-sampling layer to predict the target feature vector to obtain a first sub-prediction result; using multiple second sub-sampling layers to perform deconvolution upsampling on the target feature vector features to obtain multiple second sub-prediction results, wherein the product of the sampling factors of the multiple second sub-sampling layers is the same as the sampling factor of the first sub-sampling layer; and fusing the first sub-prediction result and the multiple second prediction results to obtain the second prediction result.

[0010] Optionally, the method further includes: obtaining a sample image and a corresponding first label and second label, wherein the first label indicates whether the sample image contains lane lines, and the second label indicates the type of lane lines contained in the sample image when lane lines are present; predicting the sample image using an initial neural network to obtain a third prediction result and a fourth prediction result, wherein the third prediction result indicates whether the sample image contains lane lines, and the fourth prediction result indicates the type of lane lines contained in the sample image when lane lines are present; constructing a target loss function based on the first label, the second label, the third prediction result, and the fourth prediction result; and adjusting the network parameters of the initial neural network based on the target loss function to obtain a target neural network.

[0011] Optionally, based on the first label, the second label, the third prediction result, and the fourth prediction result, a target loss function is constructed, including: determining the cross-entropy loss function based on the first label and the third prediction result; determining the similarity measurement function based on the second label and the fourth prediction result; and constructing the target loss function based on the similarity measurement function and the cross-entropy loss function.

[0012] According to another aspect of the present invention, an image prediction apparatus is also provided, comprising: an acquisition module for acquiring a target image; and a prediction module for predicting the target image using a target neural network to obtain a target prediction result, wherein the target prediction result is used to indicate whether lane lines exist in the target image, and, if lane lines exist in the target image, the type of lane lines contained in the target image; wherein the target neural network includes multiple sets of target convolutional layers, and each set of convolutional layers contains the same number of channels.

[0013] Optionally, the target neural network further includes: a first sampling branch and a second sampling branch, and a prediction module, including: a feature extraction unit, used to extract features from the target image using multiple sets of target convolutional layers to obtain a target feature vector; a prediction unit, used to predict the target feature vector using the first sampling branch and the second sampling branch respectively to obtain a first prediction result and a second prediction result, wherein the first prediction result is used to indicate whether the target image contains lane lines, and the second prediction result is used to indicate the type of lane lines contained in the target image when lane lines are contained in the target image; and a determination unit, used to obtain a target prediction result based on the first prediction result and the second prediction result.

[0014] According to another aspect of the present invention, an electronic device is also provided, including one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors perform the image prediction method of any one of the above embodiments of the invention.

[0015] According to another aspect of the present invention, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is running, it controls the processor of the device to execute any of the image prediction methods described above.

[0016] In this embodiment of the invention, during lane detection in an autonomous driving system, a monocular camera on the autonomous vehicle is used to acquire monocular 2D lane line images along with their mask and exist labels. These images are then batch-input into a designed variable-group convolutional neural network. After prediction by the neural network model, lane line prediction results are output. These prediction results include whether the target image contains lane lines. If lane lines are present, the relative position information and type of the lane lines are output. It is noteworthy that this image prediction method is based on image segmentation, transforming lane line detection into lane line image feature detection. Multi-scale feature extraction is performed using a convolutional neural network. Each convolutional layer in the multiple target convolutional layers of this neural network contains the same number of channels. Therefore, the network model has a more consistent computational pattern and data layout, making it more suitable for optimization and reducing time consumption during detection. This allows for more accurate and rapid determination of the relative position and type of lane lines in different scenarios, effectively improving the speed and accuracy of lane line detection. This solves the technical problem of redundant and time-consuming lane line detection processes in the field of visual perception for autonomous driving in existing technologies. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0018] Figure 1 This is a flowchart of an image prediction method according to an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of an optional lane line image prediction method for autonomous driving according to an embodiment of the present invention;

[0020] Figure 3 This is a flowchart of an optional lightweight variable-group convolutional network design according to an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of an image prediction device according to an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of a prediction module of an image prediction device according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] Example 1

[0026] According to an embodiment of the present invention, an image prediction method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] Figure 1 This is a flowchart of an image prediction method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0028] Step S102: Obtain the target image;

[0029] Step S104: Use the target neural network to predict the target image and obtain the target prediction result. The target prediction result is used to indicate whether there are lane lines in the target image and, if there are lane lines in the target image, the type of lane lines contained in the target image.

[0030] The target neural network includes multiple sets of target convolutional layers, and each set of target convolutional layers contains the same number of channels.

[0031] It should be noted that the image prediction method can be a lane detection method based on a variable-group convolutional neural network. A convolutional neural network is a feedforward neural network with a deep structure that includes convolutional computation. It converts an image into matrix data in a two-dimensional matrix format, extracts features from the matrix data, and outputs prediction results. After inputting a large training set, it achieves the technical effect of accurately and efficiently outputting prediction results for input images. Therefore, convolutional neural networks have a wide range of applications in image recognition and classification. Image processing is performed before the image is input to the neural network because, during lane detection in autonomous driving systems, lane lines in the same scene may exhibit changes in position, angle rotation, and scale. Therefore, the convolutional neural network must be able to handle these variations and complete the desired classification and output the same results. Thus, the input image is first processed to a certain extent by deformation and rotation to enhance the dataset and improve the stability of the network model. For lane detection in autonomous driving, a lightweight variable-group convolutional neural network with small Madds (representing a cumulative multiplication-addition operation) is designed based on this convolutional neural network. This network performs multi-scale feature extraction on the image to be detected and ensures that each convolutional layer in multiple target convolutional layers contains the same number of channels. This results in a more consistent computational pattern and data layout, thus reducing time consumption during detection and making it easier to optimize. By constructing a corresponding loss function based on the input labels and prediction results to train the network, accurate and efficient acquisition of lane line relative position information and types in different scenarios can be achieved.

[0032] The target image can be a monocular 2D lane line image, which is an image captured using the monocular camera on the autonomous vehicle. Monocular cameras are primarily used for road condition data acquisition. The input target image should include a lane line existence label and a lane line type mask label. The lane line existence label indicates whether the lane line exists. When a lane line exists, segmentation is performed based on information such as the size and position of different lane lines in the target image. The target image mask is mainly used to extract the matrix data of the effective region, i.e., the lane lines, and set the invalid region to 0, i.e., mask the data of this region and do not process it. Structural features similar to the lane lines are extracted from the mask to distinguish the lane line types. Therefore, the lane line type mask label can be used to indicate the type of lane line that exists. The target prediction result can be the prediction result output after the target image is predicted by this variable group convolutional neural network. The prediction result includes the lane lines and their types relative to the vehicle's position. The target neural network can be a designed variable group convolutional neural network that has been trained extensively and adjusted with a large number of target image inputs, and can accurately and quickly output the relative position and type of lane lines in different scenarios based on the input image.

[0033] In one optional embodiment, the autonomous vehicle, in the operating scenario, uses a monocular camera to acquire lane line images, along with lane line existence labels and lane line type mask labels, and inputs them in batches into a designed variable-group convolutional neural network. This neural network includes multiple sets of target convolutional layers, each containing the same number of channels. Based on the consistent computational pattern and data layout of this neural network, it can output lane line prediction results more accurately and efficiently. These prediction results include whether the target image contains lane lines; if so, the relative position information and type of the lane lines are output.

[0034] In this embodiment of the invention, during lane detection in an autonomous driving system, a monocular camera on the autonomous vehicle is used to acquire monocular 2D lane line images along with their mask and exist labels. These images are then batch-input into a designed variable-group convolutional neural network. After prediction by the neural network model, lane line prediction results are output. These prediction results include whether the target image contains lane lines. If lane lines are present, the relative position information and type of the lane lines are output. It is noteworthy that this image prediction method is based on image segmentation, transforming lane line detection into lane line image feature detection. Multi-scale feature extraction is performed using a convolutional neural network. Each convolutional layer in the multiple target convolutional layers of this neural network contains the same number of channels. Therefore, the network model has a more consistent computational pattern and data layout, making it more suitable for optimization and reducing time consumption during detection. This allows for more accurate and rapid determination of the relative position and type of lane lines in different scenarios, effectively improving the speed and accuracy of lane line detection. This solves the technical problem of redundant and time-consuming lane line detection processes in the field of visual perception for autonomous driving in existing technologies.

[0035] Optionally, according to the above embodiments of the invention, the target neural network further includes: a first sampling branch and a second sampling branch, wherein predicting the target image using the target neural network to obtain a target prediction result includes: extracting features from the target image using multiple sets of target convolutional layers to obtain a target feature vector; predicting the target feature vector using the first sampling branch and the second sampling branch respectively to obtain a first prediction result and a second prediction result, wherein the first prediction result is used to indicate whether the target image contains lane lines, and the second prediction result is used to indicate the type of lane lines contained in the target image when lane lines are contained in the target image; and obtaining a target prediction result based on the first prediction result and the second prediction result.

[0036] The target feature vector can be a feature vector obtained by extracting features from the target image using multiple sets of target convolutional layers. During target feature vector extraction, image information and its label information, after random deformation, resizing, and pixel normalization, are loaded into the training model. A lightweight variable-group convolutional neural network is used as the backbone to extract features at multiple sizes. The extracted feature vector can be divided into different sampling branches based on its different size features. The first sampling branch can be the lane line presence Exist branch, which predicts the target feature vector using the sigmoid function. The first prediction result can represent the probability of lane lines existing in the target image. The second sampling branch can be the image segmentation Seg branch, which restores the detection image to its original resolution using a designed "bilinear + ConvTranspose2d" combined upsampling module and predicts the target feature vector. The second prediction result can represent the type of lane line if the target image contains lane lines. Lane line types include road center lines, variable lane lines, white lane lines, bus lanes, etc.

[0037] In one alternative embodiment, Figure 2 This is a schematic diagram of an optional lane line image prediction method for autonomous driving according to an embodiment of the present invention, as shown below. Figure 2 As shown, the batch input includes labeled images, specifically monocular 2D lane line images along with their mask and exist labels. Preprocessing is performed on the model. First, data augmentation is applied to the original images by random rotations of -2 to 2 degrees, random horizontal flips, random Gaussian noise, and random brightness variations. This simulates target images of the same lane line acquired under various conditions, such as strong light, occlusion, and different angles. The target images are then resized using the Resize function, adjusting the original image size to (640, 480, 3) to unify the image matrix size. Finally, the pixel values ​​of the target images are normalized by dividing the pixel values ​​from 0 to 255 by 255 to obtain values ​​between 0 and 1. The processed image information and its label information are loaded into the training model. A lightweight, variable-group convolutional neural network is used as the backbone for multi-scale feature extraction. The extracted feature vectors are then used to predict lane presence using the Exist branch. This prediction method uses the sigmoid (1,1,1,0) activation function to obtain a first prediction result indicating whether lane lines exist in the image. The extracted feature vectors are then used to predict the type of lane lines present in the image using the Seg branch. Based on these two lane line prediction results, the system determines whether lane lines exist in the scene and what type of lane lines they are, ultimately outputting the lane lines relative to the vehicle's position.

[0038] Optionally, according to the above embodiments of the invention, the method further includes: receiving a preset total number of channels; grouping the target convolutional layers based on the preset total number of channels to obtain multiple groups of target convolutional layers.

[0039] The preset receiving channel can be the backbone of a lightweight variable-group convolutional neural network, representing the channel that can receive target feature vectors for prediction. Multiple target convolutional layers divide all channels of the convolutional layer into several groups, where the number of input channels in each group is fixed. Therefore, the total number of groups in each grouped convolutional layer varies with the total number of input channels. For example, if the number of input channels in each group is fixed at 8, then the number of groups is the total number of input channels divided by 8. Because this method has a more consistent computational pattern and data layout, a fixed number of channels is more suitable for optimization, allowing for a smaller number of channels and reducing time consumption during detection. At the encoding end, a lightweight variable-group convolutional neural network with small Madds is designed to downsample the image to be detected, reducing its resolution to decrease the encoding burden. The network aggregates spatial features from different directions and is friendly to low-power platforms.

[0040] In one alternative embodiment, Figure 3 This is a flowchart of a lightweight variable-group convolutional network of an optional design according to an embodiment of the present invention, such as... Figure 3 As shown, the processed target image information and its label information are loaded into the training model. The input function is Input(640,480,3), and the output function is Output(80,60,256). A lightweight variable-group convolutional neural network is designed as the backbone for multi-size feature extraction. All channels of the convolutional layer are divided into several groups, and the number of input channels in each group is fixed at 8. Therefore, the total number of groups in each grouped convolutional layer is the total number of input channels divided by 8. In this embodiment of the invention, the preset total number of channels to be received is specified as (32,64,128,256). The lightweight variable-group convolutional neural network of the backbone network is as follows: Figure 3 As shown, the channel statements are Conv3*3_s2_BN_Relu(320,240,32), Conv3*3_BN_Relu(160,120,64), Conv3*3_BN_Relu(80,60,128), and Conv3*3_BN_Relu(80,60,256). For example, GroupConv3*3_s1_g8(80,60,64)*4 represents a 4-layer grouped convolutional layer with a 3*3 kernel, a stride of 2, 8 groups, an input image resolution of 80*60, and 64 channels.

[0041] It should be noted that, as Figure 3As shown, the implementation functions of the lightweight variable-group convolutional neural network of this design are as follows: Input(640,480,3), Conv3*3_s2_BN_Relu(320,240,32), GroupConv3*3_s2_g4(160,120,32), GroupConv3*3_s1_BN_g4(160,120,32)*4, Conv3*3_BN_Relu(160,120,64), GroupConv3*3_s2_g8(80,60,64), GroupConv3*3_s1 _g8(80,60,64)*4, Conv3*3_BN_Relu(80,60,128), GroupConv3*3_s1_g16(80,60,128), GroupConv3*3_s1_g16(80,60,128)*4, C onv3*3_BN_Relu(80,60,256), GroupConv3*3_s1_g32(80,60,256), GroupConv3*3_s1_g32(80,60,256)*4, Output(80,60,256).

[0042] Optionally, according to the above embodiments of the invention, the second sampling branch includes: a first sub-sampling layer and multiple second sub-sampling layers. Predicting the target feature vector using the second sampling branch to obtain a second prediction result includes: predicting the target feature vector using the first sub-sampling layer to obtain a first sub-prediction result; performing deconvolution upsampling on the target feature vector features using the multiple second sub-sampling layers to obtain multiple second sub-prediction results, wherein the product of the sampling factors of the multiple second sub-sampling layers is the same as the sampling factor of the first sub-sampling layer; and fusing the first sub-prediction result and the multiple second prediction results to obtain the second prediction result.

[0043] The second sampling branch can be the image segmentation Seg branch. This branch predicts the input feature vector, and the prediction result is the type of lane lines present in the target image. This channel is used at the decoding end, and the detected image is restored to its original resolution through a designed "bilinear + ConvTranspose2d" combined upsampling module. Therefore, the first sub-sampling layer of this branch can be 8x bilinear upsampling, and the second sub-sampling layer can be three 2x deconvolution upsampling layers. The first prediction result can be the prediction result obtained through the 8x bilinear upsampling layer, achieving initial image magnification. The second prediction result can be the prediction result obtained through the three 2x deconvolution upsampling layers, achieving image resolution restoration. The second prediction result can be the original resolution of the detected image restored through the designed "bilinear + ConvTranspose2d" combined upsampling module. This method enables better fusion and restoration of features of different sizes.

[0044] It's important to note that the bilinear function, or bilinear interpolation function, is a commonly used sampling method. This method uses the gray values ​​of the four nearest neighbors of the mapped point in the original image as reference points to interpolate the mapped point. In other words, the value at the interpolated point is obtained by weighting the values ​​of the four nearest points to the original point. The weights are linearly negatively correlated with the distance between the reference point and the mapped point. This method is used to enlarge the image. The ConvTranspose2d function, also known as the deconvolution function, is a commonly used method for upsampling images. It uses a new convolution kernel to enlarge the image size. Essentially, it is also a convolution operation, aiming to restore the original image's size before the convolution operation.

[0045] In one alternative embodiment, such as Figure 2 As shown, post-processing of the model is performed. A lightweight variable-group convolutional neural network is used to extract features from the target image, confirming the presence of lane lines. The image segmentation Seg branch predicts the input feature vector, extracts points representing lane line information at different locations, and restores the compressed image. First, the image is enlarged by bilinear interpolation using a bilinear function, yielding the first sub-prediction result. Based on this, a newly added ConvTranspose2d function, consisting of three 2x deconvolutional upsampling layers, further restores the image, yielding the second sub-prediction result. The upsampling factor of the Bilinear function is the same as the corresponding sampling factor of the deconvolutional sampling layer, which is 8x. The designed "bilinear + ConvTranspose2d" combined upsampling module allows for better fusion of features of different sizes to restore the decoupled image, thus determining the type of lane line as the prediction result of this branch.

[0046] Optionally, according to the above embodiments of the invention, the method further includes: obtaining a sample image and a first label and a second label corresponding to the sample image, wherein the first label is used to indicate whether the sample image contains lane lines, and the second label is used to indicate the type of lane lines contained in the sample image when the sample image contains lane lines; predicting the sample image using an initial neural network to obtain a third prediction result and a fourth prediction result, wherein the third prediction result is used to indicate whether the sample image contains lane lines, and the fourth prediction result is used to indicate the type of lane lines contained in the sample image when the sample image contains lane lines; constructing a target loss function based on the first label, the second label, the third prediction result, and the fourth prediction result; and adjusting the network parameters of the initial neural network based on the target loss function to obtain a target neural network.

[0047] The sample images can be lane line images input into a batch-designed variable-group convolutional neural network. The initial neural network can be a variable-group convolutional neural network initially established according to the lane line detection requirements. The first label can be a lane line existence label, representing the probability of the lane line's existence. It should be noted that the lane line existence label is predicted by the lane line existence Exist branch through the sigmoid function. The sigmoid, also known as the S-shaped growth curve, is often used as an activation function for neural networks, mapping variables to between 0 and 1. Therefore, the value of this curve can be associated with the probability. The second label can be a lane line type mask label, used to indicate the type of lane line when the sample image contains lane lines. Based on the idea of ​​image segmentation, different lane lines are usually visualized as individual mask labels. The setting of this label is based on the regression of different lane line sizes, positions, and other information by the lane line detection algorithm. The third prediction result can be the predicted probability of lane line existence after using the initial neural network to predict the sample image. The fourth prediction result can be the predicted type of existing lane line after using the initial neural network to predict the sample image. The target loss function can be a function model used in the training phase, trained on batches of labeled data, to differentially adjust the initial target network. The target neural network can be a more accurate and faster neural network model obtained by adjusting the network parameters of the initial neural network using the target loss function.

[0048] In one alternative embodiment, such as Figure 2As shown, a batch of input sample images and their lane line existence labels (exist) and lane line type mask labels are provided. An initial neural network is used to predict the sample images. A third prediction result is obtained through the "Exist" branch, representing the probability of a lane line existing in the sample image. A fourth prediction result is obtained through the image segmentation "Seg" branch, representing the type of lane line present. Based on the lane line existence label, the probability of lane line existence in the third prediction result, the lane line type mask label, and the lane line type in the fourth prediction result, a lane line detection loss function is established. This loss function is then used to train the batch of labeled data. During training, the gradient is dynamically backpropagated according to the changes in the loss function to change the training parameters. The training parameters are then used in forward computation to reduce the loss function. During this dynamic process, the loss function gradually decreases until it reaches a convergent state. The model parameters saved at this point constitute the optimal variable-group convolutional neural network.

[0049] Optionally, according to the above embodiments of the invention, constructing a target loss function based on a first label, a second label, a third prediction result, and a fourth prediction result includes: determining a cross-entropy loss function based on the first label and the third prediction result; determining a similarity measurement function based on the second label and the fourth prediction result; and constructing a target loss function based on the similarity measurement function and the cross-entropy loss function.

[0050] The cross-entropy loss function can be the BCEWithLogitsLoss function, which normalizes the input samples for each corresponding classification output, calculates the loss value for each sample, assigns different weights to different samples, and performs a weighted calculation to obtain the overall loss. The similarity measure function can be the DICE loss function, used to evaluate the similarity between two samples. The DICE value ranges from 0 to 1; a larger value indicates a more similar set and a smaller loss, while a smaller value indicates a greater difference between the sets and a larger loss.

[0051] In one alternative embodiment, such as Figure 2 As shown, since the target image is decoupled into two branches after feature extraction and classification by the initial neural network: the image segmentation Seg branch and the lane line existence Exist branch, the construction of the loss function should also include these two parts. The lane line existence Exist branch uses BCEWithLogitsLoss to construct its loss function based on the exist label and the predicted value of whether the sample image has lane lines inferred by the model. The calculation method is shown in Equation (1):

[0052] L exist ={l1,…,l n}

[0053] l n =-[y n ·log(σ(x n ))+(1-y n )·log(1-σ(x n (1)

[0054] The image segmentation Seg branch constructs its loss function based on the mask label and the predicted value of the lane line type in the sample image inferred by the model, using the dice loss method, as shown in Equation (2):

[0055]

[0056] Therefore, the loss function of this target detection method is calculated as shown in equation (3):

[0057] L = L dice +0.5L exist (3)

[0058] In equation (3), the parameter 0.5 is used to balance the weights between different loss functions and optimize the training process.

[0059] Example 2

[0060] According to an embodiment of the present invention, an image prediction device is also provided, which can execute the image prediction method in the above embodiments. The specific implementation and preferred application scenarios are the same as those in the above embodiments, and will not be described in detail here.

[0061] Figure 4 This is a schematic diagram of an image prediction device according to an embodiment of the present invention, such as... Figure 4 As shown, the device includes:

[0062] Acquisition module 42 is used to acquire the target image;

[0063] Prediction module 44 is used to predict the target image using the target neural network to obtain the target prediction result, wherein the target prediction result is used to indicate whether there are lane lines in the target image, and if there are lane lines in the target image, the type of lane lines contained in the target image.

[0064] The target neural network includes multiple sets of target convolutional layers, and each set of target convolutional layers contains the same number of channels.

[0065] The aforementioned acquisition and prediction modules are both integrated into the target vehicle system. The functional modules and data processing are implemented through the control system within the target vehicle. The acquisition module can be an image acquisition module, and the target image can be a monocular 2D lane line image along with its mask and exist labels. The prediction module can be a designed variable-group convolutional neural network used to predict the loaded target image, obtaining an optimal model of the variable-group convolutional neural network. Unlabeled image data is fed into the trained optimal model to obtain the lane line detection result inferred by the model. The detection result includes whether a lane line exists and, if so, what type of lane line it is.

[0066] For example, in the process of lane line detection in an autonomous driving system, the autonomous vehicle acquires monocular 2D lane line images and their mask and exist labels through the acquisition module using a monocular camera, and inputs them in batches into the prediction module. The prediction module is a designed variable group convolutional neural network. The target image transforms the detection of lane lines into the detection of each pixel. After processing the image matrix through multiple target convolutional layers, the output prediction result is the probability of the existence of lane lines and their type.

[0067] Optionally, according to the above embodiments of the invention, Figure 5 This is a schematic diagram of a prediction module of an image prediction device according to an embodiment of the present invention, as shown below. Figure 5 As shown, the target neural network also includes: a first sampling branch and a second sampling branch, and the prediction module includes:

[0068] Feature extraction unit 52 is used to extract features from the target image using multiple sets of target convolutional layers to obtain target feature vectors;

[0069] The prediction unit 54 is used to predict the target feature vector using the first sampling branch and the second sampling branch respectively, and obtain a first prediction result and a second prediction result. The first prediction result is used to indicate whether the target image contains lane lines, and the second prediction result is used to indicate the type of lane lines contained in the target image when the target image contains lane lines.

[0070] The determining unit 56 is used to obtain the target prediction result based on the first prediction result and the second prediction result.

[0071] The aforementioned units are all incorporated into the prediction module of the target vehicle system image prediction device. The functional units and data processing are implemented through the control system equipped in the target vehicle. When the feature extraction unit extracts target feature vectors, it loads the data in the model preprocessing section and loads the preprocessed image information and its label information into the training model. First, the original image undergoes random rotation from -2 to 2 degrees, random horizontal flipping, random Gaussian noise, and random brightness changes. This simulates target images of the same lane line under various conditions, such as strong light, occlusion, and different angles, to enhance the dataset. The original image is resized to 640×480×3 using the Resize function (Input(640,480,3)) to unify the image matrix size. Finally, the pixels are normalized, and the processed image information and its label information are loaded into the training model. A lightweight variable-group convolutional neural network is used as the backbone for multi-size feature extraction. The prediction unit uses the Exist branch of lane line existence to predict the probability that a lane line exists in the image. It then uses the Seg branch of image segmentation to predict the type of lane line present in the image. The determination unit can output whether a lane line exists relative to the vehicle's position in the scene, and if so, what type of lane line it is, based on the prediction results obtained from the Exist and Seg branches.

[0072] Optionally, according to the above embodiments of the invention, the device further includes: a receiving module for receiving a preset total number of channels; and a grouping module for grouping the target convolutional layer based on the preset total number of channels to obtain multiple groups of target convolutional layers.

[0073] Optionally, according to the above embodiments of the invention, the second sampling branch includes: a first sub-sampling layer and multiple second sub-sampling layers. The prediction unit in the prediction module of the image prediction device is further used to predict the target feature vector using the first sub-sampling layer to obtain a first sub-prediction result; to perform deconvolution upsampling on the target feature vector features using multiple second sub-sampling layers to obtain multiple second sub-prediction results, wherein the product of the sampling factors of the multiple second sub-sampling layers is the same as the sampling factor of the first sub-sampling layer; and to fuse the first sub-prediction result and the multiple second prediction results to obtain a second prediction result.

[0074] Optionally, according to the above embodiments of the invention, the acquisition module is further configured to acquire a sample image and a first label and a second label corresponding to the sample image, wherein the first label is used to indicate whether the sample image contains lane lines, and the second label is used to indicate the type of lane lines contained in the sample image when the sample image contains lane lines; the prediction module is further configured to use an initial neural network to predict the sample image to obtain a third prediction result and a fourth prediction result, wherein the third prediction result is used to indicate whether the sample image contains lane lines, and the fourth prediction result is used to indicate the type of lane lines contained in the sample image when the sample image contains lane lines; the device further includes: a function construction module, configured to construct a target loss function based on the first label, the second label, the third prediction result, and the fourth prediction result; and a parameter adjustment module, configured to adjust the network parameters of the initial neural network based on the target loss function to obtain a target neural network.

[0075] Optionally, according to the above embodiments of the invention, the function construction module includes: a function construction unit, used to determine a similarity measurement function based on a first label and a third prediction result; determine a cross-entropy loss function based on a second label and a fourth prediction result; and construct a target loss function based on the similarity measurement function and the cross-entropy loss function.

[0076] Example 3

[0077] According to another aspect of the present invention, an electronic device is also provided, including: a processor and a storage device.

[0078] One or more processors; a storage device for storing one or more programs; when one or more programs are executed by one or more processors, causing one or more processors to perform the image prediction method of any one of the above embodiments of the invention.

[0079] Example 4

[0080] According to another aspect of the present invention, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is running, it controls the processor of the device to execute any of the image prediction methods described above.

[0081] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0082] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0086] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0087] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image prediction method characterized by, The method comprises: obtaining a target image, wherein the target image comprises an existence label and a mask label, the existence label is used to represent whether a lane line exists, and the mask label is used to represent a type of lane line existing in the target image; predicting the target image by using a target neural network to obtain a target prediction result, wherein the target prediction result is used to represent whether a lane line exists in the target image, and in the case that the lane line exists in the target image, a type of the lane line included in the target image; wherein the target neural network comprises a plurality of groups of target convolution layers, a first sampling branch and a second sampling branch, each group of convolution layers in the plurality of groups of target convolution layers comprises the same number of channels, the first sampling branch is used to predict whether a lane line exists in the target image, and the second sampling branch is used to predict a type of the lane line included in the target image in the case that the lane line exists in the target image; wherein predicting the target image by using the target neural network to obtain the target prediction result comprises: extracting features of the target image by using the plurality of groups of target convolution layers to obtain a target feature vector; predicting the target feature vector by using the first sampling branch and the second sampling branch respectively to obtain a first prediction result and a second prediction result, wherein the first prediction result is used to represent whether the target image includes the lane line, and the second prediction result is used to represent a type of the lane line included in the target image in the case that the target image includes the lane line; and obtaining the target prediction result based on the first prediction result and the second prediction result; the second sampling branch comprises a first sub-sampling layer and a plurality of second sub-sampling layers, predicting the target feature vector by using the second sampling branch to obtain the second prediction result comprises: predicting the target feature vector by using the first sub-sampling layer to obtain a first sub-prediction result; performing deconvolution upsampling on the target feature vector by using the plurality of second sub-sampling layers respectively to obtain a plurality of second sub-prediction results, wherein a product of sampling multiples corresponding to the plurality of second sub-sampling layers is the same as a sampling multiple of the first sub-sampling layer; and fusing the first sub-prediction result and the plurality of second sub-prediction results to obtain the second prediction result.

2. The method of claim 1, wherein, The method further comprises: receiving a total number of preset channels; grouping the target convolution layers based on the total number of preset channels to obtain the plurality of groups of target convolution layers.

3. The method of claim 1, wherein, The method further comprises: obtaining a sample image and first and second labels corresponding to the sample image, wherein the first label is used to represent whether a lane line is included in the sample image, and the second label is used to represent a type of the lane line included in the sample image in the case that the sample image includes the lane line; predicting the sample image by using the initial neural network to obtain a third prediction result and a fourth prediction result, wherein the third prediction result is used to indicate whether the sample image contains the lane line, and the fourth prediction result is used to indicate the type of the lane line contained in the sample image in the case that the sample image contains the lane line; constructing a target loss function based on the first label, the second label, the third prediction result and the fourth prediction result; adjusting network parameters of the initial neural network based on the target loss function to obtain the target neural network.

4. The method of claim 3, wherein, constructing a target loss function based on the first label, the second label, the third prediction result and the fourth prediction result, comprising: determining a cross-entropy loss function based on the first label and the third prediction result; determining a similarity measurement function based on the second label and the fourth prediction result; constructing the target loss function based on the similarity measurement function and the cross-entropy loss function.

5. An image prediction apparatus characterized by comprising: comprising: an acquisition module configured to acquire a target image, wherein the target image comprises an existence label and a mask label, the existence label is used to indicate whether a lane line exists, and the mask label is used to indicate the type of the lane line existing in the target image; a prediction module configured to predict the target image by using a target neural network to obtain a target prediction result, wherein the target prediction result is used to indicate whether the target image contains a lane line, and in the case that the target image contains the lane line, the type of the lane line contained in the target image; wherein the target neural network comprises a plurality of groups of target convolution layers, a first sampling branch and a second sampling branch, each group of convolution layers in the plurality of groups of target convolution layers contains the same number of channels, the first sampling branch is used to predict whether the target image contains a lane line, and the second sampling branch is used to predict the type of the lane line contained in the target image in the case that the target image contains the lane line; the prediction module comprises: a feature extraction unit configured to perform feature extraction on the target image by using the plurality of groups of target convolution layers to obtain a target feature vector; a prediction unit configured to predict the target feature vector by using the first sampling branch and the second sampling branch respectively to obtain a first prediction result and a second prediction result, wherein the first prediction result is used to indicate whether the target image contains the lane line, and the second prediction result is used to indicate the type of the lane line contained in the target image in the case that the target image contains the lane line; and a determination unit configured to obtain the target prediction result based on the first prediction result and the second prediction result. The second sampling branch comprises: a first subsampling layer, a plurality of second subsampling layers, and a prediction unit, which is further configured to predict the target feature vector by using the first subsampling layer to obtain a first sub-prediction result; perform inverse convolution upsampling on the target feature vector by using the plurality of second subsampling layers respectively to obtain a plurality of second sub-prediction results, wherein the product of the sampling multiples of the plurality of second subsampling layers is the same as the sampling multiple of the first subsampling layer; and fuse the first sub-prediction result and the plurality of second sub-prediction results to obtain the second prediction result.

6. An electronic device, comprising: comprising: one or more processors; a memory device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors perform the image prediction method of any one of claims 1-4.

7. A non-volatile storage medium, characterized by The non-volatile storage medium comprises a stored program, wherein when the program is running, the processor of the device is controlled to perform the image prediction method of any one of claims 1-4.

Images