Image processing methods and related products based on image recognition models

Abstract

This application provides an image processing method and related products based on an image recognition model, which can be applied to artificial intelligence technology or in-vehicle scenarios, wherein the method includes: obtaining an image to be processed, which is photographed on a road where a vehicle is traveling; inputting the image to be processed into a first feature extraction network for processing to obtain initial features; inputting the initial features into a second feature extraction network comprising M feature extraction subnetworks connected in series for processing to obtain target features processed by each feature extraction subnetwork; inputting the target features processed by the Xth feature extraction subnetwork into a fine-grained element recognition network for processing to determine the fine-grained element recognition result; and inputting all or part of the initial features and the target features processed by each feature extraction subnetwork into a lane number recognition network for processing to determine the lane number recognition result. The above method can improve the accuracy of road element recognition.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method based on an image recognition model, an image processing device based on an image recognition model, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of society, vehicles are becoming increasingly popular and widely used as a means of transportation. To better assist drivers in driving, technologies such as vehicle navigation and automatic vehicle control have emerged. To achieve these functions, it is necessary to identify road elements such as lane lines and lane numbers. However, how to accurately identify road elements remains a challenge. Summary of the Invention

[0003] The embodiments of the present application provide an image processing method and related products based on an image recognition model, which can improve the accuracy of road element recognition.

[0004] On the one hand, an embodiment of the present application provides an image processing method based on an image recognition model, which is applied to the image recognition model, wherein the image recognition model includes a first feature extraction network, a second feature extraction network, a lane number recognition network, and a fine-grained element recognition network; the input of the second feature extraction network is connected to the output of the first feature extraction network; the second feature extraction network includes M feature extraction subnetworks connected in series, the scale of the features extracted by the Nth feature extraction subnetwork is greater than the scale of the features extracted by the N-1th feature extraction subnetwork, M is a positive integer greater than 1, and N is any positive integer greater than 1 and less than or equal to M; the input of the fine-grained element recognition network is connected to the output of the Mth feature extraction subnetwork, and the input of the lane number recognition network is connected to the output of each feature extraction subnetwork; the method includes:

[0005] Acquire an image to be processed, where the image to be processed is captured on a road where a vehicle is traveling;

[0006] Inputting the image to be processed into a first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed;

[0007] The initial features are input into the second feature extraction network for processing to obtain the target features processed by each feature extraction sub-network;

[0008] Input the target features obtained by the X-th feature extraction subnetwork into the fine-grained element recognition network for processing to determine the fine-grained element recognition results of the vehicle driving road in the image to be processed; X is a positive integer greater than or equal to 1 and less than or equal to M;

[0009] All or part of the initial features and the target features obtained by processing the feature extraction sub-networks are input into the lane number recognition network for processing to determine the lane number recognition result of the vehicle driving road in the image to be processed.

[0010] On the one hand, an embodiment of the present application provides an image processing device based on an image recognition model, wherein the image recognition model includes a first feature extraction network, a second feature extraction network, a lane number recognition network, and a fine-grained element recognition network; the input of the second feature extraction network is connected to the output of the first feature extraction network; the second feature extraction network includes M feature extraction subnetworks connected in series, the scale of the features extracted by the Nth feature extraction subnetwork is larger than the scale of the features extracted by the N-1th feature extraction subnetwork, M is a positive integer greater than 1, and N is any positive integer greater than 1 and less than or equal to M; the input of the fine-grained element recognition network is connected to the output of the Mth feature extraction subnetwork, and the input of the lane number recognition network is connected to the output of each feature extraction subnetwork; the device includes:

[0011] An acquisition unit, configured to acquire an image to be processed, wherein the image to be processed is obtained by photographing a road on which a vehicle is traveling;

[0012] a processing unit, configured to input the image to be processed into the first feature extraction network for processing, and obtain initial features of the vehicle driving road in the image to be processed;

[0013] The processing unit is further configured to input the initial features into the second feature extraction network for processing to obtain target features processed by each feature extraction subnetwork;

[0014] The processing unit is further configured to input the target feature obtained by processing the X-th feature extraction subnetwork into the fine-grained element recognition network for processing, and determine a fine-grained element recognition result of the vehicle travel road in the image to be processed; X is a positive integer greater than or equal to 1 and less than or equal to M;

[0015] The processing unit is further used to input all or part of the initial features and the target features obtained by processing the various feature extraction subnetworks into the lane number recognition network for processing, and determine the lane number recognition result of the vehicle driving road in the image to be processed.

[0016] On the one hand, an embodiment of the present application provides a computer device, comprising: a processor, a communication interface and a memory, wherein the processor, the communication interface and the memory are interconnected, wherein the memory stores executable program code, and the processor is used to call the executable program code to implement the image processing method based on the image recognition model provided in the embodiment of the present application.

[0017] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to implement the image processing method based on the image recognition model provided in the embodiment of the present application.

[0018] Accordingly, an embodiment of the present application also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, the steps of the image processing method based on the image recognition model provided in the embodiment of the present application are implemented.

[0019] Accordingly, an embodiment of the present application also provides a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device implements the image processing method based on the image recognition model provided in an embodiment of the present application.

[0020] In an embodiment of the present application, the first feature extraction network is first used to extract the initial features of the vehicle driving road in the image to be processed, and then the initial features are processed using M feature extraction sub-networks connected in series to obtain target features of the vehicle driving road in the image to be processed at multiple different scales. Then, the lane number recognition result of the vehicle driving road in the image to be processed is determined by combining the initial features and all or part of the target features of multiple different scales, which can improve the accuracy of lane number recognition. In addition, the image recognition model provided in the embodiment of the present application includes multiple task networks. By inputting an image of a vehicle driving road, the recognition results of different road elements of the vehicle driving road can be determined, and the recognition efficiency is higher. At the same time, because multiple task networks share the backbone network, it can maximize the image recognition model to reduce the number of parameters of the image recognition model when performing road element recognition, and the computational efficiency is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 This is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application;

[0023] Figure 2 This is a flowchart of an image processing method based on an image recognition model provided in an embodiment of the present application;

[0024] Figure 3 This is a schematic diagram of the structure of an image recognition model provided in an embodiment of the present application;

[0025] Figure 4 This is a schematic diagram of vehicle navigation provided by an embodiment of the present application;

[0026] Figure 5 This is another vehicle navigation schematic diagram provided by an embodiment of the present application;

[0027] Figure 6 This is another vehicle navigation schematic diagram provided by an embodiment of the present application;

[0028] Figure 7 This is another vehicle navigation schematic diagram provided by an embodiment of the present application;

[0029] Figure 8 This is another vehicle navigation schematic diagram provided by an embodiment of the present application;

[0030] Figure 9 1 is a flow chart of another image processing method based on an image recognition model provided in an embodiment of the present application;

[0031] Figure 10 This is a schematic diagram of the structure of another image recognition model provided in an embodiment of the present application;

[0032] Figure 11 This is a schematic diagram of the structure of an image recognition model provided in an embodiment of the present application;

[0033] Figure 12 is a schematic diagram of an image processing solution provided in an embodiment of the present application;

[0034] Figure 13 1 is a schematic structural diagram of an image processing device based on an image recognition model provided in an embodiment of the present application;

[0035] Figure 14 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] It should be noted that the terms "first" and "second" in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature designated as "first" or "second" may explicitly or implicitly include at least one such feature.

[0038] Some key terms that appear in the embodiments of this application are explained below.

[0039] Lane Number: This includes the total number of lanes on the road the vehicle is currently on, as well as the lane numbers for the left and right lanes. Identifying the current lane number provides accurate guidance for lane-level guidance and determining deviation from the primary and secondary roads.

[0040] Computer vision (CV) is the science of making machines "see". Specifically, it refers to the use of cameras and computers to replace the human eye to identify, follow, and measure targets, and further perform image processing, using computers to process the images into images that are more suitable for human observation or transmission to instruments for detection.

[0041] Deep learning: Deep learning is the process of learning the inherent patterns and representational hierarchies of sample data (such as images, speech, and text), enabling machines to analyze and learn like humans, capable of recognizing text, images, and sound. It is widely used in the field of artificial intelligence. Convolutional neural networks are a common structure in deep learning.

[0042] Convolutional Neural Networks (CNNs): These are deep feedforward neural networks that incorporate convolutional computations. They can perform supervised learning using labeled training data, enabling tasks such as visual image recognition and object detection.

[0043] The embodiment of the present application provides an image processing method based on an image recognition model, which includes a method for identifying road elements in an image of a vehicle traveling on a road. Based on the image recognition model, multiple target features of different scales are obtained according to initial features, and a lane number recognition result is obtained based on the multiple target features of different scales and the initial features; at the same time, a fine-grained element recognition result is obtained based on the target features. Because lane number recognition is performed by fusing initial features and multiple target features of different scales, the accuracy of lane number recognition can be improved. At the same time, because the image recognition model includes multiple task networks, and multiple task networks share a backbone network, this can maximize the guarantee that the image recognition model reduces the number of parameters of the image recognition model when performing road element recognition, and the computational efficiency is higher.

[0044] The image processing method based on the image recognition model provided in the embodiment of the present application can be implemented based on artificial intelligence (AI) technology. AI refers to the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. AI technology is an interdisciplinary subject, and the fields it covers are relatively broad; and the image processing method based on the image recognition model provided in the embodiment of the present application mainly involves machine learning (ML) technology and computer vision technology in AI technology. Machine learning generally includes artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction and other technologies.

[0045] The image processing method based on the image recognition model provided in the embodiment of the present application can be applied to Figure 1 The data processing system architecture shown in FIG. The data processing system architecture includes a vehicle 101, an onboard data processing device 102, a server 103, a data acquisition device 104, and an onboard device display 105 (i.e., a lane prompt interface). The vehicle 101, the onboard data processing device 102, the server 103, the data acquisition device 104, and the onboard device display 105 are connected via a network.

[0046] like Figure 1 As shown, the vehicle 101 is equipped with an on-board data processing device 102. The on-board data processing device 102 obtains image data of the vehicle's driving road through the data acquisition device 104. The on-board data processing device 102 inputs the image data of the vehicle's driving road obtained by the acquisition device 104 into the server 103 for processing, and obtains recognition results of various elements of the vehicle's driving road. The on-board data processing device 102 displays the recognition results of various elements of the vehicle's driving road on the on-board device display screen 105.

[0047] in, Figure 1The server 103 shown can be configured in the vehicle-mounted data processing device 102 or a cloud server. The vehicle-mounted data processing device 102 can be a server or terminal with data (such as image) processing capabilities. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device, vehicle-mounted terminal, etc., but is not limited to these. Figure 1 The data acquisition device 104 shown can be configured in the vehicle-mounted data processing device, or can be an independent data acquisition device. The image processing method based on the image recognition model provided in the embodiment of the present application can be executed by the vehicle-mounted data processing device 102, specifically:

[0048] Applied to an image recognition model, the image recognition model includes a first feature extraction network, a second feature extraction network, a coarse-grained element recognition network, a lane number recognition network and a fine-grained element recognition network; the input of the second feature extraction network is connected to the output of the first feature extraction network; the second feature extraction network includes M feature extraction sub-networks connected in series, the scale of the features extracted by the Nth feature extraction sub-network is larger than the scale of the features extracted by the N-1th feature extraction sub-network, M is a positive integer greater than 1, and N is any positive integer greater than 1 and less than or equal to M; the input of the coarse-grained element recognition network is connected to the output of the MKth feature extraction sub-network, K is a positive integer greater than or equal to 1 and less than M; the input of the fine-grained element recognition network is connected to the output of the Mth feature extraction sub-network, and the input of the lane number recognition network is connected to the output of each feature extraction sub-network.

[0049] The data acquisition device 104 is used to acquire an image to be processed. Then, feature extraction is performed on the image to be processed to obtain initial features of the vehicle driving road in the image to be processed. Based on the initial features, target features processed by each feature extraction subnetwork are obtained. The target features processed by the Xth feature extraction subnetwork are input into the fine-grained element recognition network for processing to determine the fine-grained element recognition result of the vehicle driving road in the image to be processed. X is a positive integer greater than or equal to 1 and less than or equal to M. The target features processed by the MKth feature extraction subnetwork are input into the coarse-grained element recognition network for processing to determine the coarse-grained element recognition result of the vehicle driving road in the image to be processed. All or part of the initial features and the target features processed by each feature extraction subnetwork are input into the lane number recognition network for processing to determine the lane number recognition result of the vehicle driving road in the image to be processed. By adopting the above method, the first feature extraction network can be used to extract the initial features of the vehicle driving road in the image to be processed, and then the initial features can be processed by M feature extraction sub-networks connected in series to obtain target features of the vehicle driving road in the image to be processed at multiple different scales. Then, the lane number recognition result of the vehicle driving road in the image to be processed can be determined by combining the initial features and all or part of the target features of multiple different scales, which can improve the accuracy of lane number recognition. In addition, the image recognition model provided by the embodiment of the present application includes multiple task networks. By inputting an image of the vehicle driving road, the recognition results of different road elements of the vehicle driving road can be determined, and the recognition efficiency is higher. At the same time, because multiple task networks share the backbone network, it can maximize the image recognition model to reduce the number of parameters of the image recognition model when performing road element recognition, and the computational efficiency is higher.

[0050] The above briefly introduces the image processing method based on the image recognition model provided in the embodiment of the present application. The specific implementation method of the image processing method based on the image recognition model is described in detail below:

[0051] See also Figure 2 , Figure 2 This is a flow chart of an image processing method based on an image recognition model provided in an embodiment of the present application. The image processing method based on an image recognition model described in an embodiment of the present application can be Figure 1 The method may include the following steps:

[0052] S201: Acquire an image to be processed, where the image to be processed is captured on a road where a vehicle is traveling.

[0053] In the embodiment of the present application, the image to be processed obtained by the vehicle-mounted data processing device 102 may be Figure 1The data acquisition device 104 shown here can be sent by a camera attached to the vehicle-mounted data processing device 102 after real-time acquisition. This is often used in real-time scenarios, such as identifying lane numbers in autonomous driving scenarios. Alternatively, the data acquisition device 104 can be used to complete an image processing task, such as identifying lane elements in an image to be processed and achieving lane positioning in a map-based scenario. Alternatively, the images to be processed acquired by the vehicle-mounted data processing device 102 can be various types of images to be processed, obtained from the data acquisition device 104 during image recognition model training. These images to be processed are all screened and determined by testers.

[0054] Among them, Figure 3 As shown, the image recognition model includes a first feature extraction network, a second feature extraction network, a lane number recognition network and a fine-grained element recognition network; the input of the second feature extraction network is connected to the output of the first feature extraction network; the second feature extraction network includes M feature extraction sub-networks connected in series, the scale of the features extracted by the Nth feature extraction sub-network is larger than the scale of the features extracted by the N-1th feature extraction sub-network, M is a positive integer greater than 1, and N is any positive integer greater than 1 and less than or equal to M; the input of the fine-grained element recognition network is connected to the output of the Mth feature extraction sub-network, and the input of the lane number recognition network is connected to the output of each feature extraction sub-network.

[0055] S202: Input the image to be processed into a first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed.

[0056] In the embodiment of the present application, the process of inputting the image to be processed into the first feature extraction network for processing is to convolve the image to be processed according to the elements of the image features in the image to be processed, thereby obtaining an initial feature of the vehicle's road in the image to be processed, which includes multiple elements and their corresponding attributes. The attributes corresponding to the elements included in the initial feature include lane markings, curbs, guardrails, zebra crossings (sidewalks), stop lines, and guide strips.

[0057] In one embodiment, the image to be processed is preprocessed to obtain a preprocessed image; wherein the preprocessing includes one or more of denoising, normalization, and averaging; the preprocessed image is input into a first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed.

[0058] Image denoising (denoising) refers to the process of reducing noise in digital images. Real-world digital images are often affected by interference from the imaging device and the external environment during digitization and transmission. Images generated based on these influences are called noisy images or noisy images. Because image clarity decreases after denoising, the greater the denoising effort, the more severe the decrease in image clarity. In embodiments of the present application, whether to perform denoising and the intensity of the denoising process can be determined based on the extent of the noise impact on the image. This decision is based on the specific situation and is not intended to be limiting. For example, when the noise in a noisy image has a non-negligible impact on the elements to be identified, denoising is necessary. The intensity of the denoising process is determined based on the impact of the noise on the elements to be identified. For another example, when the noise in a noisy image has no impact or a very small impact on the elements to be identified, the noise can be ignored and denoising is not required. This approach can minimize the impact of denoising.

[0059] Among them, image normalization (normalization processing) refers to the process of performing a series of standard processing transformations on an image to transform it into a fixed standard form. This standard image is called a normalized image. That is, after the original image undergoes some processing, multiple copy images can be obtained. After these copy images undergo image normalization processing with the same parameters, they can obtain standard images of the same form. After this operation, the same images taken at different times and environments can be recognized, so that the image recognition model can better recognize the images to be processed and improve the accuracy of image recognition. For example, suppose there are images of the same location where it rains in the morning, rains in the evening, and does not rain in the morning or does not rain in the evening. When these images are input into the image recognition model, the image recognition model will treat these images as images of the same type after normalization processing, and can better recognize these images.

[0060] The most commonly used averaging method is zero-averaging. This process involves subtracting the mean of all pixels in the training set from each pixel's value to produce an image with a mean of zero. For example, if the calculated mean of all pixels is 128, then after subtracting 128, the pixel value range is now [-128, 127], which satisfies the mean of zero. This avoids "Z-shaped updates" (i.e., irregular updates), accelerates the convergence of the neural network, and thus improves the efficiency of image recognition.

[0061] S203: Input the initial features into the second feature extraction network for processing to obtain target features processed by each feature extraction sub-network.

[0062] In the embodiment of the present application, each feature extraction subnetwork is a serial upsampling feature extraction subnetwork. Each feature extraction subnetwork may have different focuses on feature extraction elements, and the clarity of elements in the obtained target features may also be different.

[0063] S204. Input the target features obtained by processing the Xth feature extraction subnetwork into the fine-grained element recognition network for processing to determine the fine-grained element recognition results of the vehicle driving road in the image to be processed, where X is a positive integer greater than or equal to 1 and less than or equal to M.

[0064] In one embodiment, the target features obtained by processing the Xth feature extraction subnetwork are input into the fine-grained element recognition network for processing to determine the fine-grained element recognition result of the vehicle driving road in the image to be processed. The method can be: the target features obtained by processing the Xth feature extraction subnetwork are input into the fine-grained element recognition network for processing, and the fine-grained foreground pixels in the image to be processed are determined based on the target features obtained by processing the Xth feature extraction subnetwork; the lane line pixels are determined based on the fine-grained foreground pixels, and the lane line pixels are subjected to color classification and line type classification; the lane line recognition result is determined based on the color classification result and the line type classification result, and the lane line recognition result is determined as the fine-grained element recognition result of the vehicle driving road in the image to be processed.

[0065] It should be noted that the embodiment of the present application splits the pixel categories into three dimensions: whether it is a foreground pixel (background, foreground or lane line pixel), color (white, yellow), and type (dashed line, solid line). In addition, multi-level learning is performed during the image recognition model training process. First, all pixels are classified into a binary classification of whether they are foreground pixels, and then only the foreground pixels, which account for a relatively small proportion of all pixels, are classified in the color and type dimensions. This method can achieve better lane line recognition results.

[0066] Among them, the target features obtained by the X-th feature extraction sub-network are input into the fine-grained element recognition network for processing. This is based on comprehensive factors such as the computing power and computing time of the image recognition model. There is no limitation on which feature extraction sub-network is used to process the target features. It is determined based on actual conditions, which is more in line with actual conditions.

[0067] Among them, determining lane line pixels based on fine-grained foreground pixels can be: analyzing the connected domain formed by the fine-grained foreground pixels, and determining the lane line according to the range of the width and length of the connected domain. The width of the lane line has a certain range, and this range is determined according to the actual situation; the length of the lane line needs to be greater than a certain value, and the longer it is, the easier it is to judge. However, there are some extreme cases, such as when the length of the lane line in the target feature is originally very short, then other auxiliary means are needed. For example, in an image to be processed obtained by shooting a vehicle on a driving road at an intersection, the length of the lane line in the target feature obtained from the image to be processed is very short. In this case, auxiliary judgment can be made based on the traffic lights at the intersection, or based on the characteristics of the intersection. The embodiments of this application do not limit the elements for auxiliary judgment.

[0068] Connected domain is an important concept for describing regions and boundaries. The two necessary conditions for two pixels to be connected are: whether the positions of the two pixels are adjacent, and whether the grayscale values of the two pixels meet a specific similarity criterion (or whether they are equal). In other words, the region formed by the connection of adjacent pixels is called a connected domain.

[0069] Among them, lane line pixels are determined based on fine-grained foreground pixels, and stop line pixels can also be determined based on fine-grained foreground pixels to determine the type of stop line, whether it is a single solid line or a double solid line. In this method, lane line pixels and stop line pixels can be determined based on the direction of fine-grained foreground pixels. The direction of lane line pixels is consistent with the direction of vehicle travel (the direction of lane line pixels is vertical), and the direction of stop line pixels is inconsistent with the direction of vehicle travel (the direction of stop line pixels is horizontal). The connected domain formed by fine-grained foreground pixels can also be analyzed, and the stop line can be determined based on the width and length range of the connected domain. The width of the stop line has a certain range, which is determined according to actual conditions; the length of the stop line also has a certain range, such as being equal to the lane width of a single lane or slightly larger than the lane width in one direction.

[0070] In one embodiment, after obtaining the lane line recognition result, the direction of vehicle travel can also be displayed on the lane prompt interface according to the lane line recognition result. Figure 4 , Figure 4 This is a schematic diagram of vehicle navigation provided by an embodiment of the present application. Figure 4 As shown, the vehicle is traveling in a lane, and the vehicle's onboard device display (lane prompt interface) will display the vehicle's current driving status on the road. The lane where the arrow is located is the lane the vehicle is currently in, and the direction the arrow points to is the direction the vehicle is currently traveling.

[0071] In another embodiment, after obtaining the lane line recognition result, based on the vehicle driving navigation map, the lane line of the current lane in which the vehicle is traveling is displayed on the lane prompt interface according to the direction of the vehicle's driving (straight driving). For example, augmented reality (AR) navigation is displayed on the corresponding display screen. Figure 5 , Figure 5 This is another vehicle navigation diagram provided by an embodiment of the present application. Figure 5 As shown in the figure, the vehicle is driving in a lane, and the vehicle's onboard device display shows the vehicle's current driving status on the road. The left side is a map navigation diagram, which shows the distance before turning, the approximate time to reach the destination, and the direction of the vehicle's travel. The right side is an AR navigation diagram, and the lane where the arrow is located is the lane where the vehicle is currently located, and the direction the arrow is pointing is the direction of the vehicle's current travel.

[0072] In another embodiment, after obtaining the lane line recognition result, based on the vehicle driving navigation map, the direction of vehicle driving (turning) is displayed on the lane prompt interface according to the lane line of the adjacent lane and the lane line of the current lane. Figure 6 , Figure 6 This is another vehicle navigation diagram provided by the embodiment of the present application. Figure 6 As shown, the vehicle is traveling in a lane, and the vehicle's onboard device display shows the vehicle's current driving status on the road. On the left is a map navigation chart, which shows the distance to turn, the approximate time to reach the destination, and the vehicle's direction of travel. On the right is an AR navigation chart, with the arrow indicating the lane the vehicle is currently in, pointing in the direction the vehicle is turning. Lane line position determines the placement of the road arrows in the AR navigation chart, and lane line type determines whether lane change prompts in the AR navigation chart can be used.

[0073] In one embodiment, determining fine-grained foreground pixels in an image to be processed based on target features obtained by processing an X-th feature extraction subnetwork may include: determining foreground pixels in an image to be processed based on target features obtained by processing an X-th feature extraction subnetwork; performing a connected domain analysis on the foreground pixels to determine one or more connected domains formed based on the foreground pixels; and determining fine-grained foreground pixels in the image to be processed based on the determined one or more connected domains.

[0074] The method of determining the fine-grained foreground pixels in the image to be processed based on the determined one or more connected domains may include determining the fine-grained foreground pixels in the image to be processed based on the width and length of the connected domains. A connected domain is represented by a solid line, and multiple connected domains are represented by dotted lines.

[0075] S205: Input all or part of the initial features and the target features obtained by processing the feature extraction sub-networks into the lane number recognition network for processing to determine the lane number recognition result of the vehicle driving road in the image to be processed.

[0076] It should be noted that the present embodiment determines the lane number recognition result for the vehicle's road in the processed image without detecting lane lines or road edges. Instead, the lane number is indirectly inferred based on lane line data or the distance from the road edge. The present embodiment directly outputs the vehicle's lane number by fusing initial features with multiple target features at different scales through an image recognition model, thereby improving the accuracy of lane number recognition.

[0077] In one embodiment, the lane number recognition result includes: the lane where the vehicle is located is the left lane number from the left and the right lane number from the right of the vehicle's driving road, the confidence level corresponding to the left lane number, and the confidence level corresponding to the right lane number; this method can also: display the left lane number and the confidence level corresponding to the left lane number in the left position area of the lane prompt interface; and display the right lane number and the confidence level corresponding to the right lane number in the right position area of the lane prompt interface.

[0078] The left lane number and the corresponding confidence level are displayed in the left area of the lane prompt interface, and the right lane number and the corresponding confidence level are displayed in the right area of the lane prompt interface. That is, by default, the left side of the prompt interface displays the relevant content of the left lane, and the right side displays the relevant content of the right lane. This can reduce the display area of the prompt interface and avoid excessive information affecting the driver's field of vision and driving concentration. For example, refer to Figure 7 , Figure 7 This is another vehicle navigation diagram provided by the embodiment of the present application. Figure 7 As shown in the figure, the vehicle is driving on a lane, and the vehicle's onboard device display will show the vehicle's current driving status on the road. The "3" displayed on the left side of the onboard device display indicates that the current lane is the third lane from the left of the road, with a corresponding confidence level of 0.972; the "2" displayed on the left side indicates that the current lane is the second lane from the right of the road, with a corresponding confidence level of 0.909.

[0079] Optionally, when "0" is displayed on the left or right side, it means that the left road edge or the right road edge does not appear in the field of view of the vehicle-mounted device, and the number of left lanes or the number of right lanes cannot be determined.

[0080] In another embodiment, the left position area of the lane prompt interface displays the number of left lanes and the corresponding confidence level; the right position area of the lane prompt interface displays the number of right lanes and the corresponding confidence level; the middle position of the lane prompt interface displays the total number of lanes. Figure 8 , Figure 8 This is another vehicle navigation diagram provided by the embodiment of the present application. Figure 8 As shown, the vehicle is driving on the lane, and the vehicle's on-board device display screen will show the vehicle's driving status on the current road. It can clearly know the total number of lanes and the number of lanes the vehicle is currently in, so that the driver can have a general understanding of the lane being driven, which can assist the driver in driving the vehicle. Among them, the "3" displayed on the left indicates that the current lane is the third lane from the left of the road, and the corresponding confidence level is 0.972; the "2" displayed on the left indicates that the current lane is the second lane from the right of the road, and the corresponding confidence level is 0.909; the "4" displayed above the horizontal line indicates that the total number of lanes in the current lane is 4 lanes.

[0081] It should be noted that the total number of lanes displayed requires the specific values of the number of left and right lanes. Otherwise, "0" or no display will be displayed, indicating that the left or right road edge is not in the field of view of the on-board device, and the total number of lanes cannot be determined. In other words, the number of left or right lanes cannot be determined.

[0082] Optionally, the content displayed on the lane prompt interface is adjustable and can be adjusted based on the driver's preferences. For example, some drivers dislike seeing the lane number displayed on the lane prompt interface and can disable the lane number display function. For another example, some drivers like to see the lane number displayed on the lane prompt interface, but prefer to see the complete lane number displayed, such as 3 lanes from the left and 5 lanes from the right. For another example, some drivers dislike the position of the displayed content and can adjust the position of the displayed content, such as moving the display to the upper left corner or upper right corner. This embodiment of the present application is not limited to this.

[0083] In an embodiment of the present application, a first feature extraction network can be used to extract the initial features of the vehicle driving road in the image to be processed, and then the initial features can be processed using M feature extraction sub-networks connected in series to obtain target features of the vehicle driving road in the image to be processed at multiple different scales. Then, the lane number recognition result of the vehicle driving road in the image to be processed can be determined by combining the initial features and all or part of the target features of multiple different scales, which can improve the accuracy of lane number recognition. In addition, the image recognition model provided by the embodiment of the present application includes multiple task networks. By inputting an image of a vehicle driving road, the recognition results of different road elements of the vehicle driving road can be determined, and the recognition efficiency is higher. At the same time, because multiple task networks share the backbone network, it can maximize the image recognition model to reduce the number of parameters of the image recognition model when performing road element recognition, and the computational efficiency is higher.

[0084] See also Figure 9 , Figure 9 This is a flow chart of another image processing method based on an image recognition model provided in an embodiment of the present application. The image processing method based on an image recognition model described in an embodiment of the present application can be Figure 1 The method may be performed by the vehicle-mounted data processing device 102 shown, and may include the following steps:

[0085] S901: Acquire an image to be processed, where the image to be processed is captured on a road where a vehicle is traveling.

[0086] It should be noted that the embodiment of this application uses another image recognition model. Figure 10 As shown, the image recognition model includes a first feature extraction network, a second feature extraction network, a coarse-grained element recognition network, a lane number recognition network and a fine-grained element recognition network; the input of the second feature extraction network is connected to the output of the first feature extraction network; the second feature extraction network includes M feature extraction sub-networks connected in series, the scale of the features extracted by the Nth feature extraction sub-network is larger than the scale of the features extracted by the N-1th feature extraction sub-network, M is a positive integer greater than 1, and N is any positive integer greater than 1 and less than or equal to M; the input of the coarse-grained element recognition network is connected to the output of the MKth feature extraction sub-network, K is a positive integer greater than or equal to 1 and less than M; the input of the fine-grained element recognition network is connected to the output of the Mth feature extraction sub-network, and the input of the lane number recognition network is connected to the output of each feature extraction sub-network.

[0087] For the specific implementation of step S901, please refer to the Figure 2 The relevant contents of step S201 in the image processing method based on the image recognition model shown are not repeated here in this embodiment.

[0088] S902: Input the image to be processed into a first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed.

[0089] For the specific implementation of step S902, please refer to the Figure 2 The relevant contents of step S202 in the image processing method based on the image recognition model shown are not repeated here in this embodiment.

[0090] S903: Input the initial features into the second feature extraction network for processing to obtain target features processed by each feature extraction sub-network.

[0091] For the specific implementation of step S903, please refer to the Figure 2 The relevant contents of step S203 in the image processing method based on the image recognition model shown are not repeated here in this embodiment.

[0092] S904: Input the target features obtained by processing the MKth feature extraction sub-network into the coarse-grained element recognition network for processing, and determine the coarse-grained element recognition result of the vehicle driving road in the image to be processed.

[0093] Coarse-grained element recognition results include one or more of zebra crossing, guardrail, curb, and guide strip recognition. In AR navigation, curbs and guardrails can be used to determine the edge of the road. Zebra crossings, stop signs, and guide strips are also important elements in the road scene and can be used for map element reflow.

[0094] Among them, the value of K is:

[0095] Case 1: The value of K is 1, that is, the target feature obtained by the penultimate feature extraction subnetwork is input into the coarse-grained element recognition network for processing to determine the coarse-grained element recognition result of the vehicle driving road in the image to be processed.

[0096] The K value of 1 generally occurs when the requirements for coarse-grained elements are small and the overall coarse-grained elements have little impact. In this way, while meeting the coarse-grained detection accuracy, more resources of the image recognition model 200 are used to learn fine-grained elements (such as lane lines and stop line pixels), thereby improving the detection accuracy of fine-grained elements. For example, in a wide lane scene, it is generally not necessary to detect coarse-grained elements (such as curbs and guardrails) very accurately. In this way, a part of the resources of the image recognition model 200 can be released for the detection of fine-grained elements, thereby improving the detection effect of fine-grained elements such as lane lines and stop lines.

[0097] Case 2: K is set to 0, that is, the target feature obtained by the last feature extraction sub-network is input into the coarse-grained element recognition network for processing to determine the coarse-grained element recognition result of the vehicle driving road in the image to be processed.

[0098] A K value of 0 typically occurs when high coarse-grained elements are required. For example, in a mountain road scenario, very accurate coarse-grained element detection is required, as most mountain roads are bordered by a mountain on one side and a cliff on the other. Inaccurate coarse-grained detection can easily lead to unnecessary problems, so very accurate coarse-grained element detection is generally required in this scenario.

[0099] Case 3: K is set to n, where n is a positive integer greater than 1 and less than M.

[0100] The value of K being n generally occurs when there is basically no requirement for coarse-grained elements and a very high requirement for fine-grained elements. For example, in a very wide lane scenario, it is generally not necessary to detect coarse-grained elements (such as curbs and guardrails) very accurately, because the lane is very wide and most vehicles are traveling in the lane. Even if there are a lot of vehicles, as long as the vehicles are traveling in the current lane, it is fine. This can free up some resources of the image recognition model 200 for the detection of fine-grained elements, thereby improving the detection effect of fine-grained elements such as lane lines and stop lines.

[0101] It should be noted that the value of K can be adjusted manually or the model can be adjusted in real time according to the current lane scene, which is not limited in the embodiments of this application.

[0102] In one implementation, the coarse-grained element recognition result includes one or more of a guardrail recognition result and a curb recognition result, wherein the guardrail recognition result includes the distance between the vehicle and the guardrail, and the curb recognition result includes the distance between the vehicle and the curb. The following operations may also be performed:

[0103] When a warning condition is detected based on the coarse-grained element recognition result, a warning message is output; wherein the warning message is used to prompt the driver that there is a collision risk, and the warning condition is satisfied, including one or more of the following: the distance between the vehicle and the guardrail is less than or equal to a first distance threshold, and the distance between the vehicle and the curb is less than or equal to a second distance threshold;

[0104] When it is detected based on the coarse-grained element recognition result that the vehicle control conditions are met, the vehicle's driving is controlled; wherein, the vehicle control conditions are met including one or more of the distance between the vehicle and the guardrail being less than or equal to a third distance threshold and the distance between the vehicle and the curb being less than or equal to a fourth distance threshold; the third distance threshold is less than the first distance threshold, and the fourth distance threshold is less than the second distance threshold.

[0105] Among them, when the warning conditions are met, the warning information is output:

[0106] Case 1: The distance between the vehicle and the guardrail is within the warning distance. In this case, a text or logo warning can be issued at a preset location in the lane prompt interface, or a voice warning can be issued through the vehicle's voice system. Alternatively, a text or logo warning can be issued at a preset location in the lane prompt interface, and a voice warning can be issued through the vehicle's voice system.

[0107] Case 2: The distance between the vehicle and the curb is within the warning distance. The method of issuing the warning is the same as that of issuing the warning in the above case 1 and will not be repeated here.

[0108] In case 3, the distances between the vehicle and the guardrail and the vehicle and the curb are both within the warning distance. The method for issuing the warning is the same as that for case 1 above and will not be repeated here.

[0109] It should be noted that when the warning conditions are met, the warning information is output to avoid unnecessary traffic problems.

[0110] Among them, when the vehicle control conditions are met, the vehicle's driving is controlled:

[0111] Case 1: The distance between the vehicle and the guardrail is within the distance where the vehicle needs to be controlled. In this case, the on-board equipment will use forced means to control the vehicle to avoid collision with the guardrail and reduce traffic problems.

[0112] Case 2: The distance between the vehicle and the curb is within the distance required to control the vehicle. The method of controlling the vehicle is the same as the method of controlling the vehicle in the above case 1 and will not be repeated here.

[0113] In case 3, the distance between the vehicle and the guardrail and the distance between the vehicle and the curb are within the distance required to control the vehicle. The method of controlling the vehicle is the same as the method of controlling the vehicle in case 1 above, and will not be repeated here.

[0114] It should be noted that when the vehicle control conditions are met, the vehicle's driving is controlled when the vehicle can be driven automatically or when the vehicle has an emergency control function.

[0115] S905. Input the target features obtained by processing the Xth feature extraction subnetwork into the fine-grained element recognition network for processing to determine the fine-grained element recognition results of the vehicle driving road in the image to be processed, where X is a positive integer greater than or equal to 1 and less than or equal to M.

[0116] For the specific implementation of step S905, please refer to the Figure 2 The relevant contents of step S204 in the image processing method based on the image recognition model shown are not repeated here in this embodiment.

[0117] S906: Input all or part of the initial features and the target features obtained by processing the feature extraction sub-networks into the lane number recognition network for processing to determine the lane number recognition result of the vehicle driving road in the image to be processed.

[0118] For the specific implementation of step S906, please refer to the Figure 2 The relevant contents of step S205 in the image processing method based on the image recognition model shown are not repeated here in this embodiment.

[0119] For example, the new version of the image recognition model in the embodiment of the present application has a better effect on detecting elements such as lane numbers and lane lines than the old version of the image recognition model. Figure 11As shown, the upper figure is the new version of the image recognition model in the embodiment of the present application, and the lower figure is the old version of the image processing model. In terms of model structure, the new version of the image recognition model in the embodiment of the present application outputs multiple elements and corresponding attributes through a network. The detection task heads of these elements share a backbone network, which can maximize the type and accuracy of model recognition elements while reducing the number of model parameters to better adapt to low-performance vehicle platforms. The new version of the image recognition model in the embodiment of the present application improves the recognition of foreground pixels of fine-grained elements, and splits the 9 tiled fine-grained elements (background, white solid line, white dashed line, yellow solid line, yellow dashed line, guardrail, curb, zebra crossing, stop line). First, large-area elements such as curbs, guardrails and zebra crossings are detected with a separate task head according to the coarse and fine granularity, and small-area lane lines and stop lines are in one task head. Secondly, the lane lines are split into three dimensions: whether they are foreground pixels (background, foreground, i.e. lane line pixels), color (white, yellow), and type (dashed line, solid line). During the training process, multi-level learning is performed. First, all pixels are classified into two categories: foreground or not (foreground: lane line pixels, background: non-lane line pixels). Then, only the foreground pixels with a smaller proportion are reclassified in terms of color and type dimensions. The old version of the image recognition model needs to identify whether all pixels in the image are lane lines, as well as color and virtuality, resulting in redundant recognition of information. Therefore, the new version of the image recognition model in the embodiment of the present application can achieve better detection results under the premise of the same amount of computation. The embodiment of the present application has made improvements in lane number recognition. The lane number recognition is achieved by collecting a large amount of data, manually annotating (marking the left and right lane numbers), and then training the model. It can better handle scenes where lane lines and road edges are not obvious, as well as other complex scenes. In addition, while sharing the backbone network, the high-dimensional features of the lane lines extracted at different scales are integrated to further improve the accuracy of lane number recognition. The embodiment of the present application has also made improvements in the recognition of coarse-grained elements. The feature input of the detection head of the zebra crossing, guardrail, and curb in the coarse-grained elements is adjusted from the last feature map of the upsampling part to the second to last feature map. Since coarse-grained elements are relatively easy to learn and the misidentification of some pixels has little impact on the overall coarse-grained elements. Therefore, on the one hand, the new version of the image recognition model in the embodiment of the present application uses more model resources to learn the detection of lane lines and stop line pixels of fine-grained elements while meeting the detection accuracy of coarse-grained elements, thereby improving the detection accuracy of fine-grained elements. On the other hand, the new version of the image recognition model in the embodiment of the present application integrates the high-dimensional features of lane lines at different scales extracted, identifies the number of lanes and their corresponding confidence levels, thereby improving the accuracy of lane number recognition.

[0120] For example, Figure 12is a schematic diagram of an image processing solution provided by an embodiment of the present application, such as Figure 12 As shown, an image is input into the image recognition model, preprocessed, and then fed into the backbone network for task classification. The classified tasks include the lane marking and stop mark task head, the zebra crossing and road edge task head, and the lane number task head. The lane marking and stop mark task head determines whether a pixel is a foreground pixel. If it is not a foreground pixel, it is considered background; if it is a foreground pixel, it is a lane line pixel. It also determines the color and type of the lane line pixel. This task head classifies foreground pixels, thereby improving foreground pixel detection. The zebra crossing and road edge task head determines zebra crossings and road edges. The lane number task head determines the left and right lane numbers and their corresponding confidence levels. This image processing solution uses a single multi-task network to detect lane markings (including location, color, and type), lane number (the total number of lanes on the current road, and the lane number of the vehicle's lane, counting from the left and right sides of the road), curbs, guardrails, zebra crossings, stop marks, and guide strips. At the same time, a multi-dimensional splitting and multi-level training method is adopted for lane lines, and a multi-granularity division network design strategy is adopted for other road elements to improve the corresponding detection effect.

[0121] This embodiment of the present application not only performs fine-grained element recognition on target features, resulting in more accurate fine-grained element recognition results, but also identifies lane numbers based on features at multiple scales, thereby improving the accuracy and robustness of lane number recognition. Coarse-grained elements can also be recognized as needed to accommodate complex lane situations.

[0122] It is understandable that in the specific implementation of this application, when image data, navigation data, user information and other related data are involved, when the embodiments of this application apply the above data to specific products or technologies, it is necessary to obtain the permission or consent of the relevant users, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0123] See also Figure 13, is a structural schematic diagram of an image processing device based on an image recognition model provided by an embodiment of the present application, wherein the image recognition model includes a first feature extraction network, a second feature extraction network, a lane number recognition network, and a fine-grained element recognition network; the input of the second feature extraction network is connected to the output of the first feature extraction network; the second feature extraction network includes M feature extraction subnetworks connected in series, the scale of the features extracted by the Nth feature extraction subnetwork is larger than the scale of the features extracted by the N-1th feature extraction subnetwork, M is a positive integer greater than 1, and N is any positive integer greater than 1 and less than or equal to M; the input of the fine-grained element recognition network is connected to the output of the Mth feature extraction subnetwork, and the input of the lane number recognition network is connected to the output of each feature extraction subnetwork; the device includes:

[0124] An acquisition unit 1301 is configured to acquire an image to be processed, where the image to be processed is captured on a road where a vehicle is traveling;

[0125] The processing unit 1302 is configured to input the image to be processed into the first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed;

[0126] The processing unit 1302 is further configured to input the initial features into the second feature extraction network for processing to obtain target features processed by each feature extraction sub-network;

[0127] The processing unit 1302 is further configured to input the target features obtained by the X-th feature extraction subnetwork into the fine-grained element recognition network for processing, and determine the fine-grained element recognition results of the vehicle travel road in the image to be processed; X is a positive integer greater than or equal to 1 and less than or equal to M.

[0128] The processing unit 1302 is further configured to input all or part of the initial features and the target features obtained by processing the feature extraction subnetworks into the lane number recognition network for processing, and determine a lane number recognition result of the vehicle driving road in the image to be processed.

[0129] In one embodiment, the image recognition model also includes a coarse-grained element recognition network, the input of which is connected to the output of the MKth feature extraction subnetwork, where K is a positive integer greater than or equal to 1 and less than M, and the processing unit 1302 is also used to: input the target features obtained by processing the MKth feature extraction subnetwork into the coarse-grained element recognition network for processing, and determine the coarse-grained element recognition results of the vehicle driving road in the image to be processed; wherein the coarse-grained element recognition results include one or more of zebra crossing recognition results, guardrail recognition results, curb recognition results, and guide strip recognition results.

[0130] In one embodiment, the target features obtained by processing the Xth feature extraction subnetwork are input into the fine-grained element recognition network for processing to determine the fine-grained element recognition results of the vehicle driving road in the image to be processed, where X is a positive integer greater than or equal to 1 and less than or equal to M.

[0131] The processing unit 1302 is further configured to:

[0132] Inputting the target features obtained by processing the Xth feature extraction subnetwork into the fine-grained element recognition network for processing, and determining fine-grained foreground pixels in the image to be processed based on the target features obtained by processing the Xth feature extraction subnetwork;

[0133] determining lane line pixels based on the fine-grained foreground pixels, and performing color classification and line type classification on the lane line pixels;

[0134] A lane line recognition result is determined based on the color classification result and the line type classification result, and the lane line recognition result is determined as a fine-grained element recognition result of the vehicle driving road in the image to be processed.

[0135] In one embodiment, the device further includes a display unit 1303, and the lane number recognition result includes: the left lane number and the right lane number of the vehicle's lane from the left and the right lane number of the vehicle's driving road, a confidence level corresponding to the left lane number, and a confidence level corresponding to the right lane number; the display unit 1303 is further configured to:

[0136] The left lane number and the confidence level corresponding to the left lane number are displayed in the left position area of the lane prompt interface;

[0137] The right lane number and the confidence level corresponding to the right lane number are displayed in the right position area of the lane prompt interface.

[0138] In one embodiment, the coarse-grained element recognition result includes one or more of a guardrail recognition result and a curb recognition result, wherein the guardrail recognition result includes the distance between the vehicle and the guardrail, and the curb recognition result includes the distance between the vehicle and the curb; the processing unit 1302 is further configured to:

[0139] When it is detected based on the coarse-grained element recognition result that a warning condition is met, outputting a warning message; wherein the warning message is used to prompt the driver that there is a collision risk, and the meeting of the warning condition includes one or more of the following: the distance between the vehicle and the guardrail is less than or equal to a first distance threshold and the distance between the vehicle and the curb is less than or equal to a second distance threshold;

[0140] When it is detected based on the coarse-grained element recognition result that the vehicle control condition is met, the vehicle's driving is controlled; wherein, the vehicle control condition is met, including one or more of the distance between the vehicle and the guardrail being less than or equal to a third distance threshold and the distance between the vehicle and the curb being less than or equal to a fourth distance threshold; the third distance threshold is less than the first distance threshold, and the fourth distance threshold is less than the second distance threshold.

[0141] In one embodiment, the determining of fine-grained foreground pixels in the image to be processed based on the target features obtained by processing the Xth feature extraction subnetwork, the processing unit 1302 is specifically configured to:

[0142] Determine foreground pixels in the image to be processed based on the target features obtained by processing the Xth feature extraction subnetwork;

[0143] Performing a connected domain analysis on the foreground pixels to determine one or more connected domains formed based on the foreground pixels;

[0144] Fine-grained foreground pixels in the image to be processed are determined based on the determined one or more connected components.

[0145] In one embodiment, the image to be processed is preprocessed to obtain a preprocessed image; wherein the preprocessing includes one or more of denoising, normalization, and averaging;

[0146] The image to be processed is input into the first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed. The processing unit 1302 is specifically configured to:

[0147] The pre-processed image is input into the first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed.

[0148] It should be noted that the functions of each functional unit of the image processing device based on the image recognition model in the embodiment of the present application can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description in the above method embodiment, and will not be repeated here.

[0149] In an embodiment of the present application, the first feature extraction network is first used to extract the initial features of the vehicle driving road in the image to be processed, and then the initial features are processed using M feature extraction sub-networks connected in series to obtain target features of the vehicle driving road in the image to be processed at multiple different scales. Then, the lane number recognition result of the vehicle driving road in the image to be processed is determined by combining the initial features and all or part of the target features of multiple different scales, which can improve the accuracy of lane number recognition. In addition, the image recognition model provided in the embodiment of the present application includes multiple task networks. By inputting an image of a vehicle driving road, the recognition results of different road elements of the vehicle driving road can be determined, and the recognition efficiency is higher. At the same time, because multiple task networks share the backbone network, it can maximize the image recognition model to reduce the number of parameters of the image recognition model when performing road element recognition, and the computational efficiency is higher.

[0150] See also Figure 14 , is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device described in this embodiment includes: a processor 1401, a memory 1402, and a communication interface 1403. The processor 1401, the memory 1402, and the communication interface 1403 can exchange data.

[0151] The processor 1401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0152] The memory 1402 may include a read-only memory and a random access memory, and provides program instructions and data to the processor 1401. A portion of the memory 1402 may also include a non-volatile random access memory.

[0153] The computer device described in the embodiment of the present application includes an image recognition model, which includes a first feature extraction network, a second feature extraction network, a lane number recognition network and a fine-grained element recognition network; the input of the second feature extraction network is connected to the output of the first feature extraction network; the second feature extraction network includes M feature extraction sub-networks connected in series, the scale of the features extracted by the Nth feature extraction sub-network is greater than the scale of the features extracted by the N-1th feature extraction sub-network, M is a positive integer greater than 1, and N is any positive integer greater than 1 and less than or equal to M; the input of the fine-grained element recognition network is connected to the output of the Mth feature extraction sub-network, and the input of the lane number recognition network is connected to the output of each feature extraction sub-network.

[0154] The processor 1401 is configured to execute the following when calling the program instructions:

[0155] Acquire an image to be processed, where the image to be processed is captured on a road where a vehicle is traveling;

[0156] Inputting the image to be processed into the first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed;

[0157] Inputting the initial features into the second feature extraction network for processing to obtain target features processed by each feature extraction subnetwork;

[0158] Inputting the target feature obtained by processing the Xth feature extraction subnetwork into the fine-grained element recognition network for processing to determine a fine-grained element recognition result of the vehicle driving road in the image to be processed; X is a positive integer greater than or equal to 1 and less than or equal to M;

[0159] All or part of the initial features and the target features obtained by processing the feature extraction subnetworks are input into the lane number recognition network for processing to determine the lane number recognition result of the vehicle driving road in the image to be processed.

[0160] In one embodiment, the image recognition model also includes a coarse-grained element recognition network, the input of which is connected to the output of the MKth feature extraction subnetwork, where K is a positive integer greater than or equal to 1 and less than M, and the processor 1401 is also used to: input the target features obtained by processing the MKth feature extraction subnetwork into the coarse-grained element recognition network for processing, and determine the coarse-grained element recognition results of the vehicle driving road in the image to be processed; wherein the coarse-grained element recognition results include one or more of zebra crossing recognition results, guardrail recognition results, curb recognition results, and guide strip recognition results.

[0161] In one embodiment, the processor 1401 inputs the target feature obtained by processing the Xth feature extraction subnetwork into the fine-grained element recognition network for processing to determine a fine-grained element recognition result of the vehicle driving road in the image to be processed, where X is a positive integer greater than or equal to 1 and less than or equal to M, and is further configured to:

[0162] Inputting the target features obtained by processing the Xth feature extraction subnetwork into the fine-grained element recognition network for processing, and determining fine-grained foreground pixels in the image to be processed based on the target features obtained by processing the Xth feature extraction subnetwork;

[0163] determining lane line pixels based on the fine-grained foreground pixels, and performing color classification and line type classification on the lane line pixels;

[0164] A lane line recognition result is determined based on the color classification result and the line type classification result, and the lane line recognition result is determined as a fine-grained element recognition result of the vehicle driving road in the image to be processed.

[0165] In one embodiment, the lane number recognition result includes: the left lane number and the right lane number of the vehicle's lane from the left of the vehicle's travel road, the confidence level corresponding to the left lane number, and the confidence level corresponding to the right lane number; the processor 1401 is further configured to:

[0166] The left lane number and the confidence level corresponding to the left lane number are displayed in the left position area of the lane prompt interface;

[0167] The right lane number and the confidence level corresponding to the right lane number are displayed in the right position area of the lane prompt interface.

[0168] In one embodiment, the coarse-grained element recognition result includes one or more of a guardrail recognition result and a curb recognition result, the guardrail recognition result includes the distance between the vehicle and the guardrail, and the curb recognition result includes the distance between the vehicle and the curb; the processor 1401 is further configured to:

[0169] When it is detected based on the coarse-grained element recognition result that a warning condition is met, outputting a warning message; wherein the warning message is used to prompt the driver that there is a collision risk, and the meeting of the warning condition includes one or more of the following: the distance between the vehicle and the guardrail is less than or equal to a first distance threshold and the distance between the vehicle and the curb is less than or equal to a second distance threshold;

[0170] When it is detected based on the coarse-grained element recognition result that the vehicle control condition is met, the vehicle's driving is controlled; wherein, the vehicle control condition is met, including one or more of the distance between the vehicle and the guardrail being less than or equal to a third distance threshold and the distance between the vehicle and the curb being less than or equal to a fourth distance threshold; the third distance threshold is less than the first distance threshold, and the fourth distance threshold is less than the second distance threshold.

[0171] In one embodiment, the processor 1401 determines fine-grained foreground pixels in the image to be processed based on the target features obtained by processing the Xth feature extraction subnetwork, specifically for:

[0172] Determine foreground pixels in the image to be processed based on the target features obtained by processing the Xth feature extraction subnetwork;

[0173] Performing a connected domain analysis on the foreground pixels to determine one or more connected domains formed based on the foreground pixels;

[0174] Fine-grained foreground pixels in the image to be processed are determined based on the determined one or more connected components.

[0175] In one embodiment, the processor 1401 preprocesses the image to be processed to obtain a preprocessed image; wherein the preprocessing includes one or more of denoising, normalization, and averaging.

[0176] The step of inputting the image to be processed into the first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed is specifically used for:

[0177] The pre-processed image is input into the first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed.

[0178] In a specific implementation, the processor 1401, the communication interface 1403 and the memory 1402 described in the embodiment of the present application can execute the implementation described in the image processing method based on the image recognition model provided in the embodiment of the present application, and can also execute the following Figure 13 The implementation method described in the image processing device based on the image recognition model shown will not be repeated here.

[0179] In an embodiment of the present application, the first feature extraction network is first used to extract the initial features of the vehicle driving road in the image to be processed, and then the initial features are processed using M feature extraction sub-networks connected in series to obtain target features of the vehicle driving road in the image to be processed at multiple different scales. Then, the lane number recognition result of the vehicle driving road in the image to be processed is determined by combining the initial features and all or part of the target features of multiple different scales, which can improve the accuracy of lane number recognition. In addition, the image recognition model provided in the embodiment of the present application includes multiple task networks. By inputting an image of a vehicle driving road, the recognition results of different road elements of the vehicle driving road can be determined, and the recognition efficiency is higher. At the same time, because multiple task networks share the backbone network, it can maximize the image recognition model to reduce the number of parameters of the image recognition model when performing road element recognition, and the computational efficiency is higher.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed methods, devices, and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is merely a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical, or other forms.

[0181] The present application also provides a computer-readable storage medium having instructions stored therein. When executed on a computer, the computer implements the image processing method based on the image recognition model provided in the present application. The specific implementation method is described above and will not be repeated here.

[0182] The present application also provides a computer program product comprising a computer program or computer instructions that, when executed by a processor, implement the steps of the image processing method based on the image recognition model provided in the present application. The specific implementation method is described above and will not be repeated here.

[0183] The present application also provides a computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, thereby enabling the computer device to implement the image processing method based on the image recognition model provided in the present application. The specific implementation method can be found in the previous description and will not be repeated here.

[0184] It should be noted that for the aforementioned various method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0185] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0186] The above is a detailed introduction to an image processing method based on an image recognition model, an apparatus based on an image recognition model, a computer device, a computer-readable storage medium and a computer program product provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image processing method based on an image recognition model, characterized in that: The image recognition model includes a first feature extraction network, a second feature extraction network, a lane number recognition network, and a fine-grained element recognition network; the input of the second feature extraction network is connected to the output of the first feature extraction network; the second feature extraction network includes M feature extraction subnetworks connected in series, the scale of the features extracted by the Nth feature extraction subnetwork is larger than the scale of the features extracted by the N-1th feature extraction subnetwork, M is a positive integer greater than 1, and N is any positive integer greater than 1 and less than or equal to M; the input of the fine-grained element recognition network is connected to the output of the Mth feature extraction subnetwork, and the input of the lane number recognition network is connected to the output of each feature extraction subnetwork; the method includes: Acquire an image to be processed, where the image to be processed is captured on a road where a vehicle is traveling; Inputting the image to be processed into the first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed; Inputting the initial features into the second feature extraction network for processing to obtain target features processed by each feature extraction subnetwork; Inputting the target feature obtained by processing the Xth feature extraction subnetwork into the fine-grained element recognition network for processing to determine a fine-grained element recognition result of the vehicle driving road in the image to be processed; X is a positive integer greater than or equal to 1 and less than or equal to M; All or part of the initial features and the target features obtained by processing the feature extraction subnetworks are input into the lane number recognition network for processing to determine the lane number recognition result of the vehicle driving road in the image to be processed.

2. The method according to claim 1, wherein The image recognition model further includes a coarse-grained element recognition network, wherein an input of the coarse-grained element recognition network is connected to an output of an MKth feature extraction subnetwork, where K is a positive integer greater than or equal to 1 and less than M. The method further includes: The target features obtained by processing the MKth feature extraction subnetwork are input into the coarse-grained element recognition network for processing to determine the coarse-grained element recognition results of the vehicle driving road in the image to be processed; wherein the coarse-grained element recognition results include one or more of the zebra crossing recognition results, guardrail recognition results, curb recognition results and guide strip recognition results.

3. The method according to claim 1, wherein Inputting the target features obtained by processing the Xth feature extraction subnetwork into the fine-grained element recognition network for processing to determine the fine-grained element recognition result of the vehicle driving road in the image to be processed includes: Inputting the target features obtained by processing the Xth feature extraction subnetwork into the fine-grained element recognition network for processing, and determining fine-grained foreground pixels in the image to be processed based on the target features obtained by processing the Xth feature extraction subnetwork; determining lane line pixels based on the fine-grained foreground pixels, and performing color classification and line type classification on the lane line pixels; A lane line recognition result is determined based on the color classification result and the line type classification result, and the lane line recognition result is determined as a fine-grained element recognition result of the vehicle driving road in the image to be processed.

4. The method according to any one of claims 1 to 3, wherein The lane number recognition result includes: the left lane number and the right lane number of the vehicle's lane, the confidence level corresponding to the left lane number, and the confidence level corresponding to the right lane number. The method further includes: The left lane number and the confidence level corresponding to the left lane number are displayed in the left position area of the lane prompt interface; The right lane number and the confidence level corresponding to the right lane number are displayed in the right position area of the lane prompt interface.

5. The method according to claim 2, wherein The coarse-grained element recognition result includes one or more of a guardrail recognition result and a curb recognition result, wherein the guardrail recognition result includes the distance between the vehicle and the guardrail, and the curb recognition result includes the distance between the vehicle and the curb; the method further includes: When it is detected based on the coarse-grained element recognition result that a warning condition is met, outputting a warning message; wherein the warning message is used to prompt the driver that there is a collision risk, and the meeting of the warning condition includes one or more of the following: the distance between the vehicle and the guardrail is less than or equal to a first distance threshold and the distance between the vehicle and the curb is less than or equal to a second distance threshold; When it is detected based on the coarse-grained element recognition result that the vehicle control condition is met, the vehicle's driving is controlled; wherein, the vehicle control condition is met, including one or more of the distance between the vehicle and the guardrail being less than or equal to a third distance threshold and the distance between the vehicle and the curb being less than or equal to a fourth distance threshold; the third distance threshold is less than the first distance threshold, and the fourth distance threshold is less than the second distance threshold.

6. The method according to claim 3, wherein The determining of fine-grained foreground pixels in the image to be processed based on the target features obtained by processing the Xth feature extraction subnetwork includes: Determine foreground pixels in the image to be processed based on the target features obtained by processing the Xth feature extraction subnetwork; Performing a connected domain analysis on the foreground pixels to determine one or more connected domains formed based on the foreground pixels; Fine-grained foreground pixels in the image to be processed are determined based on the determined one or more connected components.

7. The method according to claim 1, wherein The method further comprises: Preprocessing the image to be processed to obtain a preprocessed image; wherein the preprocessing includes one or more of denoising, normalization, and averaging; The step of inputting the image to be processed into the first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed includes: The pre-processed image is input into the first feature extraction network for processing to obtain initial features of the vehicle driving road in the image to be processed.

8. An image processing device based on an image recognition model, characterized in that: The image recognition model includes a first feature extraction network, a second feature extraction network, a lane number recognition network, and a fine-grained element recognition network; the input of the second feature extraction network is connected to the output of the first feature extraction network; the second feature extraction network includes M feature extraction subnetworks connected in series, the scale of the features extracted by the Nth feature extraction subnetwork is larger than the scale of the features extracted by the N-1th feature extraction subnetwork, M is a positive integer greater than 1, and N is any positive integer greater than 1 and less than or equal to M; the input of the fine-grained element recognition network is connected to the output of the Mth feature extraction subnetwork, and the input of the lane number recognition network is connected to the output of each feature extraction subnetwork; the device includes: An acquisition unit, configured to acquire an image to be processed, wherein the image to be processed is obtained by photographing a road on which a vehicle is traveling; a processing unit, configured to input the image to be processed into the first feature extraction network for processing, and obtain initial features of the vehicle driving road in the image to be processed; The processing unit is further configured to input the initial features into the second feature extraction network for processing to obtain target features processed by each feature extraction subnetwork; The processing unit is further configured to input the target feature obtained by processing the X-th feature extraction subnetwork into the fine-grained element recognition network for processing, and determine a fine-grained element recognition result of the vehicle travel road in the image to be processed; X is a positive integer greater than or equal to 1 and less than or equal to M; The processing unit is further used to input all or part of the initial features and the target features obtained by processing the various feature extraction subnetworks into the lane number recognition network for processing, and determine the lane number recognition result of the vehicle driving road in the image to be processed.

9. A computer device, characterized in that: include: A processor, a communication interface and a memory, wherein the processor, the communication interface and the memory are interconnected, wherein the memory stores an executable program code, and the processor is used to call the executable program code to implement the image processing method based on the image recognition model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to implement the image processing method based on the image recognition model according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, the image processing method based on the image recognition model according to any one of claims 1 to 7 is implemented.

Images