Road surface element identification method and related equipment

Abstract

The present application discloses a road surface element recognition method and related equipment, which relates to the field of intelligent transportation technology. The road surface element recognition method includes: receiving a current frame image collected by a visual sensor, obtaining a preset image processing model, using the preset image processing model to perform image processing on the current frame image, respectively obtaining a current segmentation result and a current classification result, using a historical segmentation result to replace and complete the pixels of the occluded pixels and the pixels of the missing instances in the current segmentation result, and using the lane lines in the historical lane line set to complete the current lane line, and based on the completed current segmentation result, the current classification result and the completed current lane line, obtaining structured road surface element information. The pixels marked as occluded elements in the current frame image are replaced by pixels at the same position in the historical multi-frame images, the pixels of the occluded elements in the current frame image are completed, and the unoccluded current frame image is obtained, thereby obtaining complete road surface element structured information.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a road surface element recognition method and related equipment. Background Art

[0002] With the rapid development of science and technology, determining whether traffic incidents and traffic violations have occurred within a scene is becoming increasingly important in ensuring traffic safety, maintaining traffic order, improving traffic efficiency, and supporting urban planning.

[0003] In the related technology, relevant images in the traffic scene are received, road surface elements in the relevant images are identified, and road surface element information in the relevant images is obtained to further determine whether a traffic incident or traffic violation has occurred in the traffic scene. In the actual traffic scene, the assumed position of the camera is fixed, and thus, the road surface elements in the received relevant images may be blocked by foreground objects, resulting in an inability to accurately determine whether a traffic incident or traffic violation has occurred in the scene.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are related technologies. Summary of the invention

[0005] The main purpose of this application is to provide a road surface element recognition method and related equipment, aiming to solve the technical problem that the road surface elements in the relevant image may be blocked by foreground objects, resulting in the inability to accurately determine whether traffic incidents and traffic violations have occurred in the scene.

[0006] To achieve the above objectives, the present application proposes a road surface element recognition method, the method comprising:

[0007] Receive the current frame image collected by the visual sensor;

[0008] Obtain a preset image processing model, use the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image, and obtain a semantic segmentation result of the current frame image; based on the semantic segmentation result of the current frame image, use the instance segmentation branch and the classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image, and obtain an instance segmentation result of the current frame image and a classification result of the current frame image, respectively;

[0009] Use the historical multi-frame semantic segmentation results to replace the occluded pixels in the semantic segmentation results of the current frame image to obtain a multi-frame semantic segmentation result; based on the multi-frame semantic segmentation results, use the historical multi-frame instance segmentation results to complete the pixel points of the instance segmentation results missing in the instance segmentation results of the current frame image to obtain a multi-frame instance segmentation result; based on the multi-frame instance segmentation results, determine the current lane line, use the lane line in the historical lane line set to complete the current lane line, and obtain a target-level multi-frame result;

[0010] Based on the multi-frame semantic segmentation results, the multi-frame instance segmentation results, the target-level multi-frame results and the classification results, a structured analysis is performed on the road surface elements in the scene to obtain structured road surface element information.

[0011] In one embodiment, the step of replacing the occluded pixels in the semantic segmentation result of the current frame with the historical multi-frame semantic segmentation results to obtain the multi-frame semantic segmentation results further includes:

[0012] Determine pixels corresponding to the non-road surface element category in the semantic segmentation result of the current frame image as blocked pixels;

[0013] The occluded pixel is replaced with a target pixel to obtain a multi-frame semantic segmentation result, wherein the target pixel is a pixel whose semantic category last appeared at the same position as the occluded pixel and corresponds to a road surface element category.

[0014] In one embodiment, the step of obtaining the multi-frame instance segmentation result by using the historical multi-frame instance segmentation results to complete the pixel points of the missing instance segmentation results in the instance segmentation result of the current frame image based on the multi-frame semantic segmentation results also includes:

[0015] Superimposing the multi-frame semantic segmentation results onto the instance segmentation results of the current frame image and the historical multi-frame instance segmentation results, and determining an intersection area between the first instance in the instance segmentation result of the superimposed current frame image and the second instance in the superimposed historical multi-frame instance segmentation results;

[0016] If the intersection area is greater than or equal to a preset intersection area threshold, using the second instance to complete the first instance for the first time;

[0017] Unassigned pixels that are not assigned to instances in the instance segmentation result after the first completion are found, and the instance segmentation results of the neighborhood pixels corresponding to the unassigned pixels are used to perform a second completion on the unassigned pixels to obtain multi-frame instance segmentation results.

[0018] In one embodiment, the step of determining the current lane line based on the multi-frame instance segmentation result, and using the lane lines in the historical lane line set to complete the current lane line to obtain the target-level multi-frame result further includes:

[0019] Based on the multi-frame instance segmentation results, confirm a first lane line of the current frame image, and calculate a lane line similarity value between the first lane line and a plurality of lane lines in a historical lane line set;

[0020] If the maximum value of the lane line similarity value is greater than or equal to the preset lane line similarity threshold, the first lane line is added to the corresponding first lane line set; or if the maximum value of the lane line similarity value is less than the preset lane line similarity threshold, a second lane line set is created and the first lane line is added to the second lane line set;

[0021] If there is a set in which each lane line has a similarity value with the first lane line that is less than a preset lane line similarity threshold, the set is used as a third lane line set, and lane lines in the third lane line set are used to complete the missing lane lines in the replaced current frame image;

[0022] Based on the completed lane lines in the current frame image, the target-level multi-frame results are obtained.

[0023] In one embodiment, the steps of obtaining a preset image processing model, using the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image to obtain a semantic segmentation result, and based on the semantic segmentation result, using the instance segmentation branch and the classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image to obtain an instance segmentation result and a classification result, respectively, also include:

[0024] Obtain a preset image processing model, use the image processing model to perform semantic segmentation processing on the current frame image to obtain a first semantic segmentation result, and based on the first semantic segmentation result, use the image processing model to perform instance segmentation processing and classification processing on the current frame image to obtain a first instance segmentation result and a first classification result, respectively;

[0025] Based on the first instance segmentation result, selecting a first region of interest from the current frame image;

[0026] Determine a minimum width value of the lane line in the first region of interest, and determine a difference between the width value and a preset width threshold;

[0027] If the minimum width value is less than the preset width threshold, the size of the first region of interest is enlarged to the size of the current frame image, and the enlarged first region of interest is segmented and classified using the preset image processing model to obtain a second semantic segmentation result, a second instance segmentation result, and a second classification result;

[0028] Reduce the size of the second semantic segmentation result, the second instance segmentation result and the second classification result to the size of the first region of interest, use the reduced second semantic segmentation result, the second instance segmentation result and the second classification result to replace the semantic segmentation result corresponding to the first region of interest in the first semantic segmentation result to obtain the semantic segmentation result, replace the instance segmentation result corresponding to the first region of interest in the first instance segmentation result to obtain the instance segmentation result, and replace the classification result corresponding to the first region of interest in the first classification result to obtain the classification result.

[0029] In one embodiment, before the steps of obtaining a preset image processing model, using the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image to obtain a semantic segmentation result, and based on the semantic segmentation result, using the instance segmentation branch and the classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image to obtain the instance segmentation result and the classification result respectively, the steps further include:

[0030] Determine the current image processing model;

[0031] Acquire a first image sample, select an arbitrary area from the first image sample, and enlarge it to the size of the first image sample as a second image sample;

[0032] Based on the current image processing model, semantic segmentation is performed on the first image sample and the second image sample to obtain semantic segmentation results of the first image sample and the second image sample respectively; based on the obtained semantic segmentation results of the first image sample and the second image sample, instance segmentation is performed on the first image sample and the second image sample to obtain instance segmentation results of the first image sample and the second image sample respectively;

[0033] Determine a first instance feature vector of the first image and a second instance feature vector of the second image based on the obtained instance segmentation results of the first image sample and the second image sample;

[0034] Performing feature alignment on the first instance feature vector and the second instance feature vector to obtain a feature alignment loss result, determining whether the feature alignment loss result is greater than a preset feature alignment loss threshold, and if the feature alignment loss result is greater than the preset feature alignment loss threshold, adjusting parameters of the current image processing model and recording the number of parameter adjustments;

[0035] Based on the current image processing model after adjusting the parameters, return to the step of performing semantic segmentation on the first image sample and the second image sample based on the current image processing model to obtain the semantic segmentation results of the first image sample and the second image sample respectively, until the feature alignment loss result is less than or equal to the preset feature alignment loss threshold, or the number of parameter adjustments is equal to the preset number of adjustments, and select the current image processing model corresponding to the minimum value of the feature pair loss result as the preset image processing model.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a road surface element recognition device, the road surface element recognition device comprising:

[0037] A receiving module, wherein the receiving module is used to receive a current frame image collected by a visual sensor;

[0038] An image processing module, the image processing module is used to obtain a preset image processing model, use the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image, obtain a semantic segmentation result of the current frame image, and based on the semantic segmentation result of the current frame image, use the instance segmentation branch and classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image, and obtain an instance segmentation result of the current frame image and a classification result of the current frame image, respectively;

[0039] A replacement and completion module, the replacement and completion module is used to use the historical multi-frame semantic segmentation results to replace the occluded pixels in the semantic segmentation results of the current frame image to obtain a multi-frame semantic segmentation result; based on the multi-frame semantic segmentation results, use the historical multi-frame instance segmentation results to complete the pixel points of the instance segmentation results missing in the instance segmentation results of the current frame image to obtain a multi-frame instance segmentation result; based on the multi-frame instance segmentation results, determine the current lane line, use the lane line in the historical lane line set to complete the current lane line, and obtain a target-level multi-frame result;

[0040] A structured analysis module is used to perform structured analysis on road surface elements in the scene based on the multi-frame semantic segmentation results, the multi-frame instance segmentation results, the target-level multi-frame results and the classification results to obtain road surface element information in a structured form.

[0041] In one embodiment, the replacement and completion module includes:

[0042] A first determining unit, configured to determine pixels corresponding to a non-road surface element category in a semantic segmentation result of the current frame image as blocked pixels;

[0043] A replacement unit is used to replace the occluded pixel with a target pixel to obtain a multi-frame semantic segmentation result, wherein the target pixel is a pixel whose semantic category last appeared at the same position as the occluded pixel and corresponds to a road surface element category.

[0044] In one embodiment, the replacement and completion module further includes:

[0045] A superposition unit, used to superimpose the multi-frame semantic segmentation results onto the instance segmentation results of the current frame image and the historical multi-frame instance segmentation results, and determine an intersection area between a first instance in the instance segmentation result of the superimposed current frame image and a second instance in the superimposed historical multi-frame instance segmentation results;

[0046] A first completion unit, using the second instance to complete the first instance for the first time if the intersection area is greater than or equal to a preset intersection area threshold;

[0047] The second completion unit is used to find unallocated pixels that are not allocated to instances in the instance segmentation result after the first completion, and use the instance segmentation results of the neighborhood pixels corresponding to the unallocated pixels to perform a second completion on the unallocated pixels to obtain multi-frame instance segmentation results.

[0048] In one embodiment, the replacement and completion module further includes:

[0049] a similarity value calculation unit, configured to confirm a first lane line of the current frame image based on the multi-frame instance segmentation results, and calculate a lane line similarity value between the first lane line and a plurality of lane lines in a historical lane line set;

[0050] The judgment unit adds the first lane line to the corresponding first lane line set if the maximum value of the lane line similarity value is greater than or equal to the preset lane line similarity threshold; or creates a second lane line set and adds the first lane line to the second lane line set if the maximum value of the lane line similarity value is less than the preset lane line similarity threshold;

[0051] A third completion unit, if there is a set in which each lane line has a similarity value with the first lane line that is less than a preset lane line similarity threshold, takes the set as a third lane line set, and uses the lane lines in the third lane line set to complete the missing lane lines in the replaced current frame image;

[0052] The first acquisition unit is used to obtain a target-level multi-frame result based on the lane lines in the completed current frame image.

[0053] In one embodiment, the image processing module includes:

[0054] A second acquisition unit is used to acquire a preset image processing model, use the image processing model to perform semantic segmentation processing on the current frame image to obtain a first semantic segmentation result, and based on the first semantic segmentation result, use the image processing model to perform instance segmentation processing and classification processing on the current frame image to obtain a first instance segmentation result and a first classification result, respectively;

[0055] A selection unit, configured to select a first region of interest from the current frame image based on the first instance segmentation result;

[0056] A second determination unit is used to determine a minimum width value of the lane line in the first area of ​​interest, and to determine a difference between the width value and a preset width threshold;

[0057] an enlargement unit, if the minimum width value is less than the preset width threshold, enlarging the size of the first region of interest to the size of the current frame image, and using the preset image processing model to segment and classify the enlarged first region of interest to obtain a second semantic segmentation result, a second instance segmentation result, and a second classification result;

[0058] A reduction unit is used to reduce the size of the second semantic segmentation result, the second instance segmentation result and the second classification result to the size of the first region of interest, and use the reduced second semantic segmentation result, the second instance segmentation result and the second classification result to replace the semantic segmentation result corresponding to the first region of interest in the first semantic segmentation result to obtain the semantic segmentation result, replace the instance segmentation result corresponding to the first region of interest in the first instance segmentation result to obtain the instance segmentation result, and replace the classification result corresponding to the first region of interest in the first classification result to obtain the classification result.

[0059] In one embodiment, the image processing module further includes:

[0060] A third determining unit, used to determine a current image processing model;

[0061] a third acquisition unit, configured to acquire a first image sample, select an arbitrary area from the first image sample, and enlarge the area to a size of the first image sample as a second image sample;

[0062] an image processing unit, configured to perform semantic segmentation on the first image sample and the second image sample based on the current image processing model to obtain semantic segmentation results of the first image sample and the second image sample respectively, and perform instance segmentation on the first image sample and the second image sample based on the obtained semantic segmentation results of the first image sample and the second image sample to obtain instance segmentation results of the first image sample and the second image sample respectively;

[0063] a fourth determining unit, configured to determine a first instance feature vector of the first image and a second instance feature vector of the second image based on the obtained instance segmentation results of the first image sample and the second image sample;

[0064] a feature alignment unit, configured to perform feature alignment on the first instance feature vector and the second instance feature vector to obtain a feature alignment loss result, determine whether the feature alignment loss result is greater than a preset feature alignment loss threshold, and if the feature alignment loss result is greater than the preset feature alignment loss threshold, adjust the parameters of the current image processing model and record the number of parameter adjustments;

[0065] A training unit is used to return to the step of performing semantic segmentation on the first image sample and the second image sample based on the current image processing model after adjusting the parameters, and respectively obtain the semantic segmentation results of the first image sample and the second image sample, until the feature alignment loss result is less than or equal to the preset feature alignment loss threshold, or the number of parameter adjustments is equal to the preset number of adjustments, and the current image processing model corresponding to the minimum value in the feature pair loss result is selected as the preset image processing model.

[0066] In addition, to achieve the above-mentioned purpose, the present application also proposes a road surface element identification device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the road surface element identification method described above.

[0067] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the road surface element identification method described above are implemented.

[0068] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the road surface element identification method described above are implemented.

[0069] One or more technical solutions proposed in this application have at least the following technical effects:

[0070] The present application proposes a road surface element recognition method and related equipment. In the related technology, when determining whether a traffic incident or traffic violation occurs in a traffic scene, the road surface elements in the related image may be blocked by the foreground target, resulting in an inability to accurately determine whether a traffic incident or traffic violation occurs in the scene. In the present application, in a traffic scene, first, a current frame image captured by a visual sensor is received, and then a preset image processing model is obtained. The semantic branch of the preset image processing model is used to perform semantic segmentation on the current frame image to obtain the semantic segmentation result of the current frame image. Based on the semantic segmentation result of the current frame image, the instance segmentation branch and classification branch of the preset image processing model are used to perform instance segmentation and classification on the current frame image to obtain the instance segmentation and classification results of the current frame image, respectively. The example segmentation result and the classification result of the current frame image are compared. Further, the occluded pixels in the semantic segmentation result of the current frame image are replaced with the historical multi-frame semantic segmentation results to obtain the multi-frame semantic segmentation result; based on the multi-frame semantic segmentation result, the historical multi-frame instance segmentation results are used to complete the pixel points of the missing instance segmentation result in the instance segmentation result of the current frame image to obtain the multi-frame instance segmentation result; based on the multi-frame instance segmentation result, the current lane line is determined, and the lane line in the historical lane line set is used to complete the current lane line to obtain the target-level multi-frame result; finally, based on the multi-frame semantic segmentation result, the multi-frame instance segmentation result, the target-level multi-frame result and the classification result, the road surface elements in the scene are structured analyzed to obtain the road surface element information in a structured form.

[0071] It can be understood that at the same position, the road surface element information will not change. The pixels marked as occluded elements in the current frame image are replaced by the pixels at the same position in the historical multi-frame images, and the pixels of the occluded elements in the current frame image are completed to obtain the unoccluded current frame image, and further, obtain the complete road surface element structured information. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0074] Figure 1A schematic diagram of a road surface element recognition method flow chart provided in Example 1 of the present application;

[0075] Figure 2 A flowchart of semantic-level multi-frame statistical processing provided in Example 1 of the present application;

[0076] Figure 3 A schematic diagram of the process of instance-level multi-frame statistical processing provided in Embodiment 1 of the present application;

[0077] Figure 4 A schematic diagram of the process flow of target-level multi-frame statistical processing provided in Example 1 of the present application;

[0078] Figure 5 A schematic diagram of the process of building and training a semantic segmentation model provided in Example 2 of the present application;

[0079] Figure 6 This is a schematic diagram of the module structure of the road surface element recognition device in an embodiment of the present application;

[0080] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the road surface element recognition method in the embodiment of the present application.

[0081] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0082] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0083] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0084] The main solution of the embodiment of the present application is: first, receive the current frame image collected by the visual sensor, then obtain a preset image processing model, use the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image, and obtain the semantic segmentation result of the current frame image; based on the semantic segmentation result of the current frame image, use the instance segmentation branch and classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image, and obtain the instance segmentation result of the current frame image and the classification result of the current frame image respectively; further, use the historical multi-frame semantic segmentation results to replace the semantic segmentation of the current frame image The occluded pixels in the result are obtained to obtain a multi-frame semantic segmentation result; based on the multi-frame semantic segmentation result, the historical multi-frame instance segmentation results are used to complete the pixel points of the missing instance segmentation result in the instance segmentation result of the current frame image to obtain a multi-frame instance segmentation result; based on the multi-frame instance segmentation result, the current lane line is determined, and the lane line in the historical lane line set is used to complete the current lane line to obtain a target-level multi-frame result; finally, based on the multi-frame semantic segmentation result, the multi-frame instance segmentation result, the target-level multi-frame result and the classification result, a structured analysis is performed on the road surface elements in the scene to obtain road surface element information in a structured form.

[0085] It can be understood that at the same position, the road surface element information will not change. The pixels marked as occluded elements in the current frame image are replaced by the pixels at the same position in the historical multi-frame images, and the pixels of the occluded elements in the current frame image are completed to obtain the unoccluded current frame image, and further, obtain the complete road surface element structured information.

[0086] In this embodiment, for the convenience of description, the following description is made by taking the road surface element recognition device as the execution subject.

[0087] In the related technology, when road surface elements are identified in traffic scenes, foreground elements such as people and vehicles in the traffic scenes will block the road surface elements, resulting in a technical problem that it is impossible to accurately determine whether a traffic incident or traffic violation has occurred in the scene.

[0088] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a road surface element recognition device, etc. The following takes the road surface element recognition device as an example to illustrate this embodiment and the following embodiments.

[0089] Based on this, the present application embodiment provides a road surface element recognition method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the road surface element recognition method of the present application.

[0090] In this embodiment, the road surface element recognition method includes steps S100 to S400:

[0091] Step S100, receiving a current frame image collected by a visual sensor;

[0092] In this embodiment, the execution subject is a road surface element recognition device, which can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The road surface element recognition device receives the road surface element information sent by the sensor device.

[0093] It should be noted that, in this embodiment, the road surface elements include green belts, zebra crossings, guide arrows, lane lines, etc.

[0094] In this embodiment, the specific application scenario may be:

[0095] In a road section, a road section image at the current moment is collected by a visual sensor, wherein the road section image at the current moment includes road surface element information of the road section at the current moment.

[0096] The road surface element recognition method provided in the embodiment of the present application obtains traffic scene information by receiving the current frame image collected by the visual sensor. The current frame image includes one or more images of road surface elements. It is understandable that the current frame image provides a data basis for subsequent steps.

[0097] Step S200, obtaining a preset image processing model, using the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image, obtaining a semantic segmentation result of the current frame image, and based on the semantic segmentation result of the current frame image, using the instance segmentation branch and the classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image, obtaining an instance segmentation result of the current frame image and a classification result of the current frame image, respectively;

[0098] In this embodiment, first, the road element recognition device performs semantic segmentation processing on the current frame image through the semantic segmentation branch of the preset image processing model, realizes semantic classification and labeling of pixels in the current frame image, and obtains the semantic segmentation result of the current frame image.

[0099] Among them, the preset image processing model is the current image processing model after training, and the preset image processing model includes a semantic segmentation branch, an instance segmentation branch and a classification branch.

[0100] It can be understood that the pixels in the current frame image are semantically classified and labeled to obtain the semantic segmentation result of the current frame image, so the semantic category of each pixel point can be directly obtained from the semantic segmentation result of the current frame image.

[0101] It should be noted that in the field of computer vision, semantic segmentation is a crucial technology, which is mainly used to classify each pixel in an image into different semantic categories, achieve accurate classification and labeling at the pixel level, and obtain the semantic segmentation results of the image. Among them, in computer vision, the semantic category of a pixel refers to assigning each pixel in an image to a preset category, which usually represents different objects or scene elements. For example, in an image of a traffic scene, the semantic categories of pixels may include "lane line", "zebra crossing", "green belt", "guide arrow", etc. Each pixel is labeled as one of these semantic categories, thereby providing a detailed semantic label map for the entire image. Semantic segmentation can help computers read the information of an image and perform various tasks based on the semantic label map of the image information, such as target detection tasks, scene understanding tasks, image segmentation tasks, etc. Computer devices usually perform semantic segmentation on images based on deep learning models to obtain semantic label maps of images, wherein deep learning models include convolutional neural network models. These deep learning models can learn the visual features of road surface elements in images and associate these features with preset categories.

[0102] It should also be noted that the semantic segmentation result of the current frame image is the semantic segmentation mask of the current frame image. Mask is a concept in image processing, usually a matrix or image of the same size as the original image, used to indicate or identify a specific pixel area in the original image and mark each pixel point in the specific pixel area.

[0103] After obtaining the semantic segmentation result of the current frame image, based on the semantic segmentation result of the current frame image, the road surface element recognition device performs instance segmentation processing and classification processing on the current frame image through the instance segmentation branch and classification branch of the preset image processing model, respectively, to achieve instance classification marking and lane line type classification of the pixels in the current frame image, and obtain the instance segmentation result and classification result of the current frame image.

[0104] It should be noted that instance segmentation is to classify and label the current frame image at the pixel level based on the semantic segmentation results. Instance segmentation is an important technology in computer vision, which involves segmenting each target (instance) in the image and assigning a unique label to each target. This means that it is necessary not only to identify different targets in the image, but also to separate each target from the background. Instance segmentation is a subset of semantic segmentation. Semantic segmentation is to classify each pixel in the image into predefined categories, while instance segmentation further distinguishes different instances of the same category.

[0105] In addition, it should be noted that the road surface elements of the same instance category also have different types, which need to be further classified and labeled using the classification branch. For example, the types of lane lines include solid white lines, dashed white lines, single yellow lines, double yellow lines, etc., and the types of guide arrows include straight ahead, left turn, right turn, right turn and straight ahead, etc. The classification branch makes pixel-level category predictions for the decoder output feature map that is downsampled N times (N can be 2, 4, 8, 16), which can be equivalent to gridding predictions for the input image. Specifically, the prediction of the decoder output feature map that is downsampled N times can be regarded as the input image being divided into a grid of size N*N. When there are semantic segmentation pixels of a preset road surface element in the grid area, the type prediction result of the road surface element is output. It is worth noting that each grid can predict the type information of different road surface elements, that is, when there are semantic segmentation pixels of lane lines and guide arrows in the grid at the same time, the grid can simultaneously give prediction results of the lane line type and the guide arrow type.

[0106] It can be understood that the classification branch also classifies the image based on the semantic segmentation result.

[0107] In this embodiment, the current frame image is subjected to semantic segmentation processing by a preset image processing model, and further, the current frame image is subjected to instance segmentation processing and classification processing to obtain the semantic segmentation result, instance segmentation result and classification result of the current frame image. The semantic segmentation result, instance segmentation result and classification result can help the road surface element recognition device read the image information and perform various tasks based on the image information.

[0108] Specifically, the step of obtaining a preset image processing model, using the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image to obtain a semantic segmentation result, and based on the semantic segmentation result, using the instance segmentation branch and the classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image to obtain an instance segmentation result and a classification result, respectively, also includes steps S210 to S250:

[0109] Step S210, obtaining a preset image processing model, using the image processing model to perform semantic segmentation processing on the current frame image to obtain a first semantic segmentation result, and based on the first semantic segmentation result, using the image processing model to perform instance segmentation processing and classification processing on the current frame image to obtain a first instance segmentation result and a first classification result, respectively;

[0110] Step S220, determining a minimum width value of the lane line in the first region of interest, and judging a difference between the width value and a preset width threshold;

[0111] It should be noted that if the minimum width value is smaller than the preset width threshold, it indicates that there are still thinner lane lines in the current frame image, and the perception of the thinner lane lines is relatively vague, requiring further refinement.

[0112] Step S230, if the minimum width value is less than the preset width threshold, enlarge the size of the first region of interest to the size of the current frame image, and use the preset image processing model to segment and classify the enlarged first region of interest to obtain a second semantic segmentation result, a second instance segmentation result, and a second classification result;

[0113] It should be noted that the first region of interest is ROI (region of interest). In the fields of machine vision, image processing, etc., the area to be processed is outlined in the image to be processed in the form of a box, circle, ellipse, irregular polygon, etc., which is called the region of interest. In this application, the selected ROI area is a rectangle, and the aspect ratio of the ROI area is the same as the aspect ratio of the current frame image.

[0114] Specifically, the steps of selecting the first region of interest from the current frame image are as follows:

[0115] Firstly, based on the current frame image, a plane rectangular coordinate system is established to obtain the coordinates of each pixel point in the current frame image.

[0116] After obtaining the coordinates of each pixel point in the current frame image, the minimum and maximum values ​​of the normalized vertical coordinates of the road surface elements in the current frame image are calculated:

[0117] ;

[0118] .

[0119] Among them, it should be noted that Indicates the minimum value of the normalized vertical coordinate of the ROI area. Indicates the maximum value of the normalized vertical coordinate of the ROI area. represents the normalized ordinate of the road pixel point obtained based on the semantic segmentation result of the current frame image, represents the normalized vertical coordinate of the lane line pixel point obtained based on the semantic segmentation result of the current frame image, is the area expansion parameter (the optional interval is [0, 0.25], a better choice is 0.05), Represents the perception confidence of the far lane line (the optional interval is [0.1, 0.5], and a better choice is 0.3).

[0120] After obtaining the minimum and maximum values ​​of the normalized ordinates of the road surface elements, the difference of the normalized ordinates is obtained, and then the minimum and maximum values ​​of the normalized abscissas of the road surface elements in the current frame image are determined. Specifically, the steps are as follows:

[0121] ;

[0122] ;

[0123] ;

[0124] Among them, it should be noted that represents the normalized horizontal coordinate of the lane line pixel point obtained based on the semantic segmentation result of the current frame image, It represents the center point of the normalized horizontal coordinate of the lane line pixel points obtained based on the semantic segmentation result of the current frame image, which is calculated according to the distribution of the lane line pixel points in the area.

[0125] Finally, according to the calculated , , and, , select the ROI area.

[0126] It can be understood that the region of interest is the remote region (region with poor perception effect) in the current frame image.

[0127] Step S240, enlarging the size of the first region of interest to the size of the current frame image, and using the preset image processing model to segment and classify the enlarged first region of interest to obtain a second semantic segmentation result, a second instance segmentation result, and a second classification result;

[0128] Step S250, reducing the size of the second semantic segmentation result, the second instance segmentation result and the second classification result to the size of the first region of interest, using the reduced second semantic segmentation result, the second instance segmentation result and the second classification result to replace the semantic segmentation result corresponding to the first region of interest in the first semantic segmentation result to obtain the semantic segmentation result, replace the instance segmentation result corresponding to the first region of interest in the first instance segmentation result to obtain the instance segmentation result, and replace the classification result corresponding to the first region of interest in the first classification result to obtain the classification result.

[0129] In this embodiment, the image in the ROI region is enlarged to the current frame image size, and the preset image model is re-input to obtain the second semantic segmentation result, the second instance segmentation result, and the second classification result. Then, the second semantic segmentation result, the second instance segmentation result, and the second classification result are reduced to the original ROI region size and spliced ​​back to the first semantic segmentation result, the first instance segmentation result, and the classification result, respectively, to achieve refined perception of remote small targets.

[0130] It should be noted that, for the same embodiment, the preset image processing model can obtain an approximate feature vector from the current frame image and the ROI region.

[0131] Step S300, using the historical multi-frame semantic segmentation results to replace the occluded pixels in the semantic segmentation results of the current frame image, to obtain a multi-frame semantic segmentation result; based on the multi-frame semantic segmentation results, using the historical multi-frame instance segmentation results to complete the pixel points of the instance segmentation results missing in the instance segmentation results of the current frame image, to obtain a multi-frame instance segmentation result; based on the multi-frame instance segmentation results, determining the current lane line, using the lane line in the historical lane line set to complete the current lane line, to obtain a target-level multi-frame result;

[0132] Specifically, the step of replacing the occluded pixels in the semantic segmentation result of the current frame with the historical multi-frame semantic segmentation results to obtain the multi-frame semantic segmentation results also includes steps S310 to S320:

[0133] Step S310, determining pixels corresponding to the non-road element category in the semantic segmentation result of the current frame image as blocked pixels;

[0134] In this embodiment, first, according to the semantic segmentation result of the current frame image, the pixels corresponding to the non-road element category in the current frame image are determined as blocked pixels. Specifically, reference may be made to Figure 2 .

[0135] It can be understood that if the semantic category of a pixel is a non-road element category, it means that the road element corresponding to the pixel is blocked, that is, there is a foreground object blocking the road element.

[0136] Step S320, replacing the occluded pixel with a target pixel to obtain a multi-frame semantic segmentation result, wherein the target pixel is a pixel corresponding to a road surface element category that appeared last time at the same position as the occluded pixel.

[0137] It should be noted that after the occluded pixels are replaced with the target pixels, the semantic segmentation results of the consecutive K frames are counted again, and the statistical results of the multiple frames obtained are saved as new multi-frame semantic segmentation results.

[0138] In addition, it should be noted that the statistical method for multi-frame statistical processing in this application is sliding window statistics. Sliding window statistics (Sliding Window Algorithm) is an algorithm for processing continuous data sequences, which calculates the statistics of the data sequence by moving on a fixed-size window. This algorithm has applications in many fields, such as network traffic analysis, stock market analysis, data mining, etc. Sliding window statistics algorithm is an algorithm for processing continuous data sequences, which has applications in many fields, such as network traffic analysis, stock market analysis, data mining, etc.

[0139] Suppose there is a data sequence: a1, a2, a3...a n , and want to calculate statistics for each window. You can define a window size of 3, then for each window, you can calculate statistics such as mean, median, maximum, etc.

[0140] Specifically, the steps of sliding window statistics include:

[0141] Based on each pixel in the semantic segmentation result, obtain the semantic segmentation results of K consecutive frames (based on the algorithm time consumption and device performance requirements, a better choice is K=10) of the pixel, and take the road element that appears most times in the semantic segmentation results of K consecutive frames as the multi-frame statistical result of the pixel.

[0142] In this embodiment, pixels in the multi-frame statistical results are used to replace the blocked pixels in the current frame to obtain an unblocked current frame image.

[0143] Additionally, refer to Figure 3 The step of using historical multi-frame instance segmentation results to complete the pixel points of the missing instance segmentation results in the instance segmentation results of the current frame image based on the multi-frame semantic segmentation results to obtain the multi-frame instance segmentation results also includes steps S330 to S350:

[0144] Step S330, superimposing the multi-frame semantic segmentation results onto the instance segmentation results of the current frame image and the historical multi-frame instance segmentation results, and determining an intersection area between the first instance in the instance segmentation result of the superimposed current frame image and the second instance in the superimposed historical multi-frame instance segmentation results;

[0145] It can be understood that if an intersection area exists, then the intersection area is generated when the first instance and the second instance are instances of the same instance category.

[0146] It should be noted that instance segmentation is based on the semantic segmentation result. There are occluded pixels in the semantic segmentation result of the current frame image. Furthermore, there are occluded pixels in the instance segmentation result of the current frame image. The completed semantic segmentation result is superimposed on the instance segmentation result of the current frame image to complete the semantic category label in the instance segmentation result of the current frame image.

[0147] Step S340: If the intersection area is greater than or equal to a preset intersection area threshold, use the second instance to complete the first instance for the first time;

[0148] It is understandable that if the intersection area is greater than or equal to the preset intersection area threshold, it indicates that the two instances are the same instance, and the first instance can be completed using the second instance for the first time, so that the instance category can be unified and the first instance can be completed. For example, in the historical multi-frame instance segmentation results, the instance category of the second instance may be lane line 1, and in the first instance segmentation results, the instance category of the first instance may be lane line 2. If the intersection area is greater than or equal to the preset intersection area threshold, it indicates that lane 1 and lane 2 are the same lane line, and the unification of lane line 2 using lane line 1 also changes the instance category of the first instance from lane line 2 to lane line 1.

[0149] Step S350, searching for unallocated pixels that are not allocated to instances in the instance segmentation result after the first completion, using the instance segmentation results of the neighborhood pixels corresponding to the unallocated pixels, performing a second completion on the unallocated pixels to obtain a multi-frame instance segmentation result.

[0150] It is understandable that, due to environmental factors, road surface elements in the image captured by the visual sensor may be missing. For example, at night, when there is no light, the visual sensor may be affected by light when capturing images, and road surface elements in the captured image may be missing.

[0151] In this embodiment, unallocated pixels that are not allocated to instances in the instance segmentation results after the first completion are found, and the instance segmentation results of the neighborhood pixels corresponding to the unallocated pixels are used to perform a second completion on the unallocated pixels to obtain multi-frame instance segmentation results.

[0152] Additionally, refer to Figure 4 The step of determining the current lane line based on the multi-frame instance segmentation result, using the lane lines in the historical lane line set to complete the current lane line, and obtaining the target-level multi-frame result also includes steps S360 to S390:

[0153] Step S360, based on the multi-frame instance segmentation results, confirming the first lane line of the current frame image, and calculating the lane line similarity values ​​between the first lane line and multiple lane lines in the historical lane line set;

[0154] It should be noted that, in this embodiment, a lane line skeleton is used to represent the lane line, and then the similarity between the lane lines is calculated. Specifically, the process of constructing the lane line skeleton is as follows:

[0155] Taking the first lane line as an example, based on the second instance segmentation result, along the Y-axis direction, the midpoint of all the first lane line pixels corresponding to each Y coordinate is used as the skeleton point, all the skeleton points of the first lane line are extracted, and the lane line skeleton is constructed. It can be understood that the first lane line can be any lane line in the current frame image.

[0156] Specifically, the specific method for calculating the lane line similarity value is as follows:

[0157] First, the lane line skeletons of the two lane lines whose similarity needs to be calculated are obtained, and then the lane line skeleton similarity value is calculated using a preset lane line skeleton similarity algorithm.

[0158] Among them, the preset lane line skeleton similarity algorithm is:

[0159] ;

[0160] in, represents the lane line skeleton similarity value, is the average value of the X-coordinate difference of each skeleton point along the Y-axis direction of the two lane line skeletons, is the preset lane line skeleton similarity threshold, the value range is (0, W), W is the width of the pixel in the image to be processed, and a better value is 25.

[0161] Step S370: if the maximum value of the lane line similarity value is greater than or equal to the preset lane line similarity threshold, the first lane line is added to the corresponding first lane line set; or if the maximum value of the lane line similarity value is less than the preset lane line similarity threshold, a second lane line set is created, and the first lane line is added to the second lane line set;

[0162] It should be noted that for each lane line, there is an associated lane line set. In the lane line set, there are multiple lane lines with high similarity to the associated lane line. When the maximum value of the lane line similarity value is greater than or equal to the preset lane line similarity threshold, it indicates that the first lane line is associated with the first lane line set, and the first lane line is added to the corresponding first lane line set.

[0163] When the maximum value of the lane line similarity value is less than the preset lane line similarity threshold, it indicates that there is no association between the first lane line and the first lane line set. Further, it indicates that the first lane line is a new lane line. At this time, a second lane line set associated with the first lane line is newly created, and the first lane line is added to the second lane line set.

[0164] In addition, it should be noted that after all lane lines in the current frame image are merged into a set, the average value of the lane line skeleton in each set is calculated. After the average value is obtained, the similarity relationship between each lane line skeleton in the lane line set and the average value of the lane line skeleton is calculated, and n lane lines with higher similarity are retained to reduce the amount of calculation and improve calculation efficiency.

[0165] Step S380: If there is a set in which each lane line has a similarity value with the first lane line that is less than a preset lane line similarity threshold, the set is used as a third lane line set, and lane lines in the third lane line set are used to complete the missing lane lines in the replaced current frame image;

[0166] In this embodiment, if the maximum value of the lane line similarity value is greater than or equal to the preset lane line similarity threshold, the first lane line is added to the corresponding first lane line set; or if the maximum value of the lane line similarity value is less than the preset lane line similarity threshold, a new second lane line set is created and the first lane line is added to the second lane line set.

[0167] Step S390, obtaining a target-level multi-frame result based on the completed lane lines in the current frame image.

[0168] Step S400, based on the multi-frame semantic segmentation results, the multi-frame instance segmentation results, the target-level multi-frame results and the classification results, a structured analysis is performed on the road surface elements in the scene to obtain road surface element information in a structured form.

[0169] In this embodiment, based on the multi-frame semantic segmentation results, the multi-frame instance segmentation results, the target-level multi-frame results and the classification results, a contour simplification algorithm is used to perform structured processing on the completed current lane line to obtain structured road surface element information.

[0170] It should be noted that the contour simplification algorithm usually refers to an algorithm used to reduce the complexity of polygons or curves. This algorithm is very useful in fields such as computer graphics, geographic information systems (GIS), and computer vision. Using the contour simplification algorithm to process images can reduce the number of vertices while maintaining the original shape features, thereby reducing the capacity of stored data and speeding up data processing. By reducing the storage capacity of data, storage space can be saved, storage costs can be reduced, and data processing efficiency can be improved. At the same time, speeding up data processing can reduce waiting time, improve system response speed and user experience. There are many contour simplification algorithms, some of which are common methods include: Douglas-Peucker algorithm, Ramer-Douglas-Peucker algorithm, Visvalingam-Whyatt algorithm.

[0171] In addition, it should be noted that the contour simplification algorithm used in this application is the Douglas-Peucker algorithm. The Douglas-Peucker algorithm, also known as the Larmer-Douglas-Peucker algorithm, is an algorithm for simplifying curves. The algorithm approximates the curve by reducing the number of points representing the curve, thereby achieving the purpose of data compression. Specifically, the basic principle of the Douglas-Peucker algorithm is as follows:

[0172] Based on a curve, first set the first and last points of the curve and calculate the straight line between the two points. Then, calculate the distance between all points on the curve and the straight line and find the maximum distance. If the maximum distance is less than or equal to the given threshold, only keep the first and last points. If the maximum distance is greater than the given threshold, divide the curve into two parts at the maximum distance point, and repeat the above "set the first and last points of the curve and calculate the straight line between the two points" steps for each part of the curve until the maximum distance between all points on the curve and the straight line is less than or equal to the given threshold.

[0173] In this embodiment, the Douglas-Peucker algorithm is used to fit the linear road surface elements into a broken line segment form, and the nonlinear road surface elements into a polygonal form.

[0174] The present application proposes a road surface element recognition method and related equipment. In the related technology, when determining whether a traffic incident or traffic violation occurs in a traffic scene, the road surface elements in the related image may be blocked by the foreground target, resulting in an inability to accurately determine whether a traffic incident or traffic violation occurs in the scene. In the present application, in a traffic scene, first, a current frame image captured by a visual sensor is received, and then a preset image processing model is obtained. The semantic branch of the preset image processing model is used to perform semantic segmentation on the current frame image to obtain the semantic segmentation result of the current frame image. Based on the semantic segmentation result of the current frame image, the instance segmentation branch and classification branch of the preset image processing model are used to perform instance segmentation and classification on the current frame image to obtain the instance segmentation and classification results of the current frame image, respectively. The example segmentation result and the classification result of the current frame image are compared. Further, the occluded pixels in the semantic segmentation result of the current frame image are replaced with the historical multi-frame semantic segmentation results to obtain the multi-frame semantic segmentation result; based on the multi-frame semantic segmentation result, the historical multi-frame instance segmentation results are used to complete the pixel points of the missing instance segmentation result in the instance segmentation result of the current frame image to obtain the multi-frame instance segmentation result; based on the multi-frame instance segmentation result, the current lane line is determined, and the lane line in the historical lane line set is used to complete the current lane line to obtain the target-level multi-frame result; finally, based on the multi-frame semantic segmentation result, the multi-frame instance segmentation result, the target-level multi-frame result and the classification result, the road surface elements in the scene are structured analyzed to obtain the road surface element information in a structured form.

[0175] It can be understood that at the same position, the road surface element information will not change. The pixels marked as occluded elements in the current frame image are replaced by the pixels at the same position in the historical multi-frame images, and the pixels of the occluded elements in the current frame image are completed to obtain the unoccluded current frame image, and further, obtain the complete road surface element structured information.

[0176] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 5 , before step S200, the step of obtaining a preset image processing model, using the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image to obtain a semantic segmentation result, and based on the semantic segmentation result, using the instance segmentation branch and the classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image to obtain the instance segmentation result and the classification result, respectively, also includes steps S210~S260:

[0177] Step S210, determining the current image processing model;

[0178] In this embodiment, first, a semantic segmentation model is selected, and the semantic segmentation model includes a semantic segmentation branch, an instance segmentation branch, a classification branch, and a feature alignment branch.

[0179] It should be noted that the semantic segmentation branch is used to perform semantic segmentation on the image and obtain semantic segmentation results; the instance segmentation branch is used to perform instance segmentation on the image and obtain instance segmentation results; the classification branch is used to further classify the instances in the image; the feature alignment branch is used to align the feature vector of the first image with the feature vector of the second image when training the model, and obtain the feature alignment loss result.

[0180] Step S220, obtaining a first image sample, selecting an arbitrary area from the first image sample, and enlarging the area to the size of the first image sample as a second image sample;

[0181] It should be noted that the aspect ratio of the arbitrary region is the same as the aspect ratio of the first image sample.

[0182] In addition, it should be noted that the first image sample is an image sample containing road surface element information and is used for training the model.

[0183] In this embodiment, a first image sample is obtained, and an arbitrary area is selected from the first image sample and enlarged to the size of the first image sample as the second image sample.

[0184] Step S230: Based on the current image processing model, semantic segmentation is performed on the first image sample and the second image sample to obtain semantic segmentation results of the first image sample and the second image sample respectively; based on the obtained semantic segmentation results of the first image sample and the second image sample, instance segmentation is performed on the first image sample and the second image sample to obtain instance segmentation results of the first image sample and the second image sample respectively;

[0185] Step S240, determining a first instance feature vector of the first image and a second instance feature vector of the second image based on the obtained instance segmentation results of the first image sample and the second image sample;

[0186] Step S250, aligning the first instance feature vector with the second instance feature vector to obtain a feature alignment loss result, and determining whether the feature alignment loss result is greater than a preset feature alignment loss threshold. If the feature alignment loss result is greater than the preset feature alignment loss threshold, adjusting the parameters of the current image processing model, and recording the number of parameter adjustments.

[0187] It should be noted that the step of using the feature alignment branch in the current semantic segmentation model to perform feature alignment on the first instance feature vector and the second instance feature vector to obtain a feature alignment loss result includes:

[0188] First, based on a preset feature alignment loss function, feature alignment is performed on the first instance feature vector and the second instance feature vector to obtain a feature alignment loss result;

[0189] It should be noted that the preset feature alignment loss function is:

[0190] ;

[0191] in, and Respectively represent the number of instances of the specified road surface elements in the ROI area image (second image) and the original image (first image). and Respectively represent the number of pixels contained in the specified instance c in the ROI area image and the original image, and Respectively represent the feature vector mean of all pixels of the specified instance c in the ROI area image and the original image, and They represent the feature vectors of the specified pixels of the specified instance in the ROI area image and the original image respectively. Represents the feature vector similarity threshold.

[0192] It should be noted that is the feature alignment loss result, represents a first feature alignment loss result obtained by aligning the second instance feature vector with the first instance feature vector, represents performing feature alignment on the first instance feature vector and the second instance feature vector to obtain a second feature alignment loss result, and adding the first feature alignment loss result to the second feature alignment loss result to obtain the feature alignment loss result ( ).

[0193] In addition, it should be noted that when " " is less than or equal to 0, " " is calculated as 0. Similarly, when " " is less than or equal to 0, " " evaluates to 0." "express" ” is the L2 norm of .

[0194] It can be understood that the feature alignment loss function constrains the pixel areas of the same instance in the same scene in the two images to have similar feature vector results, that is, in the same image sample, for the same target in input images of different scales, the image processing model can learn approximate features, which helps to maintain the consistency of features of the same target under inputs of different scales, so that the instance feature vector learned by the current semantic segmentation model has scale invariance.

[0195] Step S260, based on the current image processing model after adjusting the parameters, returns to the step of performing semantic segmentation on the first image sample and the second image sample based on the current image processing model to obtain the semantic segmentation results of the first image sample and the second image sample respectively, until the feature alignment loss result is less than or equal to the preset feature alignment loss threshold, or the number of parameter adjustments is equal to the preset number of adjustments, and the current image processing model corresponding to the minimum value in the feature pair loss result is selected as the preset image processing model.

[0196] It should be noted that in addition to the feature alignment loss function, the training process also includes the cross entropy loss function of semantic segmentation, the cross entropy loss function of classification, etc. The weighted sum of all loss functions is used as the final loss function. When the final loss function satisfies the condition that it is less than or equal to the preset loss threshold, the current image processing model is set to the preset image processing model.

[0197] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the road surface element identification method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0198] This application also provides a road surface element recognition device, please refer to Figure 6 , the road surface element recognition device comprises:

[0199] A receiving module 10, the receiving module is used to receive the current frame image collected by the visual sensor;

[0200] An image processing module 20, the image processing module is used to obtain a preset image processing model, use the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image, obtain the semantic segmentation result of the current frame image, and based on the semantic segmentation result of the current frame image, use the instance segmentation branch and the classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image, and obtain the instance segmentation result of the current frame image and the classification result of the current frame image respectively;

[0201] A replacement and completion module 30, the replacement and completion module is used to use the historical multi-frame semantic segmentation results to replace the occluded pixels in the semantic segmentation results of the current frame image to obtain a multi-frame semantic segmentation result; based on the multi-frame semantic segmentation results, use the historical multi-frame instance segmentation results to complete the pixel points of the instance segmentation results missing in the instance segmentation results of the current frame image to obtain a multi-frame instance segmentation result; based on the multi-frame instance segmentation results, determine the current lane line, use the lane line in the historical lane line set to complete the current lane line, and obtain a target-level multi-frame result;

[0202] The structured analysis module 40 is used to perform structured analysis on the road surface elements in the scene based on the multi-frame semantic segmentation results, the multi-frame instance segmentation results, the target-level multi-frame results and the classification results to obtain structured road surface element information.

[0203] In one embodiment, the replacement and completion module includes:

[0204] A first determining unit, configured to determine pixels corresponding to a non-road surface element category in a semantic segmentation result of the current frame image as blocked pixels;

[0205] A replacement unit is used to replace the occluded pixel with a target pixel to obtain a multi-frame semantic segmentation result, wherein the target pixel is a pixel whose semantic category last appeared at the same position as the occluded pixel and corresponds to a road surface element category.

[0206] In one embodiment, the replacement and completion module further includes:

[0207] A superposition unit, used to superimpose the multi-frame semantic segmentation results onto the instance segmentation results of the current frame image and the historical multi-frame instance segmentation results, and determine an intersection area between a first instance in the instance segmentation result of the superimposed current frame image and a second instance in the superimposed historical multi-frame instance segmentation results;

[0208] A first completion unit, using the second instance to complete the first instance for the first time if the intersection area is greater than or equal to a preset intersection area threshold;

[0209] The second completion unit is used to find unallocated pixels that are not allocated to instances in the instance segmentation result after the first completion, and use the instance segmentation results of the neighborhood pixels corresponding to the unallocated pixels to perform a second completion on the unallocated pixels to obtain multi-frame instance segmentation results.

[0210] In one embodiment, the replacement and completion module further includes:

[0211] a similarity value calculation unit, configured to confirm a first lane line of the current frame image based on the multi-frame instance segmentation results, and calculate a lane line similarity value between the first lane line and a plurality of lane lines in a historical lane line set;

[0212] The judgment unit adds the first lane line to the corresponding first lane line set if the maximum value of the lane line similarity value is greater than or equal to the preset lane line similarity threshold; or creates a second lane line set and adds the first lane line to the second lane line set if the maximum value of the lane line similarity value is less than the preset lane line similarity threshold;

[0213] A third completion unit, if there is a set in which each lane line has a similarity value with the first lane line that is less than a preset lane line similarity threshold, takes the set as a third lane line set, and uses the lane lines in the third lane line set to complete the missing lane lines in the replaced current frame image;

[0214] The first acquisition unit is used to obtain a target-level multi-frame result based on the lane lines in the completed current frame image.

[0215] In one embodiment, the image processing module includes:

[0216] A second acquisition unit is used to acquire a preset image processing model, use the image processing model to perform semantic segmentation processing on the current frame image to obtain a first semantic segmentation result, and based on the first semantic segmentation result, use the image processing model to perform instance segmentation processing and classification processing on the current frame image to obtain a first instance segmentation result and a first classification result, respectively;

[0217] A selection unit, configured to select a first region of interest from the current frame image based on the first instance segmentation result;

[0218] A second determination unit is used to determine a minimum width value of the lane line in the first area of ​​interest, and to determine a difference between the width value and a preset width threshold;

[0219] an enlargement unit, if the minimum width value is less than the preset width threshold, enlarging the size of the first region of interest to the size of the current frame image, and using the preset image processing model to segment and classify the enlarged first region of interest to obtain a second semantic segmentation result, a second instance segmentation result, and a second classification result;

[0220] A reduction unit is used to reduce the size of the second semantic segmentation result, the second instance segmentation result and the second classification result to the size of the first region of interest, and use the reduced second semantic segmentation result, the second instance segmentation result and the second classification result to replace the semantic segmentation result corresponding to the first region of interest in the first semantic segmentation result to obtain the semantic segmentation result, replace the instance segmentation result corresponding to the first region of interest in the first instance segmentation result to obtain the instance segmentation result, and replace the classification result corresponding to the first region of interest in the first classification result to obtain the classification result.

[0221] In one embodiment, the image processing module further includes:

[0222] A third determining unit, used to determine a current image processing model;

[0223] a third acquisition unit, configured to acquire a first image sample, select an arbitrary area from the first image sample, and enlarge the area to a size of the first image sample as a second image sample;

[0224] an image processing unit, configured to perform semantic segmentation on the first image sample and the second image sample based on the current image processing model to obtain semantic segmentation results of the first image sample and the second image sample respectively, and perform instance segmentation on the first image sample and the second image sample based on the obtained semantic segmentation results of the first image sample and the second image sample to obtain instance segmentation results of the first image sample and the second image sample respectively;

[0225] a fourth determining unit, configured to determine a first instance feature vector of the first image and a second instance feature vector of the second image based on the obtained instance segmentation results of the first image sample and the second image sample;

[0226] a feature alignment unit, configured to perform feature alignment on the first instance feature vector and the second instance feature vector to obtain a feature alignment loss result, determine whether the feature alignment loss result is greater than a preset feature alignment loss threshold, and if the feature alignment loss result is greater than the preset feature alignment loss threshold, adjust the parameters of the current image processing model and record the number of parameter adjustments;

[0227] A training unit is used to return to the step of performing semantic segmentation on the first image sample and the second image sample based on the current image processing model after adjusting the parameters, and respectively obtain the semantic segmentation results of the first image sample and the second image sample, until the feature alignment loss result is less than or equal to the preset feature alignment loss threshold, or the number of parameter adjustments is equal to the preset number of adjustments, and the current image processing model corresponding to the minimum value in the feature pair loss result is selected as the preset image processing model.

[0228] The present application provides a road surface element recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the road surface element recognition method in the above-mentioned embodiment one.

[0229] Reference below Figure 7 , which shows a schematic diagram of the structure of a road surface element recognition device suitable for implementing the embodiment of the present application. The road surface element recognition device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The road surface element identification device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0230] like Figure 7As shown, the road surface element recognition device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the road surface element recognition device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the road surface element recognition device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a road surface element recognition device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0231] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0232] The pavement element recognition device provided by the present application adopts the pavement element recognition method in the above embodiment, which can solve the technical problem of pavement element recognition. Compared with the relevant technology, the beneficial effects of the pavement element recognition device provided by the present application are the same as the beneficial effects of the pavement element recognition method provided by the above embodiment, and the other technical features in the pavement element recognition device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0233] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0234] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0235] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the road surface element recognition method in the above-mentioned embodiment.

[0236] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0237] The computer-readable storage medium may be included in the road surface element recognition device; or may exist independently without being assembled into the road surface element recognition device.

[0238] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the road surface element recognition device, the road surface element recognition device:

[0239] Receive the current frame image collected by the visual sensor;

[0240] Performing semantic segmentation processing on the current frame image through a preset image processing model to obtain a semantic segmentation result of the current frame image;

[0241] Determine pixels corresponding to the non-road element category in the semantic segmentation result of the current frame image as occluded pixels, and replace the occluded pixels with target pixels, where the target pixels are pixels corresponding to the road element that appears the most times at the same position as the occluded pixels in the semantic segmentation results of the previous K consecutive frames;

[0242] Based on the replaced current frame image, a structured analysis is performed on the road surface elements in the scene to obtain structured road surface element information.

[0243] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0244] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0245] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0246] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned road surface element recognition method, and can solve the technical problem of road surface element recognition. Compared with the related art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the road surface element recognition method provided in the above-mentioned embodiment, and will not be described in detail here.

[0247] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned road surface element identification method when executed by a processor.

[0248] The computer program product provided by this application can solve the technical problem of road surface element recognition. Compared with the related art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the road surface element recognition method provided by the above embodiment, which will not be repeated here.

[0249] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A road surface element recognition method, characterized in that: The method comprises: Receive the current frame image collected by the visual sensor; Obtain a preset image processing model, use the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image, and obtain a semantic segmentation result of the current frame image; based on the semantic segmentation result of the current frame image, use the instance segmentation branch and the classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image, and obtain an instance segmentation result of the current frame image and a classification result of the current frame image, respectively; Use the historical multi-frame semantic segmentation results to replace the occluded pixels in the semantic segmentation results of the current frame image to obtain a multi-frame semantic segmentation result; based on the multi-frame semantic segmentation results, use the historical multi-frame instance segmentation results to complete the pixel points of the instance segmentation results missing in the instance segmentation results of the current frame image to obtain a multi-frame instance segmentation result; based on the multi-frame instance segmentation results, determine the current lane line, use the lane line in the historical lane line set to complete the current lane line, and obtain a target-level multi-frame result; Based on the multi-frame semantic segmentation results, the multi-frame instance segmentation results, the target-level multi-frame results and the classification results, a structured analysis is performed on the road surface elements in the scene to obtain structured road surface element information.

2. The road surface element recognition method according to claim 1, characterized in that: The step of replacing the blocked pixels in the semantic segmentation result of the current frame with the historical multi-frame semantic segmentation results to obtain the multi-frame semantic segmentation results also includes: Determine pixels corresponding to the non-road surface element category in the semantic segmentation result of the current frame image as blocked pixels; The occluded pixel is replaced with a target pixel to obtain a multi-frame semantic segmentation result, wherein the target pixel is a pixel whose semantic category last appeared at the same position as the occluded pixel and corresponds to a road surface element category.

3. The road surface element recognition method according to claim 1, characterized in that: The step of using historical multi-frame instance segmentation results to complete the pixel points of the missing instance segmentation results in the instance segmentation results of the current frame image based on the multi-frame semantic segmentation results to obtain the multi-frame instance segmentation results also includes: Superimposing the multi-frame semantic segmentation results onto the instance segmentation results of the current frame image and the historical multi-frame instance segmentation results, and determining an intersection area between the first instance in the instance segmentation result of the superimposed current frame image and the second instance in the superimposed historical multi-frame instance segmentation results; If the intersection area is greater than or equal to a preset intersection area threshold, using the second instance to complete the first instance for the first time; Unassigned pixels that are not assigned to instances in the instance segmentation result after the first completion are found, and the instance segmentation results of the neighborhood pixels corresponding to the unassigned pixels are used to perform a second completion on the unassigned pixels to obtain multi-frame instance segmentation results.

4. The road surface element recognition method according to claim 1, characterized in that: The step of determining the current lane line based on the multi-frame instance segmentation result, and using the lane lines in the historical lane line set to complete the current lane line to obtain the target-level multi-frame result also includes: Based on the multi-frame instance segmentation results, confirm a first lane line of the current frame image, and calculate a lane line similarity value between the first lane line and a plurality of lane lines in a historical lane line set; If the maximum value of the lane line similarity value is greater than or equal to the preset lane line similarity threshold, the first lane line is added to the corresponding first lane line set; or if the maximum value of the lane line similarity value is less than the preset lane line similarity threshold, a second lane line set is created and the first lane line is added to the second lane line set; If there is a set in which each lane line has a similarity value with the first lane line that is less than a preset lane line similarity threshold, the set is used as a third lane line set, and lane lines in the third lane line set are used to complete the missing lane lines in the replaced current frame image; Based on the completed lane lines in the current frame image, the target-level multi-frame results are obtained.

5. The road surface element recognition method according to claim 1, characterized in that: The steps of obtaining a preset image processing model, using the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image to obtain a semantic segmentation result, and based on the semantic segmentation result, using the instance segmentation branch and the classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image to obtain an instance segmentation result and a classification result, respectively, also include: Obtain a preset image processing model, use the image processing model to perform semantic segmentation processing on the current frame image to obtain a first semantic segmentation result, and based on the first semantic segmentation result, use the image processing model to perform instance segmentation processing and classification processing on the current frame image to obtain a first instance segmentation result and a first classification result, respectively; Based on the first instance segmentation result, selecting a first region of interest from the current frame image; Determine a minimum width value of the lane line in the first region of interest, and determine a difference between the width value and a preset width threshold; If the minimum width value is less than the preset width threshold, the size of the first region of interest is enlarged to the size of the current frame image, and the enlarged first region of interest is segmented and classified using the preset image processing model to obtain a second semantic segmentation result, a second instance segmentation result, and a second classification result; Reduce the size of the second semantic segmentation result, the second instance segmentation result and the second classification result to the size of the first region of interest, use the reduced second semantic segmentation result, the second instance segmentation result and the second classification result to replace the semantic segmentation result corresponding to the first region of interest in the first semantic segmentation result to obtain the semantic segmentation result, replace the instance segmentation result corresponding to the first region of interest in the first instance segmentation result to obtain the instance segmentation result, and replace the classification result corresponding to the first region of interest in the first classification result to obtain the classification result.

6. The road surface element recognition method according to claim 1, characterized in that: Before the steps of obtaining a preset image processing model, using the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image to obtain a semantic segmentation result, and based on the semantic segmentation result, using the instance segmentation branch and the classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image to obtain an instance segmentation result and a classification result respectively, the method further includes: Determine the current image processing model; Acquire a first image sample, select an arbitrary area from the first image sample, and enlarge it to the size of the first image sample as a second image sample; Based on the current image processing model, semantic segmentation is performed on the first image sample and the second image sample to obtain semantic segmentation results of the first image sample and the second image sample respectively; based on the obtained semantic segmentation results of the first image sample and the second image sample, instance segmentation is performed on the first image sample and the second image sample to obtain instance segmentation results of the first image sample and the second image sample respectively; Determine a first instance feature vector of the first image and a second instance feature vector of the second image based on the obtained instance segmentation results of the first image sample and the second image sample; Performing feature alignment on the first instance feature vector and the second instance feature vector to obtain a feature alignment loss result, determining whether the feature alignment loss result is greater than a preset feature alignment loss threshold, and if the feature alignment loss result is greater than the preset feature alignment loss threshold, adjusting parameters of the current image processing model and recording the number of parameter adjustments; Based on the current image processing model after adjusting the parameters, return to the step of performing semantic segmentation on the first image sample and the second image sample based on the current image processing model to obtain the semantic segmentation results of the first image sample and the second image sample respectively, until the feature alignment loss result is less than or equal to the preset feature alignment loss threshold, or the number of parameter adjustments is equal to the preset number of adjustments, and select the current image processing model corresponding to the minimum value of the feature pair loss result as the preset image processing model.

7. A road surface element recognition device, characterized in that: The device comprises: A receiving module, wherein the receiving module is used to receive a current frame image collected by a visual sensor; An image processing module, the image processing module is used to obtain a preset image processing model, use the semantic branch of the preset image processing model to perform semantic segmentation on the current frame image, obtain a semantic segmentation result of the current frame image, and based on the semantic segmentation result of the current frame image, use the instance segmentation branch and classification branch of the preset image processing model to perform instance segmentation and classification on the current frame image, and obtain an instance segmentation result of the current frame image and a classification result of the current frame image, respectively; A replacement and completion module, the replacement and completion module is used to use the historical multi-frame semantic segmentation results to replace the occluded pixels in the semantic segmentation results of the current frame image to obtain a multi-frame semantic segmentation result; based on the multi-frame semantic segmentation results, use the historical multi-frame instance segmentation results to complete the pixel points of the instance segmentation results missing in the instance segmentation results of the current frame image to obtain a multi-frame instance segmentation result; based on the multi-frame instance segmentation results, determine the current lane line, use the lane line in the historical lane line set to complete the current lane line, and obtain a target-level multi-frame result; A structured analysis module is used to perform structured analysis on road surface elements in the scene based on the multi-frame semantic segmentation results, the multi-frame instance segmentation results, the target-level multi-frame results and the classification results to obtain road surface element information in a structured form.

8. A road surface element recognition device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the road surface element recognition method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the road surface element recognition method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the road surface element recognition method according to any one of claims 1 to 6 are implemented.

Images