Road surface type identification method and device, storage medium and vehicle

Abstract

The invention discloses a road surface type identification method and device, a storage medium and a vehicle, relates to the technical field of vehicle environment perception, and aims to solve the problem of inaccurate road surface type identification results in related technologies. The road surface type identification method comprises the following steps: acquiring a road image, dividing the road image to obtain a left side area sub-image and a right side area sub-image, respectively performing road surface type identification on the left side area sub-image and the right side area sub-image, and determining the road surface type of the left side area sub-image and the road surface type of the right side area sub-image.

Description

Technical Field

[0001] The present application relates to the field of vehicle environment perception technology, and in particular to a road type recognition method, device, storage medium and vehicle. Background Art

[0002] With the continuous improvement of people's living standards and the continuous advancement of science and technology, people have higher requirements for the safety and comfort of vehicles during driving. The quality of road conditions will directly affect people. Therefore, identifying the road type during vehicle driving is crucial to improving the safety and comfort of vehicles during driving.

[0003] In the related technology, the road type information is generally obtained based on the sensor, and then the various types of information are integrated through the neural network to finally predict the road type, or the road information is directly input into the neural network model for classification, and then the road type result is predicted. However, the final road type recognition results obtained by these methods have low accuracy. Summary of the invention

[0004] The purpose of the present application is to provide a method, device, storage medium and vehicle for identifying road types, aiming to solve the problem of low accuracy of road type identification results in related technologies.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a method for identifying a road surface type, comprising: obtaining a road image, dividing the road image into a left area sub-image and a right area sub-image; respectively identifying a road surface type for the left area sub-image and the right area sub-image, and determining a road surface type of the left area sub-image and a road surface type of the right area sub-image.

[0007] The road surface type recognition method provided in the embodiment of the present application improves the accuracy of the road surface type recognition result by dividing the road surface image into two parts, the left and right parts, and respectively identifying the road surface type of the left area sub-image and the road surface type of the right area sub-image. In addition, this is also beneficial in the subsequent vehicle control process, when the road surface types corresponding to the two side areas are different, the left and right wheels can be controlled differently, thereby improving the stability of vehicle driving and the comfort of user experience.

[0008] In some embodiments, the above method further includes: controlling the movement of the left wheel of the vehicle based on the road surface type of the left area sub-image; and controlling the movement of the right wheel of the vehicle based on the road surface type of the right area sub-image.

[0009] In some embodiments, the above-mentioned road surface type recognition is performed on the left area sub-image and the right area sub-image respectively to determine the road surface type of the left area and the road surface type of the right area, including: for any area sub-image of the left area sub-image and the right area sub-image, the area sub-image is recognized to determine multiple candidate road surface types of the area sub-image, and the image area corresponding to each of the multiple candidate road surface types in the area sub-image; based on the image area corresponding to each of the multiple candidate road surface types in the area sub-image, the road surface type of the area sub-image is determined from multiple candidate road surface types.

[0010] In some embodiments, the above-mentioned determining the pavement type of the regional sub-image from multiple candidate pavement types based on the image area corresponding to each pavement type in the regional sub-image includes: determining the image area corresponding to each pavement type in the regional sub-image among the multiple candidate pavement types based on the image area corresponding to each pavement type in the regional sub-image; and determining the candidate pavement type with the largest image area among the multiple candidate pavement types as the pavement type of the regional sub-image.

[0011] In some embodiments, the above-mentioned road surface type recognition is performed on the left regional sub-image and the right regional sub-image respectively to determine the road surface type of the left regional sub-image and the road surface type of the right regional sub-image, including: for any one of the left regional sub-image and the right regional sub-image, determining the similarity between the regional sub-image and the historical regional sub-image, the historical regional sub-image is obtained by segmenting the historical road image before the shooting time of the road image; when the similarity does not meet the preset condition, the road surface type is recognized on the regional sub-image to obtain the road surface type of the regional sub-image.

[0012] In some embodiments, the above method further includes: when the similarity satisfies a preset condition, using the road surface type of the historical area sub-image as the road surface type of the area sub-image.

[0013] In some embodiments, the above-mentioned preset condition includes that the similarity is greater than or equal to a threshold.

[0014] In some embodiments, the above-mentioned road surface type recognition is performed on the left regional sub-image and the right regional sub-image respectively to determine the road surface type of the left regional sub-image and the road surface type of the right regional sub-image, including: for any one of the left regional sub-image and the right regional sub-image, the regional sub-image is input into a road surface type recognition model to obtain the road surface type of the regional sub-image.

[0015] In some embodiments, the above-mentioned road surface type recognition model is a semantic segmentation network model; the semantic segmentation network model includes: a backbone network module and a post-processing module; the backbone network module is used to extract the features of the road image; the post-processing module is used to determine the road surface type of the road image based on the features of the road image.

[0016] In some embodiments, the pavement type recognition model is trained in the following manner: obtaining a training sample set, the training sample set including multiple sample area sub-images and a pavement type label corresponding to each sample area sub-image in the multiple sample area sub-images; training an initial pavement type recognition model based on the training sample set to obtain a pavement type recognition model.

[0017] In some embodiments, the road surface type includes at least one of the following: regular road, icy road, sandy road, dry dirt road, muddy road, wading road, grass, gravel road, and rocky road.

[0018] In a second aspect, the present application provides a device for identifying road surface types, including: an acquisition unit, used to acquire a road image, and divide the road image into a left area sub-image and a right area sub-image; a first processing unit, used to identify the road surface types of the left area sub-image and the right area sub-image respectively, and determine the road surface type of the left area sub-image and the road surface type of the right area sub-image.

[0019] In some embodiments, the above-mentioned device also includes: a second processing unit, which is used to control the movement of the left wheel of the vehicle based on the road surface type of the left area sub-image; and control the movement of the right wheel of the vehicle based on the road surface type of the right area sub-image.

[0020] In some embodiments, the above-mentioned first processing unit is specifically used to identify the road surface type of any one of the left and right regional sub-images, determine multiple candidate road surface types of the regional sub-image, and the image area corresponding to each of the multiple candidate road surface types in the regional sub-image; based on the image area corresponding to each of the multiple candidate road surface types in the regional sub-image, determine the road surface type of the regional sub-image from the multiple candidate road surface types.

[0021] In some embodiments, the above-mentioned first processing unit is specifically used to determine the image area corresponding to each of the multiple candidate road surface types in the regional sub-image based on the image area corresponding to each of the multiple candidate road surface types in the regional sub-image; and determine the candidate road surface type with the largest image area among the multiple candidate road surface types as the road surface type of the regional sub-image.

[0022] In some embodiments, the above-mentioned first processing unit is specifically used to determine the similarity between the regional sub-image and the historical regional sub-image for any one of the left regional sub-image and the right regional sub-image, where the historical regional sub-image is obtained by segmenting the historical road image before the shooting time of the road image; when the similarity does not meet the preset conditions, the road surface type is identified on the regional sub-image to obtain the road surface type of the regional sub-image.

[0023] In some embodiments, the first processing unit is further configured to use the road surface type of the historical regional sub-image as the road surface type of the regional sub-image when the similarity satisfies a preset condition.

[0024] In some embodiments, the above-mentioned preset condition includes that the similarity is greater than or equal to a threshold.

[0025] In some embodiments, the first processing unit is specifically used to input the regional sub-image of any one of the left regional sub-image and the right regional sub-image into a road surface type recognition model to obtain the road surface type of the regional sub-image.

[0026] In some embodiments, the above-mentioned road surface type recognition model is a semantic segmentation network model; the semantic segmentation network model includes: a backbone network module and a post-processing module; the backbone network module is used to extract the features of the road image; the post-processing module is used to determine the road surface type of the road image based on the features of the road image.

[0027] In some embodiments, the above-mentioned first processing unit is specifically used to obtain a training sample set, the training sample set includes multiple sample area sub-images and a road surface type label corresponding to each sample area sub-image in the multiple sample area sub-images; the initial road surface type recognition model is trained based on the training sample set to obtain a road surface type recognition model.

[0028] In some embodiments, the road surface type includes at least one of the following: regular road, icy road, sandy road, dry dirt road, muddy road, wading road, grass, gravel road, and rocky road.

[0029] In a third aspect, the present application provides an electronic device comprising: a processor and a memory configured to store processor executable instructions; wherein the processor is configured to execute the instructions to implement any one of the optional road surface type identification methods in the first aspect above.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium having instructions stored thereon. When the instructions in the computer-readable storage medium are executed by a device, the device is enabled to execute any one of the optional road surface type identification methods described in the first aspect above.

[0031] In a fifth aspect, the present application provides a vehicle, comprising: the electronic device of the third aspect above, or the computer-readable storage medium of the fourth aspect.

[0032] In a sixth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed on a processor of a device, the device is capable of executing any optional method for identifying a road surface type as described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 A schematic diagram of a process flow of a road surface type identification method provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram of a road image segmentation method provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of a road type recognition model training process provided in an embodiment of the present application;

[0037] Figure 4 A schematic flow chart of another method for identifying road surface types provided in an embodiment of the present application;

[0038] Figure 5 A schematic flow chart of another method for identifying road surface types provided in an embodiment of the present application;

[0039] Figure 6 A schematic flow chart of another method for identifying road surface types provided in an embodiment of the present application;

[0040] Figure 7 A schematic diagram of the structure of a road surface type identification device provided in an embodiment of the present application;

[0041] Figure 8 A schematic diagram of the structure of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] In the description of this application, it should be understood that the terms "upper", "lower", "left", "right", "front", "back", "inside", "outside", etc. indicate directions or positional relationships based on the directions or relative positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on this application. Unless otherwise specified, the above-mentioned directional description can be flexibly set in the process of actual application under the condition that the relative positional relationship shown in the accompanying drawings is met.

[0044] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0045] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection. It can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0046] In the present application, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, article or device including the element.

[0047] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0048] In the description of this specification, specific features, structures, materials or characteristics may be combined in an appropriate manner in any one or more embodiments or examples.

[0049] With the continuous improvement of people's living standards and the continuous advancement of science and technology, people have higher requirements for the safety and comfort of vehicles during driving. The quality of road conditions will directly affect people. Therefore, identifying the road type during vehicle driving is crucial to improving the safety and comfort of vehicles during driving.

[0050] In related technologies, the characteristics and data structure of the road perception information collected by each sensor can be used to extract features using different modeling methods for perception information of different modes, and then the feature vectors extracted from the perception information of each mode are fused in feature series. Finally, the LSTM deep learning network is used to transform the multimodal fusion features into a time series classification problem, and the road type recognition is completed through supervised learning. Because this method needs to collect multi-dimensional information, it will require multiple sensors and is relatively complex.

[0051] You can also use classification neural networks to classify images into road types. However, if you directly classify images, the images collected by vehicles have many interferences, which can easily interfere with the classification, and the accuracy of the road recognition results is relatively low.

[0052] Based on this, the present application provides a road surface type recognition method, device, storage medium and vehicle, including: obtaining a road image, dividing the road image into a left area sub-image and a right area sub-image, respectively recognizing the road surface type of the left area sub-image and the right area sub-image, and determining the road surface type of the left area sub-image and the road surface type of the right area sub-image. Thus, by dividing the road surface image into left and right parts, respectively recognizing the left and right parts to obtain the road surface type of the left area sub-image and the road surface type of the right area sub-image, not only the accuracy of the road surface type recognition result is improved, but also after determining the road surface types corresponding to the left and right areas, when the road surface types corresponding to the two side areas are different, the left and right wheels can be controlled differently, thereby improving the driving stability of the vehicle and the comfort of the user experience.

[0053] The road surface type identification method of the present application is introduced below with reference to the accompanying drawings.

[0054] It should be noted that the road surface type recognition method of the present application is applicable to a road surface type recognition device, and the road surface type recognition device is applied to an electronic device, and the electronic device can be configured in a vehicle so that the vehicle can realize the road surface type recognition function. Among them, the road surface type recognition device can be an on-board intelligent computing platform in the vehicle, or the road surface type recognition device can also be a functional module in the on-board intelligent computing platform for performing road surface type recognition, etc. The embodiments of the present application do not specifically limit this.

[0055] like Figure 1 As shown, the road surface type identification method of the present application includes the following steps:

[0056] S101, acquiring a road image, and dividing the road image into a left area sub-image and a right area sub-image.

[0057] The road image can be obtained by a perception camera in front of the vehicle, and the present application does not limit the position of the perception camera. It should be understood that since the vehicle is traveling at a high speed, the perception camera can be set above the front windshield to obtain a wider field of view.

[0058] In some embodiments, there may be multiple perception cameras located at different positions in front of the vehicle. The road image taken by each camera at the current moment can be acquired, and the acquired road graphics can be fused. The fused image can be used as the road image, so that the road image obtained is clearer and more accurate.

[0059] For example, the perception cameras can be set above the front windshield and at the front grille. The camera above the front windshield can obtain road images with a wider field of view, but its position is too high and may not be able to accurately capture detailed information on the ground. The camera at the front grille is closer to the ground, which is conducive to capturing detailed information on the ground. The road image obtained by combining the two is more accurate.

[0060] The left area sub-image is the area where the left wheel of the vehicle will travel, and the right area sub-image is the area where the right wheel of the vehicle will travel.

[0061] As a feasible implementation method, the acquired road is divided into two parts, and the image is segmented according to the middle position of the image width to obtain two images, namely the left area sub-image and the right area sub-image. Figure 2 As shown, the road image is cut according to the dividing line at the middle position of the image width to obtain a left area sub-image and a right area sub-image.

[0062] As another feasible implementation manner, when the position of the perception camera is not located in the middle area of ​​the width of the vehicle, the road image is divided into a left area sub-image and a right area sub-image based on the position of the perception camera on the vehicle.

[0063] S102 , respectively identify the road surface type of the left area sub-image and the right area sub-image to determine the road surface type of the left area sub-image and the road surface type of the right area sub-image.

[0064] Among them, road types include: conventional roads (including cement roads, asphalt roads), ice and snow roads, sandy roads, dry dirt roads, muddy roads, wading roads, grass roads, gravel roads, and rocky roads.

[0065] As a feasible implementation method, for any one of the left regional sub-image and the right regional sub-image, the regional sub-image can be input into a road surface type recognition model to obtain the road surface type of the regional sub-image.

[0066] Furthermore, in some embodiments, after obtaining the road surface type of the left area sub-image and the road surface type of the right area sub-image, the movement of the left wheel of the vehicle can be controlled based on the road surface type of the left area sub-image, and the movement of the right wheel of the vehicle can be controlled based on the road surface type of the right area sub-image.

[0067] Exemplarily, a damper is set at the wheel position. When driving on a regular road, the damping of the damper in the vehicle is adjusted, and the damping current is appropriately increased to improve the maneuverability, that is, to improve the stability of the vehicle. When driving on an icy road, the damping current is appropriately increased to improve the maneuverability while lowering the height of the vehicle. When driving on a sandy road or a muddy road, the damping current is appropriately increased to improve the maneuverability while increasing the height of the vehicle. When driving on a wading road or a grassy road, the height of the vehicle is increased. When driving on a gravel road or a rocky road, the damping of the damper in the vehicle is adjusted, and the damping current is appropriately reduced while increasing the height of the vehicle. When driving on a dry dirt road, the damping of the damper in the vehicle is adjusted to appropriately reduce the damping current.

[0068] Based on S101-S102, it can be seen that the road type recognition method provided in the embodiment of the present application, by dividing the road image into two parts, the left and right parts are respectively recognized to obtain the road type of the left area sub-image and the road type of the right area sub-image, which not only improves the accuracy of the road type recognition result. In addition, this is also conducive to the subsequent vehicle control process, when the road types corresponding to the two side areas are different, the left and right wheels can be controlled differently, thereby improving the stability of vehicle driving and the comfort of user experience.

[0069] It should be noted that the above-mentioned road surface type recognition model can be any model or algorithm that can recognize the road surface type, and is not limited here.

[0070] The road surface type recognition model in this application uses the semantic segmentation network model as an example to illustrate the process of identifying the road surface type. The above semantic segmentation network model includes: a backbone network module and a post-processing module. The backbone network module is used to extract the features of the road image, and the post-processing module is used to determine the road surface type of the road image according to the features of the road image. The semantic segmentation network model can accurately classify each pixel in the image and can outline the precise contours of the object, so the result obtained by using the semantic segmentation network model when identifying the road surface type is more accurate.

[0071] In some embodiments, the pavement type recognition model is trained in the following manner: obtaining a training sample set, the training sample set including multiple sample area sub-images and a pavement type label corresponding to each sample area sub-image in the multiple sample area sub-images, training an initial pavement type recognition model based on the training sample set to obtain a pavement type recognition model.

[0072] In some embodiments, the backbone network module includes: MobileNetV2. It should be understood that in this application, the backbone (backbone network module) of the semantic segmentation network model is changed from Xception39 to MobileNetV2, because MobileNetV2 is a lightweight network with fewer parameters, requires less hardware resources, and has a faster reasoning speed.

[0073] In other embodiments, referring to Figure 3 ,The semantic segmentation network model needs to be trained before use. The training process can be as follows:

[0074] S301, obtaining road image data of various road surface types, annotating the road images to obtain a data set, and dividing the data set into a training set, a validation set, and a test set.

[0075] Optionally, a semantic segmentation tool is used to annotate the road image. After the annotation is completed, a data set and a label set are obtained. The data set and the corresponding label set are randomly allocated in a ratio of 6:2:2 to obtain a training set, a validation set, and a test set, respectively.

[0076] S302. Change the backbone network module of the semantic segmentation network model to MobileNetV2.

[0077] Among them, the backbone network module of the semantic segmentation network model can be changed from Xception39 to MobileNetV2.

[0078] S303, when the semantic segmentation network model starts training, the MobliNetV2 network model is loaded on the image recognition database (ImageNet) and the network parameters are initialized.

[0079] S304: Input the training set and the validation set into the model for training to obtain a trained semantic segmentation network model.

[0080] S305: Save the trained semantic segmentation network model.

[0081] S306: For the trained semantic segmentation network model, use the verification set to verify the trained semantic segmentation network model to obtain a verification result.

[0082] S307: Use the test set to test the trained semantic segmentation network model to obtain test results.

[0083] In some embodiments, the road type is not fixed, especially in some remote roads, mountainous areas, etc., the road information is relatively complex, so there can be multiple road types, but because different road types correspond to different wheel control strategies, it is necessary to select one of the multiple road types as the target road type, so as to facilitate the maintenance of vehicle driving stability. Figure 1 and Figure 4 , the above S102 may specifically include the following steps:

[0084] S401. For any one of the left and right regional sub-images, perform road surface type recognition on the regional sub-image to determine multiple candidate road surface types of the regional sub-image and an image area corresponding to each of the multiple candidate road surface types in the regional sub-image.

[0085] In some embodiments, the left area sub-image and the right area sub-image are identified respectively to obtain multiple candidate road surface types for the left area sub-image and the image area corresponding to each road surface type in the area sub-image, and multiple candidate road surface types for the right area sub-image and the image area corresponding to each road surface type in the area sub-image.

[0086] Optionally, before identifying the road surface type corresponding to each sub-image area, each sub-image needs to be preprocessed. The processed sub-image is more adaptable to the learning and reasoning process of the neural network model, so that the road surface type recognition result obtained will be more accurate.

[0087] Exemplarily, the sub-image is scaled, and the sub-image may be proportionally scaled to (600, 600), so that the input image has a fixed size.

[0088] It should be noted that the above only describes the preprocessing process of each sub-image by taking the example of scaling the image so that the image input into the road surface type recognition model has a fixed size. In addition to this, other preprocessing operations can also be performed on each sub-image. The purpose of these preprocessing operations is to improve the quality of the image data, making the image easier to be learned by the road surface type recognition model, and thus being able to more accurately identify the road surface type in each sub-image.

[0089] S402: Determine a road surface type of the regional sub-image from among the multiple candidate road surface types based on an image area corresponding to each road surface type in the regional sub-image.

[0090] In some embodiments, for the left area sub-image, the road surface type of the left area sub-image is determined from multiple candidate road surface types corresponding to the left area sub-image, and for the right area sub-image, the road surface type of the right area sub-image is determined from multiple candidate road surface types corresponding to the right area sub-image.

[0091] As a feasible implementation method, based on the image area corresponding to each road surface type in the regional sub-image among multiple candidate road surface types, the image area corresponding to each road surface type in the regional sub-image is determined, and the candidate road surface type with the largest image area among the multiple candidate road surface types is determined as the road surface type of the regional sub-image.

[0092] The image area of ​​each road surface type can be determined according to the number of pixels of the road surface in the image, and the image area is the sum of the number of pixels.

[0093] It should be understood that the vehicle is traveling at a very high speed, and the area corresponding to the acquired road image is very small relative to the vehicle speed, so the vehicle can quickly pass through the area. Therefore, the road surface type with the largest area can determine the state in which the vehicle travels through the area. Therefore, the candidate road surface type with the largest area is determined as the road surface type of the regional sub-image.

[0094] As another feasible implementation method, different road types have different effects on vehicle driving, and the road type that has the greatest impact on vehicle driving can be determined as the road type of the sub-image in the region. For example, after a heavy rain, some parts of the road will have water accumulation due to the low terrain, so the road types corresponding to the regional sub-image include conventional roads and wading roads. Although the conventional road has a larger area, the wading road will affect vehicle driving. Therefore, in order to ensure the safety and stability of vehicle driving, the wading road can be used as the road type of the region.

[0095] Based on S401-S402, it can be seen that when there are multiple road types in a road area, the candidate road type with the largest area is taken as the road type of the area, so that the road type recognition result can be obtained even when the road surface is a mixed type road surface.

[0096] In some embodiments, the road type of the vehicle may remain unchanged during driving. For example, if there is no bad weather in the city, the vehicle will always drive on the regular road. Therefore, inputting the road image into the road type recognition model to determine the road type result after each acquisition will take up a lot of computing resources, and the speed of determining the road type result is also slow. Figure 1 and Figure 5 , the above S102 may specifically include the following steps:

[0097] S501 : For any one of the left regional sub-image and the right regional sub-image, determine the similarity between the regional sub-image and the historical regional sub-image.

[0098] In some embodiments, for the left area sub-image, the similarity between the left area sub-image and the historical left area sub-image is determined, and for the right area sub-image, the similarity between the right area sub-image and the historical right area sub-image is determined.

[0099] The historical area sub-image is obtained by segmenting the historical road image before the road image is shot.

[0100] As a feasible implementation method, the similarity between the regional sub-image and the historical regional sub-image can be determined by calculating the similarity distance between the regional sub-image and the historical regional sub-image. The larger the similarity distance, the smaller the similarity between the regional sub-image and the historical regional sub-image.

[0101] Exemplarily, the similarity distance is the Hamming distance, and the Hamming distance between the regional sub-image and the historical regional sub-image is calculated by the perceptual hash algorithm. The specific process is to obtain the hash value of the regional sub-image and the hash value of the historical regional sub-image by the perceptual hash algorithm. Based on the hash values ​​at two moments, the Hamming distance between them is calculated. The smaller the Hamming distance, the higher the similarity of the two images.

[0102] In some embodiments, before determining the hash value of the regional sub-image and the hash value of the historical regional sub-image through the perceptual hash algorithm, the image needs to be cropped to reduce the influence of interference factors in the image and thereby improve the accuracy of similarity judgment.

[0103] Exemplarily, for the left region sub-image, the first column to the 0.2th column multiplied by the width is set to 0, and the first row to the 0.3th column multiplied by the height is set to 0, to obtain two images, namely, the cropped left region sub-image and the cropped historical left region sub-image. For the right region sub-image, the 0.8th column multiplied by the width to the last column is set to 0, and the first row to the 0.3th column multiplied by the height is set to 0, to obtain two images, namely, the cropped right region sub-image and the cropped historical right region sub-image.

[0104] It should be noted that the present application determines the similarity between the regional sub-image and the historical regional sub-image through the perceptual hash algorithm, but this does not constitute a limitation to the present application. Other methods that can calculate the similarity between the regional sub-image and the historical regional sub-image, such as feature matching, deep learning and other methods, can also be applied to the road type recognition method of the present application.

[0105] S502: When the similarity does not satisfy a preset condition, a road surface type is identified for the regional sub-image to obtain the road surface type of the regional sub-image.

[0106] In some embodiments, when the similarity between the left area sub-image and the historical left area sub-image does not meet the preset condition, the road surface type is identified on the left area sub-image to obtain the road surface type of the left area sub-image; when the similarity between the right area sub-image and the historical right area sub-image does not meet the preset condition, the road surface type is identified on the right area sub-image to obtain the road surface type of the right area sub-image.

[0107] In some embodiments, when the similarity meets the preset conditions, the road surface type of the historical regional sub-image is used as the road surface type of the regional sub-image. The above preset conditions include that the similarity is greater than or equal to a threshold. Specifically, it may include: when the similarity between the left regional sub-image and the historical left regional sub-image meets the preset conditions, the road surface type of the historical left regional sub-image is used as the road surface type of the regional sub-image; when the similarity between the right regional sub-image and the historical right regional sub-image meets the preset conditions, the road surface type of the historical right regional sub-image is used as the road surface type of the regional sub-image.

[0108] Exemplarily, the present application calculates the Hamming distance between the regional sub-image and the historical regional sub-image and sets the similarity distance threshold to 10. When the Hamming distance is less than 10, the two images are considered to be similar, that is, the road surface types of the two are considered to be the same. The road surface type of the historical regional sub-image can be used as the road surface type of the regional sub-image. If the Hamming distance is greater than or equal to 10, the two images are considered to be dissimilar and the road surface type of the regional sub-image needs to be re-determined.

[0109] In other embodiments, if there is no road image at the last moment or the corresponding road surface type is not determined at the last moment, there is no need to compare the similarity between the regional sub-image and the historical regional sub-image, and the road surface type of the regional sub-image is directly determined.

[0110] Based on S501-S502, it can be seen that by judging whether the regional sub-image is similar to the historical regional sub-image, if they are similar, the road surface type of the regional sub-image at the previous moment is used as the road surface type of the regional sub-image, thereby reducing the number of road surface type recognition times and thus improving the speed of determining the recognition result.

[0111] Combination Figure 6 The road surface type identification method of the present application is introduced with a specific embodiment, which includes the following steps:

[0112] S601. Train a road surface type recognition model to obtain a trained road surface type recognition model.

[0113] S602: Load the trained road type recognition model.

[0114] S603: Pre-process the road image output by the vehicle-mounted camera.

[0115] S604: Determine whether it is the first frame of road image, if yes, execute step S606, if not, execute step S605.

[0116] It should be noted that the first frame of road image here is the road image at the first moment.

[0117] S605: Determine whether the current frame image is similar to the previous frame image, if so, execute step S607, if not, execute step S606.

[0118] It should be noted that the determination of whether the current frame image is similar to the previous frame image is the determination of whether the image at the current moment is similar to the image at the previous moment. The specific process can be referred to S501 and will not be repeated here.

[0119] S606: Input the road image into the trained road surface type recognition model to obtain a road surface type recognition result.

[0120] S607: Use the road surface type recognition result of the previous frame as the road surface type recognition result of the current frame.

[0121] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, the road surface type recognition device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0122] The embodiment of the present application can exemplarily divide the functional modules of the road surface type identification device according to the above method. For example, the road surface type identification device can include various functional modules corresponding to the various functional divisions, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0123] Figure 7 A possible structural diagram of the road surface type identification device involved in the above embodiment is shown. Figure 7 The road surface type recognition device 700 includes: an acquisition unit 701, which is used to acquire a road image and divide the road image into a left area sub-image and a right area sub-image. A first processing unit 702, which is used to recognize the road surface type of the left area sub-image and the right area sub-image respectively, and determine the road surface type of the left area sub-image and the road surface type of the right area sub-image.

[0124] In some embodiments, the above-mentioned device also includes: a second processing unit, which is used to control the movement of the left wheel of the vehicle based on the road surface type of the left area sub-image; and control the movement of the right wheel of the vehicle based on the road surface type of the right area sub-image.

[0125] In some embodiments, the first processing unit 702 is specifically used to identify the road surface type of any one of the left and right regional sub-images, determine multiple candidate road surface types of the regional sub-image, and the image area corresponding to each of the multiple candidate road surface types in the regional sub-image; determine the road surface type of the regional sub-image from the multiple candidate road surface types based on the image area corresponding to each of the multiple candidate road surface types in the regional sub-image.

[0126] In some embodiments, the above-mentioned first processing unit 702 is specifically used to determine the image area corresponding to each road surface type among multiple candidate road surface types in the regional sub-image based on the image area corresponding to each road surface type among multiple candidate road surface types in the regional sub-image; and determine the candidate road surface type with the largest image area among the multiple candidate road surface types as the road surface type of the regional sub-image.

[0127] In some embodiments, the first processing unit 702 is specifically used to determine the similarity between the regional sub-image and the historical regional sub-image for any one of the left regional sub-image and the right regional sub-image, where the historical regional sub-image is obtained by segmenting the historical road image before the shooting time of the road image; when the similarity does not meet the preset conditions, the road surface type of the regional sub-image is identified to obtain the road surface type of the regional sub-image.

[0128] In some embodiments, the first processing unit 702 is further configured to use the road surface type of the historical regional sub-image as the road surface type of the regional sub-image when the similarity meets a preset condition.

[0129] In some embodiments, the above-mentioned preset condition includes that the similarity is greater than or equal to a threshold.

[0130] In some embodiments, the first processing unit 702 is specifically used to input any one of the left and right regional sub-images into a road surface type recognition model to obtain the road surface type of the regional sub-image.

[0131] In some embodiments, the above-mentioned road surface type recognition model is a semantic segmentation network model; the semantic segmentation network model includes: a backbone network module and a post-processing module; the backbone network module is used to extract the features of the road image; the post-processing module is used to determine the road surface type of the road image based on the features of the road image.

[0132] In some embodiments, the first processing unit 702 is specifically used to obtain a training sample set, which includes multiple sample area sub-images and a road surface type label corresponding to each sample area sub-image in the multiple sample area sub-images; and train an initial road surface type recognition model based on the training sample set to obtain a road surface type recognition model.

[0133] In some embodiments, the road surface type includes at least one of the following: regular road, icy road, sandy road, dry dirt road, muddy road, wading road, grass, gravel road, and rocky road.

[0134] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0135] Figure 8 This is a schematic diagram of the structure of a device provided in an embodiment of the present application. Figure 8 As shown, the device 800 includes but is not limited to: a processor 801 and a memory 802.

[0136] The memory 802 is used to store executable instructions of the processor 801. It can be understood that the processor 801 is configured to execute instructions to implement the road surface type recognition method in the above embodiment.

[0137] It should be noted that those skilled in the art can understand that Figure 8 The device structure shown in the figure does not constitute a limitation of the device, and the device may include Figure 8 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0138] The processor 801 is the control center of the device. It uses various interfaces and lines to connect various parts of the entire device. By running or executing software programs and / or modules stored in the memory 802 and calling data stored in the memory 802, it performs various functions of the device and processes data, thereby monitoring the device as a whole. The processor 801 may include one or more processing units. Optionally, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 801.

[0139] The memory 802 may be used to store software programs and various data. The memory 802 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one functional module (such as a determination unit, a processing unit, etc.), etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0140] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 802 including instructions, and the above instructions can be executed by the processor 801 of the device 800 to implement the method in the above embodiment.

[0141] In actual implementation, Figure 7 The acquisition unit 701 and the first processing unit 702 in the embodiment can both be composed of Figure 8The processor 801 in the embodiment calls the computer program stored in the memory 802. The specific execution process can refer to the description of the method part in the above embodiment, which will not be repeated here.

[0142] Optionally, the computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0143] In an exemplary embodiment, the present application also provides a vehicle, comprising the above-mentioned device and a computer-readable storage medium.

[0144] In an exemplary embodiment, the present application also provides a computer program product including one or more instructions, and the one or more instructions can be executed by the processor 801 of the device to complete the method in the above embodiment.

[0145] It should be noted that when the instructions in the above-mentioned computer-readable storage medium or one or more instructions in the computer program product are executed by the processor of the device, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect as the above-mentioned method can be achieved. To avoid repetition, they will not be repeated here.

[0146] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the full classification part or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute the full classification part or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, disk or CD and other media that can store program code.

[0151] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for identifying road surface types, characterized in that: include: Acquire a road image, and divide the road image into a left area sub-image and a right area sub-image; The road surface type is identified for the left area sub-image and the right area sub-image respectively, and the road surface type of the left area sub-image and the road surface type of the right area sub-image are determined.

2. The method according to claim 1, characterized in that The method further comprises: Controlling the movement of the left wheel of the vehicle based on the road surface type of the left area sub-image; The movement of the right wheel of the vehicle is controlled based on the road surface type of the right area sub-image.

3. The method according to claim 1, characterized in that The step of respectively identifying the road surface type of the left area sub-image and the right area sub-image to determine the road surface type of the left area and the road surface type of the right area includes: For any one of the left regional sub-image and the right regional sub-image, road surface type recognition is performed on the regional sub-image to determine a plurality of candidate road surface types of the regional sub-image and an image area corresponding to each road surface type of the plurality of candidate road surface types in the regional sub-image; Based on the image area corresponding to each road surface type in the regional sub-image, the road surface type of the regional sub-image is determined from the multiple candidate road surface types.

4. The method according to claim 3, characterized in that The determining the road surface type of the regional sub-image from the multiple candidate road surface types based on the image area corresponding to each road surface type in the regional sub-image comprises: Determine, based on the image area corresponding to each road surface type in the plurality of candidate road surface types in the regional sub-image, the image area corresponding to each road surface type in the plurality of candidate road surface types in the regional sub-image; The candidate road surface type with the largest image area among the multiple candidate road surface types is determined as the road surface type of the regional sub-image.

5. The method according to any one of claims 1 to 4, characterized in that: The step of respectively identifying the road surface type of the left area sub-image and the right area sub-image to determine the road surface type of the left area sub-image and the road surface type of the right area sub-image comprises: For any one of the left regional sub-image and the right regional sub-image, determining a similarity between the regional sub-image and a historical regional sub-image, wherein the historical regional sub-image is obtained by segmenting a historical road image before the shooting time of the road image; When the similarity does not satisfy a preset condition, the road surface type is identified on the regional sub-image to obtain the road surface type of the regional sub-image.

6. The method according to claim 5, characterized in that The method further comprises: When the similarity satisfies a preset condition, the road surface type of the historical area sub-image is used as the road surface type of the area sub-image.

7. The method according to claim 5, characterized in that The preset condition includes that the similarity is greater than or equal to a threshold.

8. The method according to any one of claims 1 to 6, characterized in that: The step of respectively identifying the road surface type of the left area sub-image and the right area sub-image to determine the road surface type of the left area sub-image and the road surface type of the right area sub-image comprises: For any one of the left region sub-image and the right region sub-image, the region sub-image is input into a road surface type recognition model to obtain the road surface type of the region sub-image.

9. The method according to claim 8, characterized in that The road type recognition model is a semantic segmentation network model; the semantic segmentation network model includes: a backbone network module and a post-processing module; The backbone network module is used to extract features of the road image; The post-processing module is used to determine the road surface type of the road image according to the characteristics of the road image.

10. The method according to claim 8, characterized in that The road type recognition model is trained in the following way: Acquire a training sample set, wherein the training sample set includes a plurality of sample area sub-images and a road surface type label corresponding to each of the plurality of sample area sub-images; The initial road surface type recognition model is trained based on the training sample set to obtain the road surface type recognition model.

11. The method according to any one of claims 1 to 10, characterized in that: The road surface type includes at least one of the following: Regular roads, icy and snowy roads, sandy roads, dry dirt roads, muddy roads, wading roads, grass, gravel roads, and rocky roads.

12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: When the computer-executable instructions stored in the computer-readable storage medium are executed by a processor of a device, the device is capable of performing the method according to any one of claims 1 to 11.

14. A vehicle, characterized in that: include: The electronic device as claimed in claim 12, or the computer-readable storage medium as claimed in claim 13.

15. A computer program product, comprising computer instructions, characterized in that: When the computer instructions are executed on a processor of a device, the device is enabled to perform the method according to any one of claims 1 to 11.

Images