Road recognition method, device and equipment, storage medium and program product

Abstract

The invention provides a road identification method and device, equipment, a storage medium and a program product, and relates to the technical field of data processing, the method comprises the following steps: extracting a plurality of feature maps corresponding to a first route from a to-be-identified image, the plurality of feature maps being used for representing the type and / or color of the first route, the condition of the position of the first route is represented, and the first route is a lane line or a road boundary in the to-be-recognized image; generating a first instance of the first route and a first position corresponding to the first instance based on the plurality of feature maps; and identifying a first type and / or a first color corresponding to the first instance, wherein the first type and / or the first color corresponding to the first instance are / is identified based on the plurality of feature maps and a first position corresponding to the first instance.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a road recognition method, apparatus, device, storage medium, and program product. Background Technology

[0002] Road recognition is a key technology in the field of vehicle-assisted driving, enabling vehicles to drive stably on roads by identifying the road's path. Related technologies identify regions of interest (ROIs) for lane lines and road boundaries from acquired road images, and then use these ROIs to identify lane lines and road boundaries. However, the identification of ROIs for lane lines and road boundaries is affected by environmental factors; for example, excessively bright or dark weather can prevent accurate ROI identification from capturing the road images, leading to low road recognition accuracy.

[0003] It is evident that the relevant technologies suffer from low accuracy in road recognition. Summary of the Invention

[0004] This invention provides a road recognition method, apparatus, device, storage medium, and program product to address the problem of low accuracy in road recognition in related technologies.

[0005] To solve the above problems, the present invention is implemented as follows:

[0006] In a first aspect, embodiments of the present invention provide a road recognition method, comprising:

[0007] Multiple feature maps corresponding to the first route are extracted from the image to be identified. The multiple feature maps are used to characterize the type and / or color of the first route, and to characterize the location of the first route. The first route is the lane line or road boundary in the image to be identified.

[0008] A first instance of the first route and a first position corresponding to the first instance are generated based on the multiple feature maps;

[0009] Identify the first type and / or first color corresponding to the first instance, which is obtained based on the plurality of feature maps and the first position corresponding to the first instance.

[0010] Secondly, embodiments of the present invention also provide a road recognition device, comprising:

[0011] The extraction module is used to extract multiple feature maps corresponding to a first route from the image to be identified. The multiple feature maps are used to characterize the type and / or color of the first route, as well as the location of the first route. The first route is a lane line or road boundary in the image to be identified.

[0012] The generation module is used to generate a first instance of the first route and a first position corresponding to the first instance based on the plurality of feature maps;

[0013] The identification module is used to identify the first type and / or the first color corresponding to the first instance, wherein the first type and / or the first color corresponding to the first instance are identified based on the plurality of feature maps and the first position corresponding to the first instance.

[0014] Thirdly, embodiments of the present invention also provide an electronic device, including a transceiver and a processor.

[0015] The processor is used to extract multiple feature maps corresponding to a first route from the image to be identified. The multiple feature maps are used to characterize the type and / or color of the first route, and to characterize the location of the first route. The first route is a lane line or road boundary in the image to be identified.

[0016] The processor is further configured to generate a first instance of the first route and a first position corresponding to the first instance based on the plurality of feature maps;

[0017] The processor is further configured to identify a first type and / or a first color corresponding to the first instance, wherein the first type and / or the first color corresponding to the first instance are identified based on the plurality of feature maps and the first position corresponding to the first instance.

[0018] Fourthly, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the road recognition method described in the first aspect.

[0019] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the road recognition method described in the first aspect.

[0020] In a sixth aspect, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the road recognition method described in the first aspect.

[0021] In this embodiment of the invention, multiple feature maps corresponding to a first route are extracted from the image to be identified. These multiple feature maps are used to characterize the type and / or color of the first route, and to characterize the location of the first route. The first route is a lane line or road boundary in the image to be identified. A first instance of the first route and a first location corresponding to the first instance are generated based on the multiple feature maps. A first type and / or first color corresponding to the first instance are identified, which is obtained based on the multiple feature maps and the first location corresponding to the first instance. Thus, by extracting multiple feature maps corresponding to multiple first routes, generating a first instance and a first location corresponding to the first instance using these multiple feature maps, and then combining these multiple feature maps to identify the type and / or color corresponding to the first instance, the location, type, and / or color of the road can be identified using multiple feature maps. Furthermore, the environmental requirements for the image to be identified are low during the extraction of multiple feature maps, and the accuracy of the feature maps is not affected, thereby effectively improving the accuracy of the finally identified road. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a road recognition method provided in an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of the image to be identified provided in an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of lane lines provided in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the road boundary provided in an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the first offset value and the second offset value provided in an embodiment of the present invention;

[0028] Figure 6 This is a flowchart of generating the first instance provided in an embodiment of the present invention;

[0029] Figure 7 This is a schematic diagram illustrating the filtering of multiple intermediate instances provided in an embodiment of the present invention;

[0030] Figure 8 This is a schematic diagram of distance calculation provided in an embodiment of the present invention;

[0031] Figure 9 This is a structural diagram of the preset model provided in the embodiments of the present invention;

[0032] Figure 10 This is a structural diagram of a road recognition device provided in an embodiment of the present invention;

[0033] Figure 11 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Please see Figure 1 , Figure 1 This is a flowchart of a road recognition method provided in an embodiment of the present invention, such as... Figure 1 As shown, it includes the following steps:

[0036] Step 101: Extract multiple feature maps corresponding to the first route from the image to be identified. The multiple feature maps are used to characterize the type and / or color of the first route, as well as the location of the first route. The first route is the lane line or road boundary in the image to be identified.

[0037] The aforementioned image to be identified is a road image of a vehicle traveling on. In some embodiments, the image to be identified can be extracted from video footage taken during vehicle travel (e.g., video footage taken with a monocular camera). For example, images to be identified are collected from the video footage at preset intervals, and the collected images to be identified are then identified sequentially to obtain the actual road conditions, thereby enabling assisted driving of the vehicle.

[0038] The first route mentioned above is the route that needs to be identified in the road, such as lane lines and road boundaries. The route to be identified is different in different roads. For example, urban roads include lane lines and road boundaries, highways may only include lane lines, and rural roads may only include road boundaries. Therefore, it is necessary to extract features from the first route in the image to be identified to obtain multiple feature maps in order to determine the specific situation of the first route.

[0039] The aforementioned multiple feature maps are feature maps of the first route, which are used to characterize the type of the first route (e.g., solid line type, dashed line type, etc.), and / or the color of the first route (e.g., white, yellow, blue, etc.), as well as the position of the first route. By characterizing the type and / or color of the first route and the position of the first route through multiple feature maps, the type and / or color of the first route and the position of the first route can be determined through the feature maps, so as to achieve accurate identification of the first route.

[0040] Step 102: Generate a first instance of the first route and a first position corresponding to the first instance based on the multiple feature maps.

[0041] The first instance mentioned above is an instance of the first route. When generating the first instance based on multiple feature maps, a first position corresponding to the first instance will be generated. The first position is the location of the first route. Through the first instance and the first position corresponding to the first instance, the position of the vehicle relative to the road can be determined, and then the driving plan can be determined based on the first position corresponding to the first instance.

[0042] It should be noted that if the image to be identified includes multiple routes, it is necessary to generate an instance corresponding to each route and the location of each instance based on multiple feature maps, and then determine the driving plan based on each instance and the location of each instance.

[0043] like Figure 2 As shown, the image to be identified is an image of an urban road, including multiple lane lines and road boundaries. When generating instances, it is necessary to generate an instance corresponding to each lane line and road boundary, as well as the location of each instance, so that a driving plan can be determined based on each instance and its corresponding location.

[0044] Step 103: Identify the first type and / or first color corresponding to the first instance. The first type and / or first color corresponding to the first instance are identified based on the plurality of feature maps and the first position corresponding to the first instance.

[0045] Given the first instance and its corresponding first position, the position of the first route is determined. The first position corresponding to the first instance is then combined with multiple feature images to identify the type and / or color of the first route.

[0046] In some implementations, the multiple feature maps may include a type feature map and / or a color feature map. The type feature map is used to characterize the type of the first route, and the type of the first route can be determined by the type feature map and the first position. The color feature map is used to characterize the color of the first route, and the color of the first route can be determined by the color feature map and the first position.

[0047] The type of the first route can be determined by the type feature map and the first position, which may include the following steps:

[0048] (1) Divide the type feature map into n windows, where the number of windows n and the height h of each window are custom parameters; (2) For all pixels of the first instance in each window, take the type with the highest frequency as the type of the first instance in that window; (3) Summarize the types of the first instance in all windows to obtain the recognition result.

[0049] It should be noted that when the number of windows n=1, each instance outputs only one type. When the number of windows n>1, each instance outputs multiple types, representing the types of roads at different distances.

[0050] The method for determining the color of the first route using the color feature map and the first position is the same as the method for determining the type of the first route using the type feature map and the first position, and specifically includes the following steps:

[0051] (1) Divide the color feature map into n windows, where the number of windows n and the height h of each window are custom parameters; (2) For all pixels of the first instance in each window, take the color with the highest frequency as the color of the first instance in that window; (3) Summarize the colors of the first instance in all windows to obtain the recognition result.

[0052] It should be noted that when the number of windows n=1, each instance outputs only one color. When the number of windows n>1, each instance outputs multiple colors, representing the colors of the road at different distances.

[0053] In this embodiment of the invention, multiple feature maps corresponding to a first route are extracted from the image to be identified. These multiple feature maps are used to characterize the type and / or color of the first route, and to characterize the location of the first route. The first route is a lane line or road boundary in the image to be identified. A first instance of the first route and a first location corresponding to the first instance are generated based on the multiple feature maps. A first type and / or first color corresponding to the first instance are identified, which is obtained based on the multiple feature maps and the first location corresponding to the first instance. Thus, by extracting multiple feature maps corresponding to multiple first routes, generating a first instance and a first location corresponding to the first instance using these multiple feature maps, and then combining these multiple feature maps to identify the type and / or color corresponding to the first instance, the location, type, and / or color of the road can be identified using multiple feature maps. Furthermore, the environmental requirements for the image to be identified are low during the extraction of multiple feature maps, and the accuracy of the feature maps is not affected, thereby effectively improving the accuracy of the finally identified road.

[0054] In one embodiment, the plurality of feature maps include a score feature map and an offset feature map. The score feature map characterizes the shape of the first route in the image to be identified, and the offset feature map characterizes the offset of the first route relative to a grid, wherein the grid is a grid segmented from the plurality of feature maps. Generating a first instance of the first route based on the plurality of feature maps includes:

[0055] The grid of the scoring feature map is traversed according to a preset order;

[0056] When traversing to the first grid, the first score corresponding to the first grid is determined based on the score feature map, where the first grid is a grid in an uncoded state;

[0057] If the first score is greater than a set score threshold, a first offset value and a second offset value of the first grid are determined based on the offset feature map. The first offset value is used to characterize the offset of the first route in the first grid relative to the first grid, and the second offset value is used to characterize the offset of the first route in the second grid relative to the first grid. The second grid is the grid adjacent to the first grid.

[0058] A first intermediate instance is generated based on the first offset value, and the intermediate position corresponding to the first intermediate instance is also generated. The first grid is then set to an encoded state.

[0059] The first instance is generated based on the second grid and the first intermediate instance.

[0060] The aforementioned scoring feature map is used to characterize the shape of the first route in the image to be identified. The location of the first route in the image can be determined through the scoring feature map. In some embodiments, the scoring feature map of the lane lines can be as follows: Figure 3 As shown, the score feature map of the road boundary can be as follows: Figure 4 As shown.

[0061] In the scoring feature map, the score is marked as 1 at the location where the first route is located, and as 0 at the location where the first route does not exist, thus obtaining the scoring feature map used to characterize the first route.

[0062] The aforementioned offset feature map is used to characterize the offset of the first route relative to the grid, where the grid is a segmentation of the feature map. For example, as shown... Figure 5 As shown, the feature map is divided into 2×2 grids, with each grid containing 4×4 pixels. The offset feature map is used to characterize the offset of the first route relative to the grid, so that the score feature map and the offset feature map can be combined to obtain the first position corresponding to the first instance.

[0063] In some implementations, the offset feature map includes a first offset value and a second offset value. The first offset value is the lateral or vertical offset value of the first route in the first part of the first grid relative to the center of the first grid. The second offset value is the lateral or vertical offset value of the first route in the second part of the second grid relative to the center of the first grid. In this way, the offset of the first route in the first grid can be determined by the first offset value, and the second grid in which the first route is located can be determined by the second offset value. The first grid and the second grid are adjacent, thereby realizing the continuous recording of the first route so as to obtain a complete first instance later.

[0064] The first offset value and the second offset value can be the horizontal offset value of each pixel of the first route within the grid relative to the center of the grid. For example, the first offset value and the second offset value of the first route can be represented as (-1.5, -1, -0.5, +0.5, +2, +3.5, +4.5, +5), where the first 4 offset values ​​are the first offset value and the last 4 offset values ​​are the second offset value.

[0065] It should be noted that the score feature map is used to characterize the shape of the first route, while the offset feature map is used to characterize the offset of the first route relative to the grid. By combining the score feature map and the offset feature map, the shape and position of the instances in all grids can be determined, thereby obtaining the first instance and the position information corresponding to the first instance.

[0066] Specifically, such as Figure 6As shown, since both the scoring feature map and the offset feature map are segmented using grids, the grids of the scoring feature map are traversed in a preset order to determine the first instance and its corresponding position based on the content of the scoring feature map and the offset feature map within the same grid. The preset order can be set according to requirements. Considering that it is more necessary to determine the road conditions close to the vehicle in images acquired during vehicle movement, the preset order can be a bottom-up order, traversing each row of the grid sequentially.

[0067] Specifically, for meshes that have already been traversed and processed, they can be set to an encoded state; for unprocessed meshes, they can be set to an unencoded state, so that only unencoded meshes are processed during the processing, avoiding duplicate processing.

[0068] Furthermore, for grids in an uncoded state, the existence of a first instance within the current grid is determined using the score feature map. Specifically, a set score threshold is set, and the first score corresponding to the first grid is determined using the score feature map. If the first score is greater than the set score threshold, it is determined that a first instance exists within the first grid. Then, the location of the first instance within the first grid can be determined based on the offset feature map.

[0069] Specifically, the first offset value and the second offset value of the first grid are determined by the offset feature map. Since the first offset value is used to characterize the offset of the first route relative to the first grid, and the second offset value is used to characterize the offset of the first route relative to the second grid, the first intermediate instance of the first route in the first grid can be generated by the first offset value, and the second offset value can be used to determine the second grid in which other parts of the first instance may be located.

[0070] After processing the first grid, it needs to be set to an encoded state to avoid repeated processing of the first grid in subsequent iterations.

[0071] In this embodiment of the invention, the grids of the scoring feature map are traversed according to a preset order. When the first grid is reached, a first score is determined based on the scoring feature map, where the first grid is an uncoded grid. If the first score is greater than a set score threshold, a first offset value and a second offset value are determined based on the offset feature map. The first offset value represents the offset of the first route in the first grid relative to the first grid, and the second offset value represents the offset of the first route in the second grid relative to the first grid, where the second grid is an adjacent grid to the first grid. A first intermediate instance and its corresponding intermediate position are generated based on the first offset value, and the first grid is set to an encoded state. The first instance is generated based on the second grid and the first intermediate instance. Thus, the first instance and its corresponding first position are obtained through the scoring feature map and the offset feature map.

[0072] In one embodiment, generating the first instance based on the second mesh and the first intermediate instance includes:

[0073] When the second grid is in an uncoded state, the second score corresponding to the second grid is determined based on the score feature map;

[0074] If the second score is greater than a set score threshold, a third offset value and a fourth offset value of the second grid are determined based on the offset feature map. The third offset value is used to characterize the offset of the first route in the second grid relative to the second grid, and the fourth offset value is used to characterize the offset of the first route in the third grid relative to the second grid. The third grid is the grid adjacent to the second grid.

[0075] A second intermediate instance is generated based on the third offset value, along with the intermediate position corresponding to the second intermediate instance, and the second grid is set to an encoded state.

[0076] The first instance is generated based on the third grid, the first intermediate instance, and the second intermediate instance.

[0077] The required pattern is that after obtaining the first intermediate instance within the first grid, since the second grid is determined by the second offset value, it is also necessary to determine whether there is a second intermediate instance within the second grid, so as to obtain the complete first instance.

[0078] Specifically, in this embodiment of the invention, a second intermediate instance within a second grid is determined using a score feature map and an offset feature map. Specifically, a second score is determined using the score feature map; if the second score is greater than a set score threshold, a second intermediate feature map is determined to exist within the second grid. Based on this, a third offset value and a fourth offset value (relative to the first and second offset values ​​of the first grid) are determined using the offset feature map. The third offset value is used to generate a second intermediate instance, and the fourth offset value is used to determine a third grid where other intermediate instances may exist, facilitating the further generation of other intermediate instances.

[0079] In this embodiment of the invention, when the second grid is in an uncoded state, a second score corresponding to the second grid is determined based on the score feature map; when the second score is greater than a set score threshold, a third offset value and a fourth offset value of the second grid are determined based on the offset feature map. The third offset value is used to characterize the offset of the first route in the second grid relative to the second grid, and the fourth offset value is used to characterize the offset of the first route in the third grid relative to the second grid. The third grid is a grid adjacent to the second grid. A second intermediate instance and its corresponding intermediate position are generated based on the third offset value, and the second grid is set to an encoded state. The first instance is generated based on the third grid, the first intermediate instance, and the second intermediate instance. In this way, the first intermediate instance, the second intermediate instance, and other intermediate instances are generated through the score feature map and the offset feature map, thereby obtaining a complete first instance.

[0080] That is, if only the first intermediate instance exists and the first instance does not exist, the first instance is generated based on the first intermediate instance; if the first instance already exists, the first instance is updated based on the first intermediate instance.

[0081] In one embodiment, generating the first instance based on the second mesh and the first intermediate instance includes:

[0082] If the second grid is in an encoded state, or if the second score corresponding to the second grid is less than or equal to a set score threshold, the first instance is generated based on the first intermediate instance.

[0083] Understandably, if the second grid is already encoded, or if the second score corresponding to the second grid is less than or equal to a set score threshold, then it can be assumed that the second grid does not contain a second intermediate instance, and the already obtained first intermediate instance is the entirety of the first instance. Similarly, if the second grid contains intermediate instances, it is possible to continue to determine whether there are intermediate instances for newly obtained grids, until the new grid obtained through the offset feature map does not contain intermediate instances. At this point, all intermediate instances can be obtained, and then all intermediate instances are concatenated to obtain the first instance and the first position corresponding to the first instance.

[0084] After obtaining the first instance, the grid is traversed again, and new instances are obtained through the method of the present invention, until all grids are processed into an encoded state, thereby realizing the recognition of all route-corresponding instances in the image to be recognized, as well as the recognition of the corresponding positions of the instances.

[0085] In one embodiment, generating a first instance of the first route based on the plurality of feature maps includes:

[0086] Multiple initial instances of the first route are generated based on the multiple feature maps;

[0087] Multiple intermediate instances are determined from the plurality of initial instances, wherein the height of each intermediate instance corresponding to a first position is greater than a set height threshold.

[0088] Calculate the distance between the plurality of intermediate instances and the third intermediate instance, where the third intermediate instance is the intermediate instance with the highest corresponding score among the plurality of intermediate instances;

[0089] The first instance is selected from the plurality of intermediate instances whose distance from the third intermediate instance is less than a set distance threshold.

[0090] It should be noted that multiple initial instances may be generated during the instance generation process. However, there may be invalid or misidentified instances among these initial instances. Therefore, it is necessary to filter the multiple initial instances to obtain the required instance.

[0091] Specifically, in this embodiment of the invention, a height threshold and a distance threshold are set to filter and obtain instances that meet the requirements.

[0092] In some implementations, the first instance can be selected using a non-maximum suppression (NMS) algorithm. For example... Figure 7As shown, an empty set can be pre-initialized. After obtaining multiple intermediate instances based on a set height threshold, these intermediate instances are set into an instance set. Each intermediate instance is sorted based on its score, and the intermediate instance with the highest score (i.e., the third intermediate instance) is determined. The distance between the other intermediate instances and the third intermediate instance is calculated. By filtering instances whose distance to the third intermediate instance is less than a set distance threshold, instances in the instance set that are greater than the set distance threshold are deleted to obtain the first instance, which is then added to the empty set.

[0093] Furthermore, a third intermediate instance can be added to the empty set, and a new intermediate instance with the highest score can be determined from the instance set. This filtering process can be repeated until there are no instances left in the instance set. At this point, the instances added to the initial empty set are all the required instances, including the first instance.

[0094] The calculation of the distance between the plurality of intermediate instances and the third intermediate instance can be the distance between the center position of the plurality of intermediate instances and the center of the third intermediate instance, or it can be the minimum distance between the plurality of intermediate instances and the third intermediate instance.

[0095] In some implementations, such as Figure 8 As shown, calculating the distance between the plurality of intermediate instances and the third intermediate instance can be achieved by first determining the height h of the common portion overlapping between each intermediate instance and the third intermediate instance in the vertical direction, then determining the area S of the region formed by the common portion between each intermediate instance and the third intermediate instance, and setting the distance as the quotient of the area S and the height h of the common portion. Wherein, if there is no common portion between the instance and the third intermediate instance in the vertical direction, the distance between the instance and the third instance is considered to be infinite.

[0096] In this embodiment of the invention, multiple initial instances of the first route are generated based on the multiple feature maps; multiple intermediate instances are determined from the multiple initial instances, wherein the height of each intermediate instance corresponding to a first position is greater than a set height threshold; the distance between the multiple intermediate instances and a third intermediate instance is calculated, wherein the third intermediate instance is the intermediate instance with the highest corresponding score among the multiple intermediate instances; the first instance is filtered, wherein the first instance is the instance among the multiple intermediate instances whose distance to the third intermediate instance is less than a set distance threshold. Thus, by setting a height threshold and a distance threshold, the first instance is obtained through filtering, while other invalid or misidentified instances are removed.

[0097] In one embodiment, extracting multiple feature maps corresponding to the first route from the image to be recognized includes:

[0098] Based on a preset model, feature extraction is performed on the image to be identified to obtain the multiple feature maps corresponding to the first route;

[0099] The preset model is obtained in the following way:

[0100] Acquire sample training data, which includes sample images and sample feature maps corresponding to the sample images. The sample feature maps include at least one of sample score feature maps, sample offset feature maps, sample type feature maps, and sample color feature maps.

[0101] The initial model is trained based on the sample training data to obtain an intermediate training model. The initial model is used to extract feature maps corresponding to the images.

[0102] Calculate the loss value corresponding to the intermediate training model;

[0103] If the loss value is less than the set loss threshold, the intermediate training model is set as the preset model.

[0104] In this embodiment of the invention, sample training data is acquired, including sample images and sample feature maps corresponding to the images. The sample feature maps include at least one of a sample score feature map, a sample offset feature map, a sample type feature map, and a sample color feature map. An initial model is trained based on the sample training data to obtain an intermediate training model. The initial model is used to extract feature maps corresponding to the images. A loss value corresponding to the intermediate training model is calculated. If the loss value is less than a set loss threshold, the intermediate training model is set as the preset model. Thus, by pre-training to obtain the preset model, multiple feature maps corresponding to the image to be recognized are extracted during road recognition.

[0105] In one embodiment, the sample score feature map is obtained as follows:

[0106] The location of the first route in the sample image is labeled to obtain the sample score feature map; and / or,

[0107] The sample type feature map is obtained in the following way:

[0108] The type of the first route in the sample score feature map is labeled to obtain the sample type feature map; and / or,

[0109] The sample color feature map is obtained in the following way:

[0110] The color of the first route in the sample score feature map is labeled to obtain the sample color feature map.

[0111] It should be noted that the sample score feature map, sample type feature map, and sample color feature map are constructed using the same approach: directly labeling the location of the route to obtain the sample score feature map, sample type feature map, or sample color feature map.

[0112] For example, for an initial image of size W×H, it is first downsampled by a factor of r to obtain an image with a width of W / r and a height of H / r. The location of the first route in this image is first marked, and then further marked to obtain a sample score feature map, a sample type feature map, or a sample color feature map.

[0113] For the sample score feature map, the position where the first route is located is marked as 1 (indicating that the first route exists at this position), and other positions are marked as 0. At this time, the sample score feature map is a three-dimensional array with shape (W / r, H / r, 1).

[0114] For sample type feature maps or sample color feature maps, mark the location of the first route as 1, 2 or 3 (indicating the type or color of the first route at that location), and mark other locations as 0 to obtain the sample type feature map or sample color feature map.

[0115] In one embodiment, the sample offset feature map is obtained as follows:

[0116] The sample score feature map is divided based on a preset grid;

[0117] Calculate the first offset value and the second offset value for each of the multiple grids after division, wherein the multiple grids include a fourth grid, the first offset value of the fourth grid is the offset value of the first route relative to the fourth grid, the second offset value of the fourth grid is the offset value of the first route relative to the fifth grid, and the fourth grid and the fifth grid are adjacent;

[0118] The sample offset feature map is constructed based on the first and second offset values ​​of the multiple grids.

[0119] It should be noted that, unlike the construction approach of sample score feature map, sample type feature map and sample color feature map, the sample offset feature map needs to calculate the first offset value and the second offset value before labeling, so that the first offset value and the second offset value exist in the offset feature map extracted by the preset model later.

[0120] For example, the sample offset feature map is a three-dimensional array of (W / r, H / r, 2r), typically r=4. Each pixel in the sample offset feature map corresponds to a grid of r x r pixels in the original image.

[0121] First, for the pixel points with a sample score feature map of 0, the r offset values at this pixel position are all set to 0; for the pixel points with all pixel values of 1 in the sample score feature map, it is necessary to calculate the lateral offset values of the n routes in the grid corresponding to this pixel relative to the grid center as the first r offset values at this pixel position. Usually, n = r, and when n < r, 0 is filled in the missing positions.

[0122] Then, calculate the lateral offset values of the n center points of the lane lines in the row above the grid corresponding to this pixel relative to the original grid center as the last r offset values at this pixel position. Similarly, usually, n = r, and when n < r, 0 is filled in the missing positions. For example, as Figure 5 shown, the left side is a small part of the original image, containing 8x8 pixels, that is, 2x2 grids, and the right side is a small part of the corresponding ground truth map of lane line offsets, which is a three-dimensional array of (2, 2, 8). The red part of the ground truth map of lane line offsets corresponds to the red grid in the original image. Through the above encoding process, the ground truth of the lane line offsets in the red part is obtained as (-1.5, -1, -0.5, +0.5, +2, +3.5, +4.5, +5), where the first 4 offset values are the lateral offset values of the center point of the current lane line relative to the grid center, and the last 4 offset values are the lateral offset values of the center point of the lane line in the row above relative to the grid center.

[0123] In an embodiment of the present invention, the sample score feature map is divided based on a preset grid; the first offset value and the second offset value corresponding to each of the multiple grids after division are calculated, the multiple grids include a fourth grid, the first offset value of the fourth grid is the offset value of the first route relative to the fourth grid, the second offset value of the fourth grid is the offset value of the first route relative to the fifth grid, and the fourth grid and the fifth grid are adjacent; the sample offset feature map is constructed based on the first offset value and the second offset value of the multiple grids. In this way, by calculating the first offset value and the second offset value and then constructing the sample offset feature map, it is convenient for the offset feature map obtained by subsequent extraction through a preset model to have the first offset value and the second offset value.

[0124] In one embodiment, the calculating the loss value corresponding to the intermediate training model includes:

[0125] Based on the intermediate training model, feature extraction is performed on the sample image to obtain a predicted score feature map, a predicted offset feature map, a predicted type feature map, and / or a predicted color feature map;

[0126] A first loss value is calculated based on the sample score feature map and the predicted score feature map; and / or, a second loss value is calculated based on the sample offset feature map and the predicted offset feature map; and / or, a third loss value is calculated based on the sample type feature map and the predicted type feature map; and / or, a fourth loss value is calculated based on the sample color feature map and the predicted color feature map.

[0127] The first loss value, the second loss value, the third loss value, and / or the fourth loss value are weighted to obtain the loss value.

[0128] In this embodiment of the invention, since the preset model needs to extract multiple feature maps, a corresponding loss value needs to be set for each feature map during the training process, and the loss value corresponding to the model is obtained by weighting, so that the preset model after training can accurately extract the score feature map, offset feature map, type feature map and / or color feature map.

[0129] In some implementations, the loss values ​​for lane lines and road boundaries can be set separately for the first route, and then weighted to obtain the loss value of the preset model. For example, the loss value of the preset model can be expressed by the following formula:

[0130] L=k1L lane_score +k2L lane_offset +k3L lane_type +k4L lane_color +k5L curb_score +k6L curb_offset +k7L curb_type ;

[0131] In the formula, k1 to k7 are weights, and L lane_score L represents the loss value corresponding to the score feature map of the lane lines. lane_offset L represents the loss value corresponding to the lane offset feature map. lane_type L represents the loss value corresponding to the lane line type feature map. lane_color L represents the loss value corresponding to the color feature map of the lane lines. curb_score L represents the loss value corresponding to the score feature map of the road boundary. curb_offset L represents the loss value corresponding to the offset feature map of the road boundary. curb_type This represents the loss value corresponding to the type feature map of the road boundary.

[0132] In some implementations, L lane_score and L curb_score It can be obtained using the focal loss function, which can be expressed by the following formula:

[0133] ;

[0134] In the formula, p t α represents the probability of a correct prediction. t And γ are hyperparameters, FL(p t ) is the loss value (i.e., L) lane_score or L curb_score ).

[0135] In some implementations, L lane_offset or L curb_offset It can be obtained using the smooth-L1 loss function, which can be expressed by the following formula:

[0136] ;

[0137] In the formula The hyperparameters are x, which is the predicted offset feature map, and smooth-L1(x), which is the loss value (i.e., L). lane_offset or L curb_offset ).

[0138] In some implementations, L lane_type L lane_color and L curb_type It can be calculated using the cross-entropy loss function.

[0139] In one embodiment, the preset model includes a plurality of first convolutional modules and a plurality of second convolutional modules connected in sequence. Each of the plurality of first convolutional modules is skip-connected to a second convolutional module with the same input size. The plurality of first convolutional modules are used to extract features of different scales of the image, and the plurality of second convolutional modules are used to output the plurality of feature maps.

[0140] It should be noted that the preset model processes the input image and outputs lane score map, lane offset map, lane type map, lane color map, road boundary score map, road boundary offset map, and road boundary type map. Each feature map is actually a three-dimensional vector. Therefore, the structure of the preset model in this embodiment is designed as follows: Figure 9 As shown, it includes multiple first convolutional modules and multiple second convolutional modules. The multiple first convolutional modules are used to extract features of different scales of the image, and the multiple second convolutional modules are used to output the multiple feature maps.

[0141] The system comprises multiple first convolutional modules and multiple second convolutional modules, each including a convolutional layer with a 3x3 kernel size and a stride of 1, a batch normalization layer, and a Rectified Linear Unit (ReLU) activation function layer. Each first convolutional module (except the last one) is followed by a downsampling convolutional layer, specifically a convolutional layer with a stride of 2 or a max pooling layer, to downsample the vector output by each first convolutional module. Each second convolutional module (except the last one) is followed by an upsampling convolutional layer, including a deconvolutional layer with a stride of 2 or a bilinear interpolation layer, to upsampling the vector output by each second convolutional module.

[0142] Furthermore, each of the multiple first convolutional modules is hop-connected to a second convolutional module with the same input size, thereby achieving the fusion of features at different scales of the image, taking into account both spatial local detail features and spatial context features.

[0143] In some implementations, to reduce computational cost, a third convolutional module is added after the last second convolutional module. The third convolutional module includes a convolutional layer with a kernel size of 1x1 and a stride of 1, a batch normalization layer, and a rectified linear unit (ReLU) activation function layer. This way, multiple feature maps are output through the third convolutional module, discarding the final upsampling part. This avoids adding extra computation to the high-resolution feature maps, which could affect the model's inference speed and effectively improve the model's inference efficiency.

[0144] Please see Figure 10 , Figure 10 This is a structural diagram of a road recognition device provided in an embodiment of the present invention, as shown below. Figure 10 As shown, the road recognition device 1000 includes:

[0145] The extraction module 1001 is used to extract multiple feature maps corresponding to a first route from the image to be identified. The multiple feature maps are used to characterize the type and / or color of the first route, and to characterize the position of the first route. The first route is a lane line or road boundary in the image to be identified.

[0146] The generation module 1002 is used to generate a first instance of the first route and a first position corresponding to the first instance based on the plurality of feature maps;

[0147] The identification module 1003 is used to identify the first type and / or the first color corresponding to the first instance, wherein the first type and / or the first color corresponding to the first instance are identified based on the plurality of feature maps and the first position corresponding to the first instance.

[0148] In one embodiment, the plurality of feature maps include a score feature map and an offset feature map. The score feature map is used to characterize the shape of the first route in the image to be identified, and the offset feature map is used to characterize the offset of the first route relative to a grid. The grid is a grid used to segment the plurality of feature maps. The generation module 1002 includes:

[0149] The traversal submodule is used to traverse the grid of the scoring feature map according to a preset order;

[0150] The first determining submodule is used to determine the first score corresponding to the first grid based on the scoring feature map when traversing to the first grid, wherein the first grid is a grid in an uncoded state;

[0151] The second determining submodule is used to determine a first offset value and a second offset value of the first grid based on the offset feature map when the first score is greater than a set score threshold. The first offset value is used to characterize the offset of the first route in the first grid relative to the first grid, and the second offset value is used to characterize the offset of the first route in the second grid relative to the first grid. The second grid is a grid adjacent to the first grid.

[0152] The first generation submodule is used to generate a first intermediate instance and the intermediate position corresponding to the first intermediate instance based on the first offset value, and set the first grid to an encoded state;

[0153] The second generation submodule is used to generate the first instance based on the second grid and the first intermediate instance.

[0154] In one embodiment, the second generation submodule includes:

[0155] The first determining unit is configured to determine the second score corresponding to the second grid based on the score feature map when the second grid is in an uncoded state.

[0156] The second determining unit is used to determine a third offset value and a fourth offset value of the second grid based on the offset feature map when the second score is greater than a set score threshold. The third offset value is used to characterize the offset of the first route in the second grid relative to the second grid, and the fourth offset value is used to characterize the offset of the first route in the third grid relative to the second grid. The third grid is a grid adjacent to the second grid.

[0157] The first generation unit is used to generate a second intermediate instance and the intermediate position corresponding to the second intermediate instance based on the third offset value, and to set the second grid to an encoded state;

[0158] The second generation unit is used to generate the first instance based on the third grid, the first intermediate instance, and the second intermediate instance.

[0159] In one embodiment, generating the first instance based on the second mesh and the first intermediate instance includes:

[0160] The third generation unit is used to generate the first instance based on the first intermediate instance when the second grid is in an encoded state, or when the second score corresponding to the second grid is less than or equal to a set score threshold.

[0161] In one embodiment, the generation module 1002 includes:

[0162] The third generation submodule is used to generate multiple initial instances of the first route based on the multiple feature maps;

[0163] The third determining submodule is used to determine multiple intermediate instances from the multiple initial instances, wherein the height of each intermediate instance corresponding to the first position is greater than a set height threshold.

[0164] The calculation submodule is used to calculate the distance between the plurality of intermediate instances and the third intermediate instance, wherein the third intermediate instance is the intermediate instance with the highest corresponding score among the plurality of intermediate instances;

[0165] The filtering submodule is used to filter the first instance, which is an instance among the plurality of intermediate instances whose distance from the third intermediate instance is less than a set distance threshold.

[0166] In one embodiment, the extraction module 1001 includes:

[0167] The extraction submodule is used to extract features from the image to be identified based on a preset model, and obtain the multiple feature maps corresponding to the first route;

[0168] The preset model is obtained in the following way:

[0169] Acquire sample training data, which includes sample images and sample feature maps corresponding to the sample images. The sample feature maps include at least one of sample score feature maps, sample offset feature maps, sample type feature maps, and sample color feature maps.

[0170] The initial model is trained based on the sample training data to obtain an intermediate training model. The initial model is used to extract feature maps corresponding to the images.

[0171] Calculate the loss value corresponding to the intermediate training model;

[0172] If the loss value is less than the set loss threshold, the intermediate training model is set as the preset model.

[0173] In one embodiment, the sample score feature map is obtained as follows:

[0174] The location of the first route in the sample image is labeled to obtain the sample score feature map; and / or,

[0175] The sample type feature map is obtained in the following way:

[0176] The type of the first route in the sample score feature map is labeled to obtain the sample type feature map; and / or,

[0177] The sample color feature map is obtained in the following way:

[0178] The color of the first route in the sample score feature map is labeled to obtain the sample color feature map.

[0179] In one embodiment, the sample offset feature map is obtained as follows:

[0180] The sample score feature map is divided based on a preset grid;

[0181] Calculate the first offset value and the second offset value for each of the multiple grids after division, wherein the multiple grids include a fourth grid, the first offset value of the fourth grid is the offset value of the first route relative to the fourth grid, the second offset value of the fourth grid is the offset value of the first route relative to the fifth grid, and the fourth grid and the fifth grid are adjacent;

[0182] The sample offset feature map is constructed based on the first and second offset values ​​of the multiple grids.

[0183] In one embodiment, calculating the loss value corresponding to the intermediate trained model includes:

[0184] Based on the intermediate training model, feature extraction is performed on the sample image to obtain a predicted score feature map, a predicted offset feature map, a predicted type feature map, and / or a predicted color feature map.

[0185] A first loss value is calculated based on the sample score feature map and the predicted score feature map; and / or, a second loss value is calculated based on the sample offset feature map and the predicted offset feature map; and / or, a third loss value is calculated based on the sample type feature map and the predicted type feature map; and / or, a fourth loss value is calculated based on the sample color feature map and the predicted color feature map.

[0186] The first loss value, the second loss value, the third loss value, and / or the fourth loss value are weighted to obtain the loss value.

[0187] In one embodiment, the preset model includes a plurality of first convolutional modules and a plurality of second convolutional modules connected in sequence. Each of the plurality of first convolutional modules is skip-connected to a second convolutional module with the same input size. The plurality of first convolutional modules are used to extract features of different scales of the image, and the plurality of second convolutional modules are used to output the plurality of feature maps.

[0188] The road recognition device provided in this embodiment of the invention can realize each process of each embodiment of the above-mentioned road recognition method, with one-to-one correspondence of technical features and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0189] It should be noted that the road recognition device in the embodiments of the present invention can be a device, or it can be a component, integrated circuit, or chip in an electronic device.

[0190] This invention also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the above-described functionality. Figure 1 The various processes of the road recognition method embodiments shown are all capable of achieving the same technical effect, and will not be described again here to avoid repetition.

[0191] For details, see Figure 11 As shown, this embodiment of the invention also provides an electronic device, including a bus 1101, a transceiver 1102, an antenna 1103, a bus interface 1104, a processor 1105, and a memory 1106.

[0192] The processor 1105 is used to extract multiple feature maps corresponding to a first route from the image to be identified. The multiple feature maps are used to characterize the type and / or color of the first route, and to characterize the location of the first route. The first route is a lane line or road boundary in the image to be identified.

[0193] The processor 1105 is further configured to generate a first instance of the first route and a first position corresponding to the first instance based on the plurality of feature maps;

[0194] The processor 1105 is further configured to identify a first type and / or a first color corresponding to the first instance, wherein the first type and / or the first color corresponding to the first instance are identified based on the plurality of feature maps and the first position corresponding to the first instance.

[0195] In one embodiment, the plurality of feature maps include a score feature map and an offset feature map. The score feature map characterizes the shape of the first route in the image to be identified, and the offset feature map characterizes the offset of the first route relative to a grid, wherein the grid is a grid segmented from the plurality of feature maps. Generating a first instance of the first route based on the plurality of feature maps includes:

[0196] The grid of the scoring feature map is traversed according to a preset order;

[0197] When traversing to the first grid, the first score corresponding to the first grid is determined based on the score feature map, where the first grid is a grid in an uncoded state;

[0198] If the first score is greater than a set score threshold, a first offset value and a second offset value of the first grid are determined based on the offset feature map. The first offset value is used to characterize the offset of the first route in the first grid relative to the first grid, and the second offset value is used to characterize the offset of the first route in the second grid relative to the first grid. The second grid is the grid adjacent to the first grid.

[0199] A first intermediate instance is generated based on the first offset value, and the intermediate position corresponding to the first intermediate instance is also generated. The first grid is then set to an encoded state.

[0200] The first instance is generated based on the second grid and the first intermediate instance.

[0201] In one embodiment, generating the first instance based on the second mesh and the first intermediate instance includes:

[0202] When the second grid is in an uncoded state, the second score corresponding to the second grid is determined based on the score feature map;

[0203] If the second score is greater than a set score threshold, a third offset value and a fourth offset value of the second grid are determined based on the offset feature map. The third offset value is used to characterize the offset of the first route in the second grid relative to the second grid, and the fourth offset value is used to characterize the offset of the first route in the third grid relative to the second grid. The third grid is the grid adjacent to the second grid.

[0204] A second intermediate instance is generated based on the third offset value, along with the intermediate position corresponding to the second intermediate instance, and the second grid is set to an encoded state.

[0205] The first instance is generated based on the third grid, the first intermediate instance, and the second intermediate instance.

[0206] In one embodiment, generating the first instance based on the second mesh and the first intermediate instance includes:

[0207] If the second grid is in an encoded state, or if the second score corresponding to the second grid is less than or equal to a set score threshold, the first instance is generated based on the first intermediate instance.

[0208] In one embodiment, generating a first instance of the first route based on the plurality of feature maps includes:

[0209] Multiple initial instances of the first route are generated based on the multiple feature maps;

[0210] Multiple intermediate instances are determined from the plurality of initial instances, wherein the height of each intermediate instance corresponding to a first position is greater than a set height threshold.

[0211] Calculate the distance between the plurality of intermediate instances and the third intermediate instance, where the third intermediate instance is the intermediate instance with the highest corresponding score among the plurality of intermediate instances;

[0212] The first instance is selected from the plurality of intermediate instances whose distance from the third intermediate instance is less than a set distance threshold.

[0213] In one embodiment, extracting multiple feature maps corresponding to the first route from the image to be recognized includes:

[0214] Based on a preset model, feature extraction is performed on the image to be identified to obtain the multiple feature maps corresponding to the first route;

[0215] The preset model is obtained in the following way:

[0216] Acquire sample training data, which includes sample images and sample feature maps corresponding to the sample images. The sample feature maps include at least one of sample score feature maps, sample offset feature maps, sample type feature maps, and sample color feature maps.

[0217] The initial model is trained based on the sample training data to obtain an intermediate training model. The initial model is used to extract feature maps corresponding to the images.

[0218] Calculate the loss value corresponding to the intermediate training model;

[0219] If the loss value is less than the set loss threshold, the intermediate training model is set as the preset model.

[0220] In one embodiment, the sample score feature map is obtained as follows:

[0221] The location of the first route in the sample image is labeled to obtain the sample score feature map; and / or,

[0222] The sample type feature map is obtained in the following way:

[0223] The type of the first route in the sample score feature map is labeled to obtain the sample type feature map; and / or,

[0224] The sample color feature map is obtained in the following way:

[0225] The color of the first route in the sample score feature map is labeled to obtain the sample color feature map.

[0226] In one embodiment, the sample offset feature map is obtained as follows:

[0227] The sample score feature map is divided based on a preset grid;

[0228] Calculate the first offset value and the second offset value for each of the multiple grids after division, wherein the multiple grids include a fourth grid, the first offset value of the fourth grid is the offset value of the first route relative to the fourth grid, the second offset value of the fourth grid is the offset value of the first route relative to the fifth grid, and the fourth grid and the fifth grid are adjacent;

[0229] The sample offset feature map is constructed based on the first and second offset values ​​of the multiple grids.

[0230] In one embodiment, calculating the loss value corresponding to the intermediate trained model includes:

[0231] Based on the intermediate training model, feature extraction is performed on the sample image to obtain a predicted score feature map, a predicted offset feature map, a predicted type feature map, and / or a predicted color feature map.

[0232] A first loss value is calculated based on the sample score feature map and the predicted score feature map; and / or, a second loss value is calculated based on the sample offset feature map and the predicted offset feature map; and / or, a third loss value is calculated based on the sample type feature map and the predicted type feature map; and / or, a fourth loss value is calculated based on the sample color feature map and the predicted color feature map.

[0233] The first loss value, the second loss value, the third loss value, and / or the fourth loss value are weighted to obtain the loss value.

[0234] In one embodiment, the preset model includes a plurality of first convolutional modules and a plurality of second convolutional modules connected in sequence. Each of the plurality of first convolutional modules is skip-connected to a second convolutional module with the same input size. The plurality of first convolutional modules are used to extract features of different scales of the image, and the plurality of second convolutional modules are used to output the plurality of feature maps.

[0235] exist Figure 11 In this document, a bus architecture (represented by bus 1101) is used. Bus 1101 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 1105 and memory represented by memory 1106. Bus 1101 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 1104 provides an interface between bus 1101 and transceiver 1102. Transceiver 1102 may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 1105 is transmitted over a wireless medium via antenna 1103, which further receives data and transmits it to processor 1105.

[0236] Processor 1105 is responsible for managing bus 1101 and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 1106 can be used to store data used by processor 1105 during operation.

[0237] Optionally, the processor 1105 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU).

[0238] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the above-described functions. Figure 1 The various processes corresponding to the road recognition method embodiments, and which achieve the same technical effect, will not be described again here to avoid repetition. The computer-readable storage medium mentioned includes, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0239] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The various processes corresponding to the road recognition method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0240] In the embodiments of this invention, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing eight possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0241] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0242] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of this application.

[0243] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A road recognition method, characterized in that, include: Multiple feature maps corresponding to the first route are extracted from the image to be identified. The multiple feature maps are used to characterize the type and / or color of the first route, and to characterize the location of the first route. The first route is the lane line or road boundary in the image to be identified. A first instance of the first route and a first position corresponding to the first instance are generated based on the multiple feature maps; Identify the first type and / or first color corresponding to the first instance, which is obtained based on the plurality of feature maps and the first position corresponding to the first instance.

2. The method as described in claim 1, characterized in that, The plurality of feature maps include a score feature map and an offset feature map. The score feature map is used to characterize the shape of the first route in the image to be identified, and the offset feature map is used to characterize the offset of the first route relative to a grid. The grid is a grid segmented from the plurality of feature maps. Generating a first instance of the first route based on the plurality of feature maps includes: The grid of the scoring feature map is traversed according to a preset order; When traversing to the first grid, the first score corresponding to the first grid is determined based on the score feature map, where the first grid is a grid in an uncoded state; If the first score is greater than a set score threshold, a first offset value and a second offset value of the first grid are determined based on the offset feature map. The first offset value is used to characterize the offset of the first route in the first grid relative to the first grid, and the second offset value is used to characterize the offset of the first route in the second grid relative to the first grid. The second grid is the grid adjacent to the first grid. A first intermediate instance is generated based on the first offset value, and the intermediate position corresponding to the first intermediate instance is also generated. The first grid is then set to an encoded state. The first instance is generated based on the second grid and the first intermediate instance.

3. The method as described in claim 2, characterized in that, The generation of the first instance based on the second grid and the first intermediate instance includes: When the second grid is in an uncoded state, the second score corresponding to the second grid is determined based on the score feature map; If the second score is greater than a set score threshold, a third offset value and a fourth offset value of the second grid are determined based on the offset feature map. The third offset value is used to characterize the offset of the first route in the second grid relative to the second grid, and the fourth offset value is used to characterize the offset of the first route in the third grid relative to the second grid. The third grid is the grid adjacent to the second grid. A second intermediate instance is generated based on the third offset value, along with the intermediate position corresponding to the second intermediate instance, and the second grid is set to an encoded state. The first instance is generated based on the third grid, the first intermediate instance, and the second intermediate instance.

4. The method as described in claim 2, characterized in that, The generation of the first instance based on the second grid and the first intermediate instance includes: If the second grid is in an encoded state, or if the second score corresponding to the second grid is less than or equal to a set score threshold, the first instance is generated based on the first intermediate instance.

5. The method as described in claim 1, characterized in that, The generation of the first instance of the first route based on the plurality of feature maps includes: Multiple initial instances of the first route are generated based on the multiple feature maps; Multiple intermediate instances are determined from the plurality of initial instances, wherein the height of each intermediate instance corresponding to a first position is greater than a set height threshold. Calculate the distance between the plurality of intermediate instances and the third intermediate instance, where the third intermediate instance is the intermediate instance with the highest corresponding score among the plurality of intermediate instances; The first instance is selected, which includes instances among the plurality of intermediate instances whose distance to the third intermediate instance is less than a set distance threshold.

6. The method according to any one of claims 1 to 5, characterized in that, The extraction of multiple feature maps corresponding to the first route from the image to be recognized includes: Based on a preset model, feature extraction is performed on the image to be identified to obtain the multiple feature maps corresponding to the first route; The preset model is obtained in the following way: Acquire sample training data, which includes sample images and sample feature maps corresponding to the sample images. The sample feature maps include at least one of sample score feature maps, sample offset feature maps, sample type feature maps, and sample color feature maps. The initial model is trained based on the sample training data to obtain an intermediate training model. The initial model is used to extract feature maps corresponding to the images. Calculate the loss value corresponding to the intermediate training model; If the loss value is less than the set loss threshold, the intermediate training model is set as the preset model.

7. The method as described in claim 6, characterized in that, The sample score feature map is obtained in the following way: The location of the first route in the sample image is labeled to obtain the sample score feature map; and / or, The sample type feature map is obtained in the following way: The type of the first route in the sample score feature map is labeled to obtain the sample type feature map; and / or, The sample color feature map is obtained in the following way: The color of the first route in the sample score feature map is labeled to obtain the sample color feature map.

8. The method as described in claim 7, characterized in that, The sample offset feature map is obtained in the following way: The sample score feature map is divided based on a preset grid; Calculate the first offset value and the second offset value for each of the multiple grids after division, wherein the multiple grids include a fourth grid, the first offset value of the fourth grid is the offset value of the first route relative to the fourth grid, the second offset value of the fourth grid is the offset value of the first route relative to the fifth grid, and the fourth grid and the fifth grid are adjacent; The sample offset feature map is constructed based on the first and second offset values ​​of the multiple grids.

9. The method as described in claim 6, characterized in that, The calculation of the loss value corresponding to the intermediate training model includes: Based on the intermediate training model, feature extraction is performed on the sample image to obtain a predicted score feature map, a predicted offset feature map, a predicted type feature map, and / or a predicted color feature map. A first loss value is calculated based on the sample score feature map and the predicted score feature map; and / or, a second loss value is calculated based on the sample offset feature map and the predicted offset feature map; and / or, a third loss value is calculated based on the sample type feature map and the predicted type feature map; and / or, a fourth loss value is calculated based on the sample color feature map and the predicted color feature map. The first loss value, the second loss value, the third loss value, and / or the fourth loss value are weighted to obtain the loss value.

10. The method as described in claim 6, characterized in that, The preset model includes multiple first convolutional modules and multiple second convolutional modules connected in sequence. Each of the multiple first convolutional modules is skip-connected to a second convolutional module with the same input size. The multiple first convolutional modules are used to extract features of different scales of the image, and the multiple second convolutional modules are used to output the multiple feature maps.

11. A road recognition device, characterized in that, include: The extraction module is used to extract multiple feature maps corresponding to a first route from the image to be identified. The multiple feature maps are used to characterize the type and / or color of the first route, as well as the location of the first route. The first route is a lane line or road boundary in the image to be identified. The generation module is used to generate a first instance of the first route and a first position corresponding to the first instance based on the plurality of feature maps; The identification module is used to identify the first type and / or the first color corresponding to the first instance, wherein the first type and / or the first color corresponding to the first instance are identified based on the plurality of feature maps and the first position corresponding to the first instance.

12. An electronic device, characterized in that, Including transceivers and processors, The processor is used to extract multiple feature maps corresponding to a first route from the image to be identified. The multiple feature maps are used to characterize the type and / or color of the first route, and to characterize the location of the first route. The first route is a lane line or road boundary in the image to be identified. The processor is further configured to generate a first instance of the first route and a first position corresponding to the first instance based on the plurality of feature maps; The processor is further configured to identify a first type and / or a first color corresponding to the first instance, wherein the first type and / or the first color corresponding to the first instance are identified based on the plurality of feature maps and the first position corresponding to the first instance.

13. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the road recognition method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the road recognition method as described in any one of claims 1 to 10.

15. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the road recognition method as described in any one of claims 1 to 10.

Images