A signboard recognition method and device, computer equipment and storage medium

Abstract

This disclosure provides a sign recognition method, apparatus, computer device, and storage medium. The method includes: acquiring an image to be detected and inputting the image to be detected into a pre-trained target neural network to determine an initial feature map corresponding to the image to be detected; determining a first initial detection result, a second initial detection result, and a depth detection result corresponding to the image to be detected based on the initial feature map; wherein the first initial detection result is used to characterize whether each detection region of the image to be detected contains a sign, and the second initial detection result is used to characterize the center position of the sign in the image to be detected; determining a region to be confirmed in the image to be detected based on the first initial detection result and the second initial detection result; adjusting the initial feature map based on the target depth information corresponding to the region to be confirmed in the depth detection result, and determining a target detection result corresponding to the image to be detected based on the adjusted target feature map.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a sign recognition method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Traffic signs are an important part of road infrastructure, and their detection and recognition are also an important component of public digital assets for roads, as well as of advanced technologies such as autonomous driving and high-precision map production.

[0003] In related technologies, when recognizing traffic signs, the image to be detected is often input into a pre-trained convolutional neural network to obtain the sign recognition result output by the convolutional neural network. However, when the sign in the image to be detected is occluded, the use of the convolutional neural network may result in missed detections. Therefore, how to reduce the missed detections of traffic signs has become an urgent problem to be solved in this field. Summary of the Invention

[0004] This disclosure provides at least one sign recognition method, apparatus, computer device, and storage medium.

[0005] In a first aspect, embodiments of this disclosure provide a sign recognition method, including:

[0006] The image to be detected is acquired and input into a pre-trained target neural network to determine the initial feature map corresponding to the image to be detected.

[0007] Based on the initial feature map, a first initial detection result, a second initial detection result, and a depth detection result are determined for the image to be detected; wherein, the first initial detection result is used to characterize whether each detection region of the image to be detected contains a sign, the second initial detection result is used to characterize the center position of the sign in the image to be detected, and the depth detection result is used to characterize the depth information corresponding to each pixel in the image to be detected;

[0008] Based on the first initial detection result and the second initial detection result, the region to be confirmed in the image to be detected is determined;

[0009] Based on the target depth information corresponding to the region to be confirmed in the depth detection results, the initial feature map is adjusted, and the target detection result corresponding to the image to be detected is determined based on the adjusted target feature map.

[0010] In one possible implementation, determining the region to be confirmed in the image to be detected based on the first initial detection result and the second initial detection result includes:

[0011] Based on the first initial detection result, a first detection region corresponding to the sign in the image to be detected is determined; and based on the second initial detection result, a second detection region corresponding to the sign in the image to be detected is determined.

[0012] The area to be confirmed is determined based on the first detection area and the second detection area.

[0013] In one possible implementation, adjusting the initial feature map based on the target depth information corresponding to the region to be confirmed in the depth detection result includes:

[0014] Determine the first target feature point in the initial feature map that corresponds to the region to be confirmed;

[0015] Based on the target depth information, the values ​​of the target channels of the first target feature points are adjusted.

[0016] In one possible implementation, the second initial detection result is determined by the center point identification module of the target neural network;

[0017] The step of determining the target detection result corresponding to the image to be detected based on the adjusted target feature map includes:

[0018] The target feature map is input into the center point recognition module to determine the target detection result.

[0019] In one possible implementation, the second initial detection result includes a first feature map and a second feature map, wherein the value of each first feature point in the first feature map is used to characterize the probability that the detection area corresponding to the first feature point in the image to be detected contains the center point of each category of sign; the value of each second feature point in the second feature map is used to characterize the coordinate offset information of the target pixel point corresponding to each second feature point as the center point of each category of sign, and the size information of each category of sign.

[0020] In one possible implementation, determining the second detection region corresponding to the sign in the image to be detected based on the second initial detection result includes:

[0021] Based on the values ​​of each first feature point in the first feature map and a preset probability threshold, the second target feature point is determined;

[0022] The second detection region is determined based on the value of the third target feature point corresponding to the second target feature point in the second feature map and the position coordinates of the second target feature point in the first feature map.

[0023] In one possible implementation, the target neural network includes a feature extraction module, a region recognition module, a center point recognition module, and a depth recognition module. The feature extraction module is used to determine the initial feature map, the region recognition module is used to determine the first initial detection result, the center point recognition module is used to determine the second initial detection result, and the depth recognition module is used to determine the depth detection result.

[0024] The method further includes training the target neural network according to the following steps:

[0025] Obtain a sample image and a corresponding sample label, wherein the sample label is used to characterize the position and category of the sign in the sample image and the sample depth information of the sample image;

[0026] Based on the sample image and the sample label, the feature extraction module is trained simultaneously with the region recognition module, the center point recognition module and the depth recognition module in sequence. After the feature extraction module and any one of the modules have been trained, the trained feature extraction module is trained together with another module to be trained.

[0027] The trained feature extraction module, region recognition module, center point recognition module, and depth recognition module are fine-tuned to obtain the target neural network.

[0028] In one possible implementation, the step of simultaneously training the feature extraction module, the region recognition module, the center point recognition module, and the depth recognition module in sequence based on the sample image and the sample label includes:

[0029] Based on the detection boxes in the sample labels used to characterize the position of the sign in the sample image, the feature extraction module and the region recognition module are trained.

[0030] Based on the detection boxes and sign categories in the sample labels, the feature extraction module and the center point recognition module are trained.

[0031] The feature extraction module and the depth recognition module are trained based on the sample depth information in the sample labels.

[0032] Secondly, embodiments of this disclosure also provide a sign recognition device, comprising:

[0033] An acquisition module is used to acquire an image to be detected and input the image to be detected into a pre-trained target neural network to determine the initial feature map corresponding to the image to be detected;

[0034] The first determining module is used to determine a first initial detection result, a second initial detection result, and a depth detection result corresponding to the image to be detected based on the initial feature map; wherein, the first initial detection result is used to characterize whether each detection region of the image to be detected contains a sign, the second initial detection result is used to characterize the center position of the sign in the image to be detected, and the depth detection result is used to characterize the depth information corresponding to each pixel in the image to be detected;

[0035] The second determining module is used to determine the region to be confirmed in the image to be detected based on the first initial detection result and the second initial detection result;

[0036] The adjustment module is used to adjust the initial feature map based on the target depth information corresponding to the region to be confirmed in the depth detection result, and to determine the target detection result corresponding to the image to be detected based on the adjusted target feature map.

[0037] In one possible implementation, the second determining module, when determining the region to be confirmed in the image to be detected based on the first initial detection result and the second initial detection result, is configured to:

[0038] Based on the first initial detection result, a first detection region corresponding to the sign in the image to be detected is determined; and based on the second initial detection result, a second detection region corresponding to the sign in the image to be detected is determined.

[0039] The area to be confirmed is determined based on the first detection area and the second detection area.

[0040] In one possible implementation, when adjusting the initial feature map based on the target depth information corresponding to the region to be confirmed in the depth detection result, the adjustment module is used to:

[0041] Determine the first target feature point in the initial feature map that corresponds to the region to be confirmed;

[0042] Based on the target depth information, the values ​​of the target channels of the first target feature points are adjusted.

[0043] In one possible implementation, the second initial detection result is determined by the center point identification module of the target neural network;

[0044] The adjustment module, when determining the target detection result corresponding to the image to be detected based on the adjusted target feature map, is used for:

[0045] The target feature map is input into the center point recognition module to determine the target detection result.

[0046] In one possible implementation, the second initial detection result includes a first feature map and a second feature map, wherein the value of each first feature point in the first feature map is used to characterize the probability that the detection area corresponding to the first feature point in the image to be detected contains the center point of each category of sign; the value of each second feature point in the second feature map is used to characterize the coordinate offset information of the target pixel point corresponding to each second feature point as the center point of each category of sign, and the size information of each category of sign.

[0047] In one possible implementation, the second determining module, when determining the second detection region corresponding to the sign in the image to be detected based on the second initial detection result, is used to:

[0048] Based on the values ​​of each first feature point in the first feature map and a preset probability threshold, the second target feature point is determined;

[0049] The second detection region is determined based on the value of the third target feature point corresponding to the second target feature point in the second feature map and the position coordinates of the second target feature point in the first feature map.

[0050] In one possible implementation, the target neural network includes a feature extraction module, a region recognition module, a center point recognition module, and a depth recognition module. The feature extraction module is used to determine the initial feature map, the region recognition module is used to determine the first initial detection result, the center point recognition module is used to determine the second initial detection result, and the depth recognition module is used to determine the depth detection result.

[0051] The acquisition module is also used to train the target neural network according to the following steps:

[0052] Obtain a sample image and a corresponding sample label, wherein the sample label is used to characterize the position and category of the sign in the sample image and the sample depth information of the sample image;

[0053] Based on the sample image and the sample label, the feature extraction module is trained simultaneously with the region recognition module, the center point recognition module and the depth recognition module in sequence. After the feature extraction module and any one of the modules have been trained, the trained feature extraction module is trained together with another module to be trained.

[0054] The trained feature extraction module, region recognition module, center point recognition module, and depth recognition module are fine-tuned to obtain the target neural network.

[0055] In one possible implementation, the acquisition module, when training the feature extraction module, the region recognition module, the center point recognition module, and the depth recognition module simultaneously based on the sample image and the sample label, is used to:

[0056] Based on the detection boxes in the sample labels used to characterize the position of the sign in the sample image, the feature extraction module and the region recognition module are trained.

[0057] Based on the detection boxes and sign categories in the sample labels, the feature extraction module and the center point recognition module are trained.

[0058] The feature extraction module and the depth recognition module are trained based on the sample depth information in the sample labels.

[0059] Thirdly, embodiments of this disclosure also provide a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.

[0060] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect or any possible implementation of the first aspect.

[0061] The sign recognition method, apparatus, computer equipment, and storage medium provided in this disclosure can determine the areas in the image to be detected that may contain signs based on the first and second initial detection results of the image to be detected identified under different recognition dimensions. Then, based on the target depth information corresponding to the areas to be detected in the depth detection results, the initial feature map corresponding to the image to be detected is adjusted, and the target detection result corresponding to the image to be detected is determined based on the adjusted target feature map. In this way, the adjusted target feature map contains the depth information of the areas to be detected, which can improve the detection effect of the areas to be detected and reduce the missed detections caused by occlusion. On the other hand, determining the areas to be detected based on the first and second initial detection results, and then adjusting the initial feature map based on the target depth information of the areas to be detected, reduces the amount of computation and improves the detection efficiency compared to adjusting the initial feature map based on the depth information of the entire image.

[0062] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0064] Figure 1 A flowchart of a sign recognition method provided by an embodiment of this disclosure is shown;

[0065] Figure 2 This diagram illustrates the network structure of the target neural network in the sign recognition method provided in this embodiment of the present disclosure.

[0066] Figure 3 A schematic diagram of the architecture of a sign recognition device provided in an embodiment of this disclosure is shown;

[0067] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0069] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0070] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0071] Research has found that in the process of recognizing traffic signs, the image to be detected is often input into a pre-trained convolutional neural network to obtain the sign recognition result output by the convolutional neural network. However, when the sign in the image to be detected is occluded, the convolutional neural network may miss some signs. Therefore, how to reduce the missed detection of traffic signs has become an urgent problem to be solved in this field.

[0072] Based on the above research, this disclosure provides a sign recognition method, apparatus, computer device, and storage medium. It can determine the areas in the image to be identified that may contain signs based on first and second initial detection results of the image to be detected identified under different recognition dimensions. Then, based on the target depth information corresponding to the areas to be identified in the depth detection results, the initial feature map corresponding to the image to be detected is adjusted. Finally, the target detection result corresponding to the image to be detected is determined based on the adjusted target feature map. In this way, the adjusted target feature map contains the depth information of the areas to be identified, which can improve the detection effect of the areas to be identified and reduce missed detections due to occlusion. On the other hand, determining the areas to be identified based on the first and second initial detection results, and then adjusting the initial feature map based on the target depth information of the areas to be identified, reduces the computational load and improves the detection efficiency compared to adjusting the initial feature map based on the depth information of the entire image.

[0073] To facilitate understanding of this embodiment, a detailed description of the sign recognition method disclosed in this disclosure is provided first. The execution entity of the sign recognition method provided in this disclosure is generally a computer device with a certain computing capability. This computer device may include, for example, a terminal device, a server, or other processing equipment. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, the sign recognition method can be implemented by a processor calling computer-readable instructions stored in memory.

[0074] See Figure 1 The diagram shows a flowchart of a sign recognition method provided in this embodiment of the present disclosure. The method includes steps S101 to S104, wherein:

[0075] S101: Acquire the image to be detected and input the image to be detected into a pre-trained target neural network to determine the initial feature map corresponding to the image to be detected.

[0076] S102: Based on the initial feature map, determine the first initial detection result, the second initial detection result, and the depth detection result corresponding to the image to be detected; wherein, the first initial detection result is used to characterize whether each detection area of ​​the image to be detected contains a sign, the second initial detection result is used to characterize the center position of the sign in the image to be detected, and the depth detection result is used to characterize the depth information corresponding to each pixel in the image to be detected.

[0077] S103: Based on the first initial detection result and the second initial detection result, determine the region to be confirmed in the image to be detected.

[0078] S104: Based on the target depth information corresponding to the region to be confirmed in the depth detection result, adjust the initial feature map, and determine the target detection result corresponding to the image to be detected based on the adjusted target feature map.

[0079] The following is a detailed explanation of the steps described above.

[0080] For S101, the image to be detected may be acquired by an image acquisition device deployed on the driving device; the target neural network may include a feature extraction module, a region recognition module, a center point recognition module, and a depth recognition module. The feature extraction module is used to determine the initial feature map, the region recognition module is used to determine the first initial detection result, the center point recognition module is used to determine the second initial detection result, and the depth recognition module determines the depth detection result.

[0081] For example, the network structure diagram of the target neural network can be as follows: Figure 2 As shown, Figure 2 In this process, after the image to be detected is input into the target neural network, the feature extraction module of the target neural network performs feature extraction to obtain an initial feature map corresponding to the image to be detected; the region recognition module in the target neural network determines the first initial detection result based on the initial feature map; the center point recognition module in the target neural network determines the second initial detection result based on the initial feature map; and the depth recognition module in the target neural network determines the depth detection result based on the initial feature map.

[0082] The feature extraction module can be composed of a backbone network module and a high-dimensional feature extraction network (neck module). The backbone module can be, for example, a ResNet residual network, which can contain multiple network modules. Each network module can consist of convolutional layers, batch normalization layers, and activation layers. The output of the residual network can be formed by combining the outputs of each network module. The neck module can be, for example, a Feature Pyramid Network (FPN) or other network capable of high-dimensional feature extraction.

[0083] For example, taking a backbone containing network module 1, network module 2, network module 3, and network module 4 as an example, the output of network module 1 is the first output, the output of network module 2 is the second output, the output of network module 3 is the third output, and the output of network module 4 is the fourth output. The fourth output is deconvolved to obtain the fifth output, the fifth output is deconvolved to obtain the sixth output, the sixth output is deconvolved to obtain the seventh output, the fourth output is upsampled by eight times to obtain the eighth output, the third output and the fifth output are concatenated and superimposed, and then upsampled by four times to obtain the ninth output, the second output and the sixth output are concatenated and superimposed, and then upsampled by two times to obtain the tenth output. The first output, the seventh output, the eighth output, the ninth output, and the tenth output are concatenated and superimposed and then input into a convolutional layer with a kernel size of 1×1 to obtain a feature map with reduced channel count.

[0084] S102: Based on the initial feature map, determine the first initial detection result, the second initial detection result, and the depth detection result corresponding to the image to be detected; wherein, the first initial detection result is used to characterize whether each detection area of ​​the image to be detected contains a sign, the second initial detection result is used to characterize the center position of the sign in the image to be detected, and the depth detection result is used to characterize the depth information corresponding to each pixel in the image to be detected.

[0085] Here, the first initial detection result may include a third feature map, where the value of each feature point in the third feature map is used to characterize whether the corresponding detection area in the image to be detected contains a sign; the second initial detection result may include a first feature map and a second feature map, wherein the value of each first feature point in the first feature map is used to characterize the probability that the corresponding detection area in the image to be detected contains the center point of each category of sign; the value of each second feature point in the second feature map is used to characterize the coordinate offset information of the target pixel corresponding to each second feature point as the center point of each category of sign, and the size information of each category of sign; the depth information in the depth detection result represents the distance of the object at the corresponding position from the image acquisition device, for example, if the depth information corresponding to pixel A is 5, it can mean that the distance of the object corresponding to pixel A from the image acquisition device is 5m.

[0086] Specifically, the region recognition module in the target neural network can consist of three convolutional layers and two fully connected layers. The region recognition module can determine the first initial detection result based on the initial feature map. The center point recognition module in the target neural network can include a first branch and a second branch. The first branch can consist of three convolutional layers and can determine the first feature map based on the initial feature map. The second branch can consist of three convolutional layers and can determine the second feature map based on the initial feature map. The depth recognition module can consist of a decoder network of a Transformer model and multiple convolutional layers. The depth recognition module can determine a fourth feature map based on the initial feature map. The value corresponding to each feature point in the fourth feature map is used to represent the depth value from each pixel in the detection area corresponding to the feature point in the image to be detected to the optical center of the image acquisition device.

[0087] It should be noted that the first initial detection result, the second initial detection result, and the depth detection result corresponding to the image to be detected can be obtained simultaneously after parallel processing by the region recognition module, the center point recognition module, and the depth recognition module. That is, the region recognition module, the center point recognition module, and the depth recognition module can process the initial feature map at the same time; or, the region recognition module, the center point recognition module, and the depth recognition module can also process the initial feature map according to a preset processing order. The preset processing order may be, for example, using the region recognition module, the center point recognition module, and the depth recognition module to process the initial feature map in sequence, so as to reduce the real-time processing pressure of the terminal device.

[0088] S103: Based on the first initial detection result and the second initial detection result, determine the region to be confirmed in the image to be detected.

[0089] Here, the area to be confirmed refers to the area in the image to be detected where signs may exist.

[0090] Specifically, the first initial detection result and the second initial detection result are determined by different detection methods. The region identification module is used to determine whether there are pixels on the sign in each detection region. Therefore, when the sign is partially obscured but a part of the sign area is exposed, the first initial detection result determined by the region identification module may include these partially obscured signs. However, since only a part of the pixels are identified, these pixels may not necessarily belong to the sign. Therefore, there may be multiple detections in the first initial detection result. The center point identification module is used to determine whether there is a center point of the sign in each detection region. Therefore, when the sign is partially obscured (the center point is obscured), it is difficult to identify the center point. Therefore, there may be missed detections in the first initial detection result determined by the center point identification module. Therefore, the regions that may contain signs can be determined by combining the regions that may contain signs determined by different methods, and the regions to be confirmed can be reconfirmed to see if they contain signs, thereby reducing detection anomalies such as multiple detections and missed detections.

[0091] In one possible implementation, the region to be confirmed in the image to be detected can be determined through the following steps A1 to A2:

[0092] A1: Based on the first initial detection result, determine the first detection region corresponding to the sign in the image to be detected; and based on the second initial detection result, determine the second detection region corresponding to the sign in the image to be detected.

[0093] Here, the first detection area is used to represent the area in the image to be detected that may contain signs, as determined based on the first initial detection result; the second detection area represents the area in the image to be detected that may contain signs, as determined based on the second initial detection result.

[0094] Specifically, when determining the first detection region corresponding to the sign in the image to be detected based on the first initial detection result, the first detection region can be determined by representing the feature points containing the sign in the corresponding detection region according to the values ​​in the third feature map.

[0095] In one possible implementation, the determination of the second detection area can be achieved through the following steps A11 to A12:

[0096] A11: Based on the values ​​of each first feature point in the first feature map and a preset probability threshold, determine the second target feature point.

[0097] Here, the value of the first feature point can be contained in the region of multiple channels. The number of channels can be the number of sign categories. Each category of sign corresponds to a value that represents whether the detection region contains the center point of that category of sign.

[0098] Specifically, if the value of any first feature point in any channel is greater than the preset probability threshold, then the detection area corresponding to the first feature point includes the center point of the sign corresponding to that channel, and the first feature point can be determined as the second target feature point.

[0099] A12: Based on the value of the third target feature point corresponding to the second target feature point in the second feature map and the position coordinates of the second target feature point in the first feature map, the second detection area is determined.

[0100] Here, the position coordinates of the second target feature point in the first feature map can be used to characterize the center point of each category of sign in the image to be detected, and the value of the third target feature point can be used to characterize the coordinate offset information of the target pixel corresponding to each second feature point as the center point of each category of sign and the size information of each sign. Therefore, based on the value of the third target feature point corresponding to the second target feature point in the second feature map and the position coordinates of the second target feature point in the first feature map, the second detection area can be determined.

[0101] For example, the number of channels for feature points in the second feature map can be 4C, where C represents the number of sign categories. Each category of sign corresponds to four channel parameters, namely, horizontal coordinate offset information, vertical coordinate offset information, width of the sign category, and height of the sign category. Based on the coordinate offset information and vertical coordinate offset information in the value of the third target feature point, and the position coordinates of the second target feature point in the first feature map, the position coordinates of the center point of the sign can be determined. Then, combined with the width and height of the sign category in the value of the third target feature point, the area where the sign is located (i.e., the second detection area) can be determined.

[0102] A2: Determine the area to be confirmed based on the first detection area and the second detection area.

[0103] Here, when determining the area to be confirmed, the area in the first detection area that is different from the second detection area can be taken as the area to be confirmed.

[0104] For example, if the first detection area is area 1 and area 2, and the second detection area is area 1, then area 2 can be used as the area to be confirmed.

[0105] S104: Based on the target depth information corresponding to the region to be confirmed in the depth detection result, adjust the initial feature map, and determine the target detection result corresponding to the image to be detected based on the adjusted target feature map.

[0106] In one possible implementation, when adjusting the initial feature map based on the target depth information corresponding to the region to be confirmed in the depth detection results, the following steps B1 to B2 can be used:

[0107] B1: Determine the first target feature point in the initial feature map that corresponds to the region to be confirmed.

[0108] B2: Based on the target depth information, adjust the values ​​of the target channels of the first target feature points.

[0109] Here, the target channel can be a channel related to image depth features, and the value on the target channel can be a preset value.

[0110] Specifically, after adjusting the values ​​on the target channel of the first target feature point based on the target depth information, a target feature map containing depth information can be obtained. Since the values ​​on the target channel in the target feature map are no longer all the preset values, the target feature map can characterize the depth information corresponding to each pixel in the image to be detected to a certain extent. Compared with not using the depth information, adding depth information can improve the detection rate when the sign is obscured by an object in front of it.

[0111] Furthermore, when determining the target detection result corresponding to the image to be detected based on the adjusted target feature map, the target feature map can be input to the center point recognition module to determine the target detection result.

[0112] In this way, the target feature map has more channel values ​​than the initial feature map, which can distinguish the depth information corresponding to different pixels. Therefore, after inputting the target feature map into the center point recognition module, a more accurate target detection result can be obtained.

[0113] In one possible implementation, sample images for training the target neural network and corresponding sample labels for the sample images can be generated through the following steps C1 to C3:

[0114] C1: Acquire sample data corresponding to the sample signboard, simultaneously acquired by the image acquisition device and the lidar; wherein, the sample data includes a sample image containing the sample signboard acquired by the image acquisition device, and sample point cloud data acquired by the lidar.

[0115] Here, the sample signs in the sample images can be partially obscured, thereby improving the target neural network's ability to recognize obscured signs after training the target neural network using the sample images.

[0116] C2: Based on the target point cloud data corresponding to the sample sign in the sample point cloud data, and the relative positional relationship between the image acquisition device and the lidar, determine the sample depth information between the sample sign and the image acquisition device.

[0117] Here, the initial depth information between the sample sign and the lidar can be determined based on the target point cloud data. Based on the initial depth information and the relative positional relationship between the image acquisition device and the lidar, the sample depth information can be determined.

[0118] C3: Obtain sign labels to characterize the location and category of signs in the sample image, and use the depth information and sign labels as the sample labels corresponding to the sample image.

[0119] Here, the sign label may include a detection box marked by the developer to characterize the sign's location, as well as the category of the sample sign.

[0120] In one possible implementation, the target neural network can be trained using the following steps D1 to D3:

[0121] D1: Obtain the sample image and the corresponding sample label, wherein the sample label is used to characterize the position and category of the sign in the sample image and the sample depth information of the sample image.

[0122] D2: Based on the sample image and the sample label, the feature extraction module is trained simultaneously with the region recognition module, the center point recognition module and the depth recognition module in sequence. After the feature extraction module and any one of the modules have been trained, the trained feature extraction module is trained together with another module to be trained.

[0123] Here, during training, the feature extraction module and the region recognition module can be trained based on the detection boxes in the sample labels that represent the positions of signs in the sample images; the feature extraction module and the center point recognition module can be trained based on the detection boxes in the sample labels and the categories of signs; and the feature extraction module and the depth recognition module can be trained based on the sample depth information in the sample labels.

[0124] Specifically, when training the feature extraction module and the region recognition module, the cross-entropy loss function can be used to determine the first loss value during training, and the network parameters of the feature extraction module and the region recognition module can be adjusted using the first loss value, and the network parameters of the region recognition module can be fixed after adjustment.

[0125] When training the feature extraction module and the center point recognition module, a second loss value corresponding to the first branch can be determined based on the cross-entropy loss function, and the network parameters of the first branch of the center point recognition module and the feature extraction module can be adjusted based on the second loss value. Also, a third loss value corresponding to the second branch can be determined based on the cross-union loss function, and the network parameters of the second branch of the center point recognition module and the feature extraction module can be adjusted based on the third loss value. After adjustment, the network parameters of the center point recognition module are fixed.

[0126] When training the feature extraction module and the depth recognition module, a fourth loss value corresponding to the depth recognition module can be determined based on the sample depth information in the sample labels and the predicted depth information output by the depth recognition module, and the network parameters of the depth recognition module and the feature extraction module can be adjusted based on the fourth loss value.

[0127] It should be noted that when the feature extraction module, the region recognition module, the center point recognition module, and the depth recognition module are trained simultaneously in sequence, they can be trained in a preset training order. The training order can be, for example, region recognition module-center point recognition module-depth recognition module, depth recognition module-region recognition module-center point recognition module, center point recognition module-depth recognition module-region recognition module, etc. The specific training order and training parameters such as the learning rate during the training process can be set according to actual needs, and this embodiment does not limit this.

[0128] D3: Fine-tune the trained feature extraction module, region recognition module, center point recognition module, and depth recognition module to obtain the target neural network.

[0129] In this way, the network accuracy of the target neural network can be obtained through fine-tuning.

[0130] For example, taking the training order as depth recognition module - region recognition module - center point recognition module, the first stage of training can use sample images and corresponding sample labels to train the feature extraction module and the depth recognition module; the second stage of training can use sample images and corresponding sample labels to train the feature extraction module and the region recognition module; the third stage of training can use sample images and corresponding sample labels to train the feature extraction module and the center point recognition module; and the fourth stage of training can fine-tune the training of each network module of the target neural network to improve the network accuracy of the target neural network.

[0131] The sign recognition method provided in this disclosure can determine the areas in the image to be identified that may contain signs based on the first and second initial detection results of the image to be detected identified under different recognition dimensions. Then, based on the target depth information corresponding to the areas to be identified in the depth detection results, the initial feature map corresponding to the image to be detected is adjusted, and the target detection result corresponding to the image to be detected is determined based on the adjusted target feature map. In this way, the adjusted target feature map contains the depth information of the areas to be identified, which can improve the detection effect of the areas to be identified and reduce the missed detections caused by occlusion. On the other hand, determining the areas to be identified through the first and second initial detection results, and then adjusting the initial feature map based on the target depth information of the areas to be identified, reduces the amount of computation and improves the detection efficiency compared to adjusting the initial feature map based on the depth information of the entire image.

[0132] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0133] Based on the same inventive concept, this disclosure also provides a sign recognition device corresponding to the sign recognition method. Since the principle of the device in this disclosure is similar to that of the sign recognition method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0134] Reference Figure 3 The diagram shown is a schematic representation of the architecture of a sign recognition device according to an embodiment of this disclosure. The device includes: an acquisition module 301, a first determination module 302, a second determination module 303, and an adjustment module 304; wherein,

[0135] The acquisition module 301 is used to acquire the image to be detected and input the image to be detected into a pre-trained target neural network to determine the initial feature map corresponding to the image to be detected;

[0136] The first determining module 302 is used to determine a first initial detection result, a second initial detection result, and a depth detection result corresponding to the image to be detected based on the initial feature map; wherein, the first initial detection result is used to characterize whether each detection area of ​​the image to be detected contains a sign, the second initial detection result is used to characterize the center position of the sign in the image to be detected, and the depth detection result is used to characterize the depth information corresponding to each pixel in the image to be detected;

[0137] The second determining module 303 is used to determine the region to be confirmed in the image to be detected based on the first initial detection result and the second initial detection result;

[0138] The adjustment module 304 is used to adjust the initial feature map based on the target depth information corresponding to the region to be confirmed in the depth detection result, and to determine the target detection result corresponding to the image to be detected based on the adjusted target feature map.

[0139] In one possible implementation, the second determining module 303, when determining the region to be confirmed in the image to be detected based on the first initial detection result and the second initial detection result, is used to:

[0140] Based on the first initial detection result, a first detection region corresponding to the sign in the image to be detected is determined; and based on the second initial detection result, a second detection region corresponding to the sign in the image to be detected is determined.

[0141] The area to be confirmed is determined based on the first detection area and the second detection area.

[0142] In one possible implementation, when adjusting the initial feature map based on the target depth information corresponding to the region to be confirmed in the depth detection result, the adjustment module 304 is used to:

[0143] Determine the first target feature point in the initial feature map that corresponds to the region to be confirmed;

[0144] Based on the target depth information, the values ​​of the target channels of the first target feature points are adjusted.

[0145] In one possible implementation, the second initial detection result is determined by the center point identification module of the target neural network;

[0146] The adjustment module 304, when determining the target detection result corresponding to the image to be detected based on the adjusted target feature map, is used for:

[0147] The target feature map is input into the center point recognition module to determine the target detection result.

[0148] In one possible implementation, the second initial detection result includes a first feature map and a second feature map, wherein the value of each first feature point in the first feature map is used to characterize the probability that the detection area corresponding to the first feature point in the image to be detected contains the center point of each category of sign; the value of each second feature point in the second feature map is used to characterize the coordinate offset information of the target pixel point corresponding to each second feature point as the center point of each category of sign, and the size information of each category of sign.

[0149] In one possible implementation, the second determining module 303, when determining the second detection region corresponding to the sign in the image to be detected based on the second initial detection result, is used to:

[0150] Based on the values ​​of each first feature point in the first feature map and a preset probability threshold, the second target feature point is determined;

[0151] The second detection region is determined based on the value of the third target feature point corresponding to the second target feature point in the second feature map and the position coordinates of the second target feature point in the first feature map.

[0152] In one possible implementation, the target neural network includes a feature extraction module, a region recognition module, a center point recognition module, and a depth recognition module. The feature extraction module is used to determine the initial feature map, the region recognition module is used to determine the first initial detection result, the center point recognition module is used to determine the second initial detection result, and the depth recognition module is used to determine the depth detection result.

[0153] The acquisition module 301 is further configured to train the target neural network according to the following steps:

[0154] Obtain a sample image and a corresponding sample label, wherein the sample label is used to characterize the position and category of the sign in the sample image and the sample depth information of the sample image;

[0155] Based on the sample image and the sample label, the feature extraction module is trained simultaneously with the region recognition module, the center point recognition module and the depth recognition module in sequence. After the feature extraction module and any one of the modules have been trained, the trained feature extraction module is trained together with another module to be trained.

[0156] The trained feature extraction module, region recognition module, center point recognition module, and depth recognition module are fine-tuned to obtain the target neural network.

[0157] In one possible implementation, the acquisition module 301, when training the feature extraction module, the region recognition module, the center point recognition module, and the depth recognition module simultaneously based on the sample image and the sample label, is used to:

[0158] Based on the detection boxes in the sample labels used to characterize the position of the sign in the sample image, the feature extraction module and the region recognition module are trained.

[0159] Based on the detection boxes and sign categories in the sample labels, the feature extraction module and the center point recognition module are trained.

[0160] The feature extraction module and the depth recognition module are trained based on the sample depth information in the sample labels.

[0161] The sign recognition device provided in this disclosure can determine the region in the image to be detected that may contain a sign based on the first and second initial detection results of the image to be detected identified under different recognition dimensions. Then, based on the target depth information corresponding to the region to be detected in the depth detection results, the initial feature map corresponding to the image to be detected is adjusted, and the target detection result corresponding to the image to be detected is determined based on the adjusted target feature map. In this way, the adjusted target feature map contains the depth information of the region to be detected, which can improve the detection effect of the region to be detected and reduce the missed detection due to occlusion. On the other hand, determining the region to be detected based on the first and second initial detection results, and then adjusting the initial feature map based on the target depth information of the region to be detected, reduces the amount of computation and improves the detection efficiency compared to adjusting the initial feature map based on the depth information of the entire image.

[0162] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0163] Based on the same technical concept, this disclosure also provides a computer device. (See also...) Figure 4The diagram shows the structure of a computer device 400 provided in this embodiment, including a processor 401, a memory 402, and a bus 403. The memory 402 stores execution instructions and includes main memory 4021 and external memory 4022. The main memory 4021, also called internal memory, is used to temporarily store computational data in the processor 401 and data exchanged with external memory 4022 such as a hard disk. The processor 401 exchanges data with the external memory 4022 through the main memory 4021. When the computer device 400 is running, the processor 401 and the memory 402 communicate through the bus 403, causing the processor 401 to execute the following instructions:

[0164] The image to be detected is acquired and input into a pre-trained target neural network to determine the initial feature map corresponding to the image to be detected.

[0165] Based on the initial feature map, a first initial detection result, a second initial detection result, and a depth detection result are determined for the image to be detected; wherein, the first initial detection result is used to characterize whether each detection region of the image to be detected contains a sign, the second initial detection result is used to characterize the center position of the sign in the image to be detected, and the depth detection result is used to characterize the depth information corresponding to each pixel in the image to be detected;

[0166] Based on the first initial detection result and the second initial detection result, the region to be confirmed in the image to be detected is determined;

[0167] Based on the target depth information corresponding to the region to be confirmed in the depth detection results, the initial feature map is adjusted, and the target detection result corresponding to the image to be detected is determined based on the adjusted target feature map.

[0168] In one possible implementation, the instructions of the processor 401, wherein determining the region to be confirmed in the image to be detected based on the first initial detection result and the second initial detection result, includes:

[0169] Based on the first initial detection result, a first detection region corresponding to the sign in the image to be detected is determined; and based on the second initial detection result, a second detection region corresponding to the sign in the image to be detected is determined.

[0170] The area to be confirmed is determined based on the first detection area and the second detection area.

[0171] In one possible implementation, the instructions of the processor 401, wherein adjusting the initial feature map based on the target depth information corresponding to the region to be confirmed in the depth detection result, includes:

[0172] Determine the first target feature point in the initial feature map that corresponds to the region to be confirmed;

[0173] Based on the target depth information, the values ​​of the target channels of the first target feature points are adjusted.

[0174] In one possible implementation, in the instructions of the processor 401, the second initial detection result is determined by the center point recognition module of the target neural network;

[0175] The step of determining the target detection result corresponding to the image to be detected based on the adjusted target feature map includes:

[0176] The target feature map is input into the center point recognition module to determine the target detection result.

[0177] In one possible implementation, the instructions of the processor 401 include a first feature map and a second feature map, wherein the value of each first feature point in the first feature map is used to characterize the probability that the detection area corresponding to the first feature point in the image to be detected contains the center point of each category of sign; the value of each second feature point in the second feature map is used to characterize the coordinate offset information of the target pixel point corresponding to each second feature point as the center point of each category of sign, and the size information of each category of sign.

[0178] In one possible implementation, the instructions of the processor 401, wherein determining the second detection region corresponding to the sign in the image to be detected based on the second initial detection result, includes:

[0179] Based on the values ​​of each first feature point in the first feature map and a preset probability threshold, the second target feature point is determined;

[0180] The second detection region is determined based on the value of the third target feature point corresponding to the second target feature point in the second feature map and the position coordinates of the second target feature point in the first feature map.

[0181] In one possible implementation, in the instructions of the processor 401, the target neural network includes a feature extraction module, a region recognition module, a center point recognition module, and a depth recognition module. The feature extraction module is used to determine the initial feature map, the region recognition module is used to determine the first initial detection result, the center point recognition module is used to determine the second initial detection result, and the depth recognition module is used to determine the depth detection result.

[0182] It also includes training the target neural network according to the following steps:

[0183] Obtain a sample image and a corresponding sample label, wherein the sample label is used to characterize the position and category of the sign in the sample image and the sample depth information of the sample image;

[0184] Based on the sample image and the sample label, the feature extraction module is trained simultaneously with the region recognition module, the center point recognition module and the depth recognition module in sequence. After the feature extraction module and any one of the modules have been trained, the trained feature extraction module is trained together with another module to be trained.

[0185] The trained feature extraction module, region recognition module, center point recognition module, and depth recognition module are fine-tuned to obtain the target neural network.

[0186] In one possible implementation, the instructions of the processor 401, which include simultaneously training the feature extraction module, the region recognition module, the center point recognition module, and the depth recognition module based on the sample image and the sample label, include:

[0187] Based on the detection boxes in the sample labels used to characterize the position of the sign in the sample image, the feature extraction module and the region recognition module are trained.

[0188] Based on the detection boxes and sign categories in the sample labels, the feature extraction module and the center point recognition module are trained.

[0189] The feature extraction module and the depth recognition module are trained based on the sample depth information in the sample labels.

[0190] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the sign recognition method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0191] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the sign recognition method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0192] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0193] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0194] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0195] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0196] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0197] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A sign recognition method, characterized in that, include: The image to be detected is acquired and input into a pre-trained target neural network to determine the initial feature map corresponding to the image to be detected. Based on the initial feature map, a first initial detection result, a second initial detection result, and a depth detection result are determined for the image to be detected. The first initial detection result characterizes whether each detection region of the image to be detected contains a sign; the second initial detection result characterizes the center position of the sign in the image to be detected; and the depth detection result characterizes the depth information corresponding to each pixel in the image to be detected. The second initial detection result includes a first feature map and a second feature map. The value of each first feature point in the first feature map characterizes the probability that the detection region corresponding to the first feature point in the image to be detected contains the center point of each type of sign. The value of each second feature point in the second feature map characterizes the coordinate offset information of the target pixel corresponding to each second feature point as the center point of each type of sign, and the size information of each type of sign. Based on the first initial detection result and the second initial detection result, the region to be confirmed in the image to be detected is determined; Based on the target depth information corresponding to the region to be confirmed in the depth detection results, the initial feature map is adjusted, and the target detection result corresponding to the image to be detected is determined based on the adjusted target feature map.

2. The method according to claim 1, characterized in that, The step of determining the region to be confirmed in the image to be detected based on the first initial detection result and the second initial detection result includes: Based on the first initial detection result, a first detection region corresponding to the sign in the image to be detected is determined; and based on the second initial detection result, a second detection region corresponding to the sign in the image to be detected is determined. The area to be confirmed is determined based on the first detection area and the second detection area.

3. The method according to claim 1, characterized in that, The adjustment of the initial feature map based on the target depth information corresponding to the region to be confirmed in the depth detection results includes: Determine the first target feature point in the initial feature map that corresponds to the region to be confirmed; Based on the target depth information, the values ​​of the target channels of the first target feature points are adjusted.

4. The method according to claim 1, characterized in that, The second initial detection result is determined by the center point recognition module of the target neural network; The step of determining the target detection result corresponding to the image to be detected based on the adjusted target feature map includes: The target feature map is input into the center point recognition module to determine the target detection result.

5. The method according to claim 2, characterized in that, The step of determining the second detection region corresponding to the sign in the image to be detected based on the second initial detection result includes: Based on the values ​​of each first feature point in the first feature map and a preset probability threshold, the second target feature point is determined; The second detection region is determined based on the value of the third target feature point corresponding to the second target feature point in the second feature map and the position coordinates of the second target feature point in the first feature map.

6. The method according to claim 1, characterized in that, The target neural network includes a feature extraction module, a region recognition module, a center point recognition module, and a depth recognition module. The feature extraction module is used to determine the initial feature map, the region recognition module is used to determine the first initial detection result, the center point recognition module is used to determine the second initial detection result, and the depth recognition module is used to determine the depth detection result. The method further includes training the target neural network according to the following steps: Obtain a sample image and a corresponding sample label, wherein the sample label is used to characterize the position and category of the sign in the sample image and the sample depth information of the sample image; Based on the sample image and the sample label, the feature extraction module is trained simultaneously with the region recognition module, the center point recognition module and the depth recognition module in sequence. After the feature extraction module and any one of the modules have been trained, the trained feature extraction module is trained together with another module to be trained. The trained feature extraction module, region recognition module, center point recognition module, and depth recognition module are fine-tuned to obtain the target neural network.

7. The method according to claim 6, characterized in that, The step of simultaneously training the feature extraction module, the region recognition module, the center point recognition module, and the depth recognition module based on the sample image and the sample label includes: The feature extraction module and the region recognition module are trained based on the detection boxes in the sample labels that represent the positions of signs in the sample images; Based on the detection boxes and sign categories in the sample labels, the feature extraction module and the center point recognition module are trained. The feature extraction module and the depth recognition module are trained based on the sample depth information in the sample labels.

8. A sign recognition device, characterized in that, include: An acquisition module is used to acquire an image to be detected and input the image to be detected into a pre-trained target neural network to determine the initial feature map corresponding to the image to be detected; A first determining module is used to determine a first initial detection result, a second initial detection result, and a depth detection result corresponding to the image to be detected based on the initial feature map. The first initial detection result characterizes whether each detection region of the image to be detected contains a sign; the second initial detection result characterizes the center position of the sign in the image to be detected; and the depth detection result characterizes the depth information corresponding to each pixel in the image to be detected. The second initial detection result includes a first feature map and a second feature map. The value of each first feature point in the first feature map characterizes the probability that the detection region corresponding to the first feature point in the image to be detected contains the center point of each type of sign. The value of each second feature point in the second feature map characterizes the coordinate offset information of the target pixel corresponding to each second feature point as the center point of each type of sign, and the size information of each type of sign. The second determining module is used to determine the region to be confirmed in the image to be detected based on the first initial detection result and the second initial detection result; The adjustment module is used to adjust the initial feature map based on the target depth information corresponding to the region to be confirmed in the depth detection result, and to determine the target detection result corresponding to the image to be detected based on the adjusted target feature map.

9. A computer device, characterized in that, include: The system includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the sign recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the sign recognition method as described in any one of claims 1 to 7.

Images