Lane guide sign recognition method and device, server and storage medium

Abstract

The application discloses a lane guide sign identification method and device, a server and a storage medium. The lane guide sign identification method comprises the following steps: acquiring a state image of a lane guide sign; performing sign region detection processing on the state image according to a preset first network model to obtain a region image of the lane guide sign; and sequentially performing optical character recognition on characters in the region image according to a preset second network model to obtain arrow information of the lane guide sign. According to the embodiment of the application, each lane arrow does not need to be detected separately, the problem of missing detection of the lane arrow is avoided, and the recognition accuracy of the lane guide sign is improved to a certain extent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a lane guide sign recognition method and device, a server and a storage medium. BACKGROUND

[0002] The rapid development of automobiles has brought convenience to people's daily life, but also brought various problems, such as traffic congestion and frequent traffic accidents. Drivers can obtain real-time and accurate road information, which can effectively help drivers make danger predictions and improve driving safety. Therefore, traffic sign recognition has attracted widespread attention.

[0003] Among them, the lane guide sign is a form of traffic sign, and the lane guide sign generally contains multiple lane indication arrows. For the recognition of the lane guide sign, the existing recognition method is to realize it based on the detection and classification tasks in computer vision. That is, each lane arrow is first detected from the image of the lane guide sign, and then each arrow is classified respectively.

[0004] However, the existing recognition algorithm needs to detect each lane arrow separately, which is prone to missing detection. When one of the lane arrows is missed, it will lead to the identification of the entire lane guide sign sequence.

[0005] It can be seen that the existing recognition algorithm based on the detection and classification tasks in computer vision has low recognition accuracy for the lane guide sign. SUMMARY

[0006] The embodiments of the present application provide a lane guide sign recognition method, device, server and storage medium, which do not need to detect each lane arrow separately, avoid the problem of missing detection of lane arrows, and improve the recognition accuracy of the lane guide sign to a certain extent.

[0007] In a first aspect, the present application provides a lane guide sign recognition method, which comprises:

[0008] obtaining a state image of a lane guide sign;

[0009] performing sign region detection processing on the state image according to a preset first network model to obtain a region image of the lane guide sign;

[0010] sequentially performing optical character recognition on the characters of the region image according to a preset second network model to obtain arrow information of the lane guide sign.

[0011] In some embodiments of the present application, the second network model comprises a convolutional neural network and a recurrent neural network, and the arrow information of the lane guide sign is obtained by sequentially identifying the characters of the region image according to the preset second network model, comprising:

[0012] According to the convolutional neural network, the feature extraction processing is performed on the region image to obtain the feature sequence information of the region image.

[0013] According to the recurrent neural network and the feature sequence information, the optical character sequence recognition is performed to obtain the arrow information.

[0014] In some embodiments of the present application, before the arrow information of the lane guide sign is obtained by sequentially identifying the characters of the region image according to the preset second network model, the method further comprises:

[0015] Obtaining a first sample image, wherein the first sample image contains a first sample sign, and the first sample image is labeled with actual arrow sequence information of the first sample sign.

[0016] According to the preset recognition network model, the feature extraction processing is performed on the first sample image to obtain the first image feature of the first sample image.

[0017] According to the recognition network model and the first image feature, the identification processing is performed to obtain arrow sequence prediction information of the first sample sign.

[0018] According to the arrow sequence prediction information and the actual arrow sequence information, the model parameters of the recognition network model are updated until the recognition network model converges, and the recognition network model is taken as the second network model.

[0019] In some embodiments of the present application, the first sample image is obtained, comprising:

[0020] Obtaining an original image of the first sample sign.

[0021] Performing arrow extraction processing on the original image to obtain a plurality of sample arrow images of the first sample sign.

[0022] Randomly combining the plurality of sample arrow images to obtain an augmented image of the first sample sign.

[0023] Taking the original image and the augmented image as the first sample image, respectively.

[0024] In some embodiments of the present application, before the region image of the lane guide sign is obtained by performing the sign region detection processing on the state image according to the preset first network model, the method further comprises:

[0025] obtain a second sample image, wherein the second sample image contains a second sample sign, and the second sample image labels actual region information of the second sample sign;

[0026] perform feature extraction processing on the second sample image according to a preset detection network, to obtain a second image feature of the second sample image;

[0027] perform prediction processing according to the detection network and the second image feature, to obtain region prediction information of the second sample sign;

[0028] update model parameters of the detection network according to the region prediction information and the actual region information, and when the detection network converges, use the detection network as the first network model.

[0029] In some embodiments of the present application, the method further comprises:

[0030] obtain position information of the lane guide sign;

[0031] generate a road indication map according to the arrow information and the position information.

[0032] In some embodiments of the present application, the obtaining of the state image of the lane guide sign comprises:

[0033] obtain a video of the lane guide sign;

[0034] parse the video into multiple frames of images;

[0035] use each frame of image in the multiple frames of images as the state image.

[0036] In some embodiments of the present application, the first network model is a CenterNet network.

[0037] In another aspect, the present application provides a lane guide sign recognition device, comprising:

[0038] an obtaining unit, configured to obtain a state image of a lane guide sign;

[0039] a detection unit, configured to perform sign region detection processing on the state image according to a preset first network model, to obtain a region image of the lane guide sign;

[0040] a recognition unit, configured to sequentially perform optical character recognition on characters in the region image according to a preset second network model, to obtain arrow information of the lane guide sign.

[0041] In some embodiments of the present application, the second network model comprises a convolutional neural network and a recurrent neural network, and the identification unit is specifically configured to:

[0042] perform feature extraction processing on the region image according to the convolutional neural network to obtain feature sequence information of the region image;

[0043] perform optical character sequence identification according to the recurrent neural network and the feature sequence information to obtain the arrow information.

[0044] In some embodiments of the present application, the lane guide sign identification device further comprises a first training unit, and the first training unit is specifically configured to:

[0045] obtain a first sample image, wherein the first sample image contains a first sample sign, and the first sample image is labeled with actual arrow sequence information of the first sample sign;

[0046] perform feature extraction processing on the first sample image according to a preset identification network model to obtain first image features of the first sample image;

[0047] perform identification processing according to the identification network model and the first image features to obtain arrow sequence prediction information of the first sample sign;

[0048] update model parameters of the identification network model according to the arrow sequence prediction information and the actual arrow sequence information until the identification network model converges, and then use the identification network model as the second network model.

[0049] In some embodiments of the present application, the first training unit is specifically configured to:

[0050] obtain an original image of the first sample sign;

[0051] perform arrow extraction processing on the original image to obtain a plurality of sample arrow images of the first sample sign;

[0052] perform random combination processing on the plurality of sample arrow images to obtain an augmented image of the first sample sign;

[0053] use the original image and the augmented image as the first sample image, respectively.

[0054] In some embodiments of the present application, the lane guide sign recognition device further comprises a second training unit, before the step of detecting the region image of the lane guide sign according to the preset first network model, the second training unit is specifically used for:

[0055] obtaining a second sample image, wherein the second sample image contains a second sample sign, and the second sample image is labeled with actual region information of the second sample sign;

[0056] performing feature extraction processing on the second sample image according to a preset detection network, to obtain a second image feature of the second sample image;

[0057] performing prediction processing according to the detection network and the second image feature, to obtain region prediction information of the second sample sign;

[0058] updating model parameters of the detection network according to the region prediction information and the actual region information, until the detection network converges, and taking the detection network as the first network model.

[0059] In some embodiments of the present application, the lane guide sign recognition device further comprises a generation unit, and the generation unit is specifically used for:

[0060] obtaining position information of the lane guide sign;

[0061] generating a road indication map according to the arrow information and the position information.

[0062] In some embodiments of the present application, the obtaining unit is specifically used for:

[0063] obtaining a video of the lane guide sign;

[0064] parsing the video into multiple frames of images;

[0065] taking each frame of image in the multiple frames of images as the state image.

[0066] In some embodiments of the present application, the recognition unit is specifically used for:

[0067] performing optical character recognition on characters in the region image in sequence according to a CenterNet network, to obtain arrow information of the lane guide sign.

[0068] On the other hand, the present application also provides a server, comprising:

[0069] one or more processors;

[0070] a memory; and

[0071] one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the lane guide sign recognition method of any one of the first aspect.

[0072] In another aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is loaded by a processor to execute the steps of the lane guide sign recognition method of any one of the first aspect.

[0073] In the case that the recognition accuracy of the lane guide sign is low in the existing recognition algorithm based on the computer vision detection and classification task, the present application embodiment can output the arrow information of the lane guide sign in sequence by detecting and extracting the lane guide sign from the image first, and then performing optical character recognition on the characters of the lane guide sign. Since it is not necessary to detect each lane arrow separately, the problem of missing detection of arrows in the traditional computer vision detection and classification is avoided, so as to avoid errors in the arrow recognition sequence of the entire lane guide sign, and improve the recognition accuracy of the lane guide sign to a certain extent. BRIEF DESCRIPTION OF DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0075] Figure 1 is a scene schematic diagram of the lane guide sign recognition system provided by the present application embodiment;

[0076] Figure 2 is an embodiment flowchart of the lane guide sign recognition method provided by the present application embodiment;

[0077] Figure 3 is a schematic diagram of the lane guide sign in the present application embodiment;

[0078] Figure 4 is another schematic diagram of the lane guide sign in the present application embodiment;

[0079] Figure 5 is an embodiment flowchart of step 203 in the present application embodiment;

[0080] Figure 6 is an embodiment flowchart of the second network model training process in the present application embodiment;

[0081] Figure 7is an embodiment flow diagram of a first network model training process in the embodiments of the present application;

[0082] Figure 8 is an embodiment structure diagram of a lane guide sign recognition device provided in the embodiments of the present application;

[0083] Figure 9 is an embodiment structure diagram of a server provided in the embodiments of the present application. DETAILED DESCRIPTION

[0084] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0085] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly and specifically limited.

[0086] In the present application, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Any implementation described as "exemplary" in the present application is not necessarily to be construed as preferred or advantageous over other implementations. The following description is presented to enable any person skilled in the art to make and use the present application. In the following description, for the purposes of explanation, numerous details are set forth. It should be appreciated that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known structures and processes are not presented in order to avoid obscuring the description of the application. Thus, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features presented herein.

[0087] Firstly, some basic concepts involved in the embodiments of the present application are introduced as follows:

[0088] Computer vision (CV) Computer vision is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing to make computer processing more suitable for human eye observation or image transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, and attempts to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common face recognition, fingerprint recognition, and other biometric recognition technologies.

[0089] Target detection: target detection is an important processing task commonly used in computer vision technology. Target detection, also known as target extraction, is an image segmentation based on target geometry and statistical features, which combines target segmentation and recognition. Its accuracy and real-time performance are important capabilities of the entire system. In particular, in complex scenes, real-time processing of multiple targets is required, and automatic target extraction and recognition are particularly important. In the embodiments of the present application, the target specifically refers to a lane guide sign.

[0090] The embodiments of the present application provide a lane guide sign recognition method, device, server and storage medium, which are described in detail below.

[0091] Referring to Figure 1 , Figure 1 is a scene schematic diagram of a lane guide sign recognition system provided by the embodiments of the present application. The lane guide sign recognition system can include a server 100, and the server 100 integrates a lane guide sign recognition device. For example, the server can obtain a state image of a lane guide sign; perform sign region detection processing on the state image according to a preset first network model to obtain a region image of the lane guide sign; and sequentially perform optical character recognition on characters of the region image according to a preset second network model to obtain arrow information of the lane guide sign.

[0092] In the embodiments of the present application, the server 100 can be a stand-alone server, or a server network or server cluster composed of servers. For example, the server 100 described in the embodiments of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0093] In addition, as shown in Figure 1 The lane guide sign recognition system can further include a memory 200 for storing data, such as image data and video data.

[0094] It should be noted that Figure 1 The scene diagram of the lane guide sign recognition system shown is only an example. The lane guide sign recognition system and the scene described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, as the lane guide sign recognition system evolves and new business scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0095] First, the present application provides a lane guide sign recognition method. The execution subject of the lane guide sign recognition method is a lane guide sign recognition device. The lane guide sign recognition method includes: acquiring a state image of a lane guide sign; performing sign region detection processing on the state image according to a preset first network model to obtain a region image of the lane guide sign; and sequentially performing optical character recognition on characters in the region image according to a preset second network model to obtain arrow information of the lane guide sign.

[0096] As shown in Figure 2 Fig. 1 is a flowchart of an embodiment of the lane guide sign recognition method in the embodiments of the present application. The lane guide sign recognition method includes the following steps.

[0097] 201. Acquire a state image of a lane guide sign.

[0098] Traffic signs are road facilities that convey guidance, restriction, warning, or indication information through words or symbols. Also known as road signs or road traffic signs, traffic sign guide signs are a way of displaying traffic signs through signs. Lane guide signs are a type of traffic sign guide sign, and are mainly used for lane guidance.

[0099] As shown in Figure 3 Figure 3 ​Fig. 1 is a schematic diagram of a lane guide sign in an embodiment of the present application. The lane guide sign contains driving direction arrows of multiple lanes. The purpose of the embodiment of the present application is to identify each driving direction arrow in the lane guide sign in turn. In this document, the driving direction arrow of a lane is also referred to as a "lane arrow" or "arrow" for short. The state image refers to an image containing the lane guide sign.

[0100] Specifically, in actual applications, the lane guide sign identification device provided in the embodiment of the present application can directly include a camera (which is mainly used to collect images of lane guide signs) in hardware, and store the images captured by the camera locally, which can be directly read inside the lane guide sign identification device. Alternatively, the lane guide sign can also be connected to the camera through a network connection, and the images obtained by the camera can be acquired online from the camera according to the network connection. Alternatively, the lane guide sign can also read the images captured by the camera from a related storage medium that stores the images captured by the camera, and the specific acquisition method is not limited here.

[0101] The camera can capture images according to a preset shooting mode, for example, the shooting height, shooting direction or shooting distance can be set, and the specific shooting mode can be adjusted according to the camera itself, which is not limited here. The multiple images captured by the camera can form a video through a time line.

[0102] In a specific embodiment, the state image of the lane guide sign can be obtained by capturing the image of the lane guide sign through the camera on the vehicle driving on the road.

[0103] In a specific implementation, the camera on the vehicle driving on the road captures a road video. At this time, the captured video can be parsed into individual frames of images to detect the arrow information of multiple lane guide signs on a section of road respectively. At this time, the state image of the lane guide sign obtained in step 201 can specifically include: obtaining a video of the lane guide sign; parsing the video into multiple images; and taking each image in the multiple images as the state image. By collecting the video, the efficiency of obtaining the state image of the lane guide sign can be improved, thereby improving the identification speed of the traffic signs on the road to a certain extent. The driver can obtain the road information in real time and accurately, which can effectively help the driver to make a danger prediction and improve the safety of driving.

[0104] 202. Perform sign region detection processing on the state image according to a preset first network model to obtain a region image of the lane guide sign.

[0105] Among them, the area image refers to the image obtained by cropping the area where the lane guidance sign is located from the status image.

[0106] The first network model can be trained on a training dataset using an open-source network model suitable for detection tasks, such as the YOLO network or the SSD network. The training dataset consists of sample images of the marked lane sign locations. The training process of the first network model will be detailed later and will not be repeated here.

[0107] Due to limitations in shooting conditions, the position of lane guidance signs in the status image may not be unique, and the proportion of lane guidance signs in the status image may be relatively small. In order to ensure that the arrow information on the lane guidance signs can be correctly identified, the lane guidance signs are first detected by a preset first network model, so as to ensure that the character optical recognition can be accurately performed on the regional image to obtain the arrow information of the lane guidance signs.

[0108] like Figure 4 As shown, in one specific embodiment, images of lane guidance signs are captured by cameras on vehicles traveling on the road. However, because the lane guidance signs occupy a small portion of the image, normal optical character recognition is not possible. In this case, a detection task in computer vision can be used to first detect the lane guidance signs and then extract them.

[0109] 203. Based on the preset second network model, perform optical character recognition on the characters in the region image in sequence to obtain the arrow information of the lane guidance sign.

[0110] The arrow information refers to the specific direction of travel indicated by the arrows on the lane guidance signs.

[0111] The second network model can be trained on a training dataset using an open-source network model suitable for optical character recognition, such as an Attention OCR network. The training dataset consists of sample images labeled with the arrow sequence information of lane guidance signs. The training process of the second network model will be described in detail later and will not be repeated here.

[0112] The arrow sequence information refers to the specific direction of travel indicated by the arrows on the lane guidance signs, arranged in a certain order (such as from left to right or from right to left). For example, the first sample sign is as follows: Figure 4 As shown, the arrows from right to left correspond to "Right Turn", "Straight", "Straight", "Straight", and "Left Turn U-turn". The arrow sequence information is "Right Turn", "Straight", "Straight", "Straight", and "Left Turn U-turn".

[0113] In the case that the recognition accuracy of the existing recognition algorithm based on the detection and classification task in computer vision is low, the lane guide sign is detected and extracted from the image, and then the characters of the lane guide sign are sequentially subjected to optical character recognition, so that the arrow information of the lane guide sign can be sequentially output. Since each lane arrow does not need to be detected separately, the problem of missing detection of arrows in traditional computer vision detection and classification is avoided, so that the error of the arrow recognition sequence of the entire lane guide sign is avoided, and the recognition accuracy of the lane guide sign is improved to a certain extent.

[0114] In some embodiments of the present application, the second network model can include a convolutional neural network and a recurrent neural network, as shown in Figure 5 As shown, the step 203 of sequentially performing optical character recognition on the characters of the region image according to the preset second network model to obtain the arrow information of the lane guide sign can further include:

[0115] 501. performing feature extraction processing on the region image according to the convolutional neural network to obtain feature sequence information of the region image.

[0116] The target image feature refers to the image space feature obtained after the feature extraction processing of the region image. The feature sequence information refers to the target image feature sorted according to a certain positional relationship.

[0117] In one specific embodiment, the second network model can include a convolutional neural network (CNN) and a recurrent neural network (RNN). The recurrent neural network can be a long short-term memory (LSTM), a deep recurrent neural network (deep RNN), etc.

[0118] The convolutional neural network (CNN) is used to output the target image features of the region image, providing feature sequence information for the subsequent recurrent neural network (RNN) to ensure the correct prediction of the arrow information of the lane guidance sign. The CNN takes the region image as input, and the region image undergoes convolution and pooling operations to extract features, outputting the feature sequence information of the region image. Specifically, the CNN can include convolutional layers and pooling layers. The region image undergoes convolution operations after passing through the convolutional layers and pooling operations after passing through the pooling layers. The kernel size, stride, and padding of the convolutional layers can be set according to the feature map size required for the specific application scenario. For example, after setting the kernel size, stride, and padding of the region image input to a convolutional layer of 5×5, 1, and no padding respectively, a 12×12 spatial feature map will be output. The 12×12 spatial feature map undergoes feature selection and information filtering through the pooling layer, finally outputting the feature map of the region image, i.e., the feature sequence information.

[0119] 502. Optical character serialization recognition is performed based on the recurrent neural network and the feature sequence information to obtain the arrow information.

[0120] The recurrent neural network (RNN) is used to output the arrow information of the lane guidance signs. The RNN takes feature sequence information as input and recursively processes the evolution of the feature sequence information to perform optical character recognition on the characters in the region image, ultimately outputting the arrow information of the lane guidance signs sequentially.

[0121] Specifically, the feature sequence information is sequentially input into the recurrent neural network, which then recursively calls the recurrent neural network in the direction of the feature sequence information evolution to realize optical character serialization recognition of characters in the regional image and output the arrow information of the lane guidance sign.

[0122] By leveraging the contextual understanding capabilities of recurrent neural networks (RNNs), feature sequence information extracted from regional images via convolutional neural networks (CNNs) is used as input to the RNN for sequential recognition of lane guidance signs. On one hand, the RNN can combine information from the left and right sequences of the lane guidance signs to predict arrow information, effectively improving the accuracy of arrow information recognition. On the other hand, since there is no need to annotate detection boxes for individual arrows, the RNN can sequentially recognize each arrow on the lane guidance sign, avoiding the need to use a detector to detect each arrow individually and reducing redundancy in the data processing flow.

[0123] like Figure 6 As shown, in some embodiments of this application, the second network model can be trained through the following steps:

[0124] 601. Obtain the first sample image.

[0125] The first sample image contains a first sample sign. The first sample image labels actual arrow sequence information of the first sample sign. The first sample sign refers to a lane guide sign used as a sample. The actual arrow sequence information refers to arrow sequence information of the first sample sign labeled. For example, as shown in FIG. 16, the arrows "turn left U-turn", "go straight", "go straight", "go straight", and "turn right" from left to right are represented by symbols "i16", "i13", "i13", "i13", and "i10", respectively. The actual arrow sequence information of the first sample sign is "i16", "i13", "i13", "i13", and "i10". Figure 4

[0126] The specific acquisition process of the first sample image can refer to the description and examples of the state image acquisition, which will not be repeated here.

[0127] Further, the first sample image can be an image after cropping and containing only the first sample sign.

[0128] 602. Perform feature extraction processing on the second sample image according to the preset recognition network model to obtain first image features of the first sample image.

[0129] The first image features refer to image spatial features obtained after performing feature extraction processing on the first sample image. The first image features are serialized spatial feature information.

[0130] Correspondingly, the network structure of the preset recognition network model is the same as the network structure of the second network model mentioned in step 501. The specific network structure and function of the preset recognition network model can refer to the description in steps 501-502 above, which will not be repeated here. The difference is that the preset recognition network model is before training, and the second network model is after training.

[0131] Specifically, the first sample image is input into the recognition network model, and the convolutional neural network in the recognition network model performs convolution and pooling operations on the first image features to extract the serialized spatial feature information of the first sample image, i.e., to obtain the first image features.

[0132] 603. Perform recognition processing according to the recognition network model and the first image features to obtain arrow sequence prediction information of the first sample sign.

[0133] The arrow sequence prediction information refers to the predicted arrow sequence information of the first sample sign.

[0134] ​Specifically, the first image features output by the convolutional neural network are input into a recurrent neural network in the recognition network model as sequence data of the recurrent neural network. Then, the recurrent neural network recursively evolves in the direction of the first image features to realize optical character recognition of the characters in the first sample image, and finally sequentially outputs arrow sequence prediction information of the first sample sign.

[0135] 604、According to the arrow sequence prediction information and the actual arrow sequence information, the model parameters of the recognition network model are updated until the recognition network model converges, and the recognition network model is taken as the second network model.

[0136] First, according to the arrow sequence prediction information, the actual arrow sequence information, and a preset loss function (such as a cross entropy loss function Cross Entropy Loss), a training loss value of the recognition network model can be determined. Then, according to the back propagation of the training loss value of the recognition network model, the model parameters of the recognition network model are updated to make the recognition network model have better performance in the recognition of arrow information.

[0137] Until the recognition network model converges (for example, the number of training iterations of the recognition network model reaches a preset number, or the objective function of model training is optimal), the recognition network model is taken as the second network model. At this time, the second network model can be used for arrow information recognition of lane guide signs.

[0138] In one specific embodiment, the second network model is trained by an Attention OCR network. When actually recognizing a lane guide sign, after the first network model detects a regional image of the lane guide sign, the trained Attention OCR network extracts feature sequence information of the regional image and recursively recognizes the characters in the regional image based on the feature sequence information, and finally sequentially outputs arrow information in the lane guide sign.

[0139] The recurrent convolutional network is very useful for scenarios where the number of character categories to be recognized is small. Since the recurrent convolutional network does not need to detect each arrow separately, it can directly input an image for arrow recognition. Therefore, the annotation of text boxes is not required in the training process, making the collection of training data very convenient. End-to-end training can be performed, and the training process is simpler and more general. At the same time, it does not require that the recognized characters must be arranged in a single row, and double-row or multi-row arrangements can also be recognized.

[0140] Since optical character recognition (OCR) requires a large amount of training data to prevent model overfitting. Therefore, in some embodiments of the present application, the training data is also augmented. For example, the first sample image can be rotated, randomly cropped, randomly adjusted in contrast, brightness, chroma and saturation. Further, the augmentation of the training data can be achieved by artificially synthesizing lane guide signs from various arrows collected.

[0141] Further, at this time, the obtaining of the first sample image in step 601 can specifically include: obtaining an original image of the first sample sign; performing arrow extraction processing on the original image to obtain a plurality of sample arrow images of the first sample sign; performing random combination processing on the plurality of sample arrow images to obtain an augmented image of the first sample sign; and taking the original image and the augmented image as the first sample image respectively.

[0142] Wherein, the original image refers to an image of the first sample sign obtained by a camera. The sample arrow image refers to an image of an arrow region extracted from the original image. The augmented image refers to an augmented image of the first sample sign.

[0143] In a specific embodiment, each arrow in the original image is manually extracted, and a certain number of arrows are randomly combined into a new first sample sign, thereby obtaining an augmented image of the first sample sign. Wherein, each original image and each augmented image can be used as a first sample image for training a preset recognition network model.

[0144] As shown in FIG. 7, in some embodiments of the present application, the first network model can be trained by the following steps: Figure 7

[0145] 701, obtaining a second sample image.

[0146] Wherein, the second sample image contains a second sample sign, and the second sample image is labeled with actual region information of the second sample sign.

[0147] The second sample sign refers to a lane guide sign used as a sample. Wherein, the second sample sign and the first sample sign can be the same lane guide sign. Correspondingly, the first sample image can be obtained by cropping the second sample image.

[0148] The actual region information refers to the bounding box of the second sample sign obtained by labeling.

[0149] 702, performing feature extraction processing on the second sample image according to a preset detection network to obtain a second image feature of the second sample image.

[0150] ​The second image feature refers to a spatial feature of the second sample image obtained after feature extraction processing.

[0151] Specifically, the second sample image is input into a preset detection network, and the detection network performs convolution and pooling operations on the first image feature to extract spatial feature information of the second sample image, i.e., to obtain the second image feature.

[0152] 703. The second sample label region prediction information is obtained by performing prediction processing on the detection network and the second image feature.

[0153] The region prediction information refers to a detection box of the second sample label predicted by the detection network.

[0154] Subsequently, the detection network performs classification regression prediction processing according to the second image feature, and finally outputs the region prediction information of the second sample label.

[0155] 704. The model parameters of the detection network are updated according to the region prediction information and the actual region information, and when the detection network converges, the detection network is used as the first network model.

[0156] First, according to the region prediction information, the actual region information, and a preset loss function (such as a cross-entropy loss function Focal Loss), the training loss value of the detection network can be determined. Then, according to the training loss value of the detection network, the model parameters of the detection network are updated to make the detection network have better performance in detecting lane guide labels.

[0157] When the detection network converges (for example, when the number of training iterations of the detection network reaches a preset number, or when the objective function of model training is optimal), the detection network is used as the first network model. At this time, the first network model can be used to detect lane guide labels.

[0158] Further, the first network model in the embodiment of the present application is trained by an open-source CenterNet network. The CenterNet network, i.e., a center point-based detection network, is an end-to-end unique, simple and fast target detector.

[0159] If the predicted bounding box has a high IoU (Intersection over Union) with the actual bounding box, the probability that the center key point predicts the same class is high, and vice versa. The CenterNet network treats each target as a three-key point instead of a pair of key points by detecting each target. The CenterNet network explores the information of the middle region (approximate geometric center) of the proposal by adding a key point, which improves the accuracy and recall rate.

[0160] In some cases, there can be multiple lane guide signs in the state image, and there can be overlapping. In view of this, in some embodiments of the present application, a first network model is trained by an open source CenterNet network to improve the detection accuracy of the region image of the lane guide sign. Through the CenterNet network, when a bounding box is generated by a set of key points during inference, it is observed whether a key point of the same class falls into the center of the region, that is, three points are used to represent the target; that is, overlapping lane guide signs can be detected, and each lane guide sign can be positioned more accurately.

[0161] Using CenterNet as a detector for lane guide sign positioning, due to the anchor-free nature of CenterNet, it densely predicts target positions, which has obvious advantages for small target detection in lane guide signs.

[0162] In order to enable the driver to obtain road information in real time and accurately, the driver can be effectively helped to make danger prediction, and the safety of driving can be improved. The lane guide sign recognition method of the embodiments of the present application can also output the arrow information of each lane in time when the vehicle reaches the arrow corresponding mark section of the lane guide sign, so as to remind the driver to drive according to the corresponding lane guide.

[0163] In some embodiments of the present application, the position information of the lane guide sign is obtained at the same time as the state image is obtained in step 201; and the position information of the lane guide sign and the arrow information are stored in association. When the vehicle travels to the corresponding section, the arrow information of the lane guide sign of the section is output. The way of outputting the arrow information of the lane guide sign of the section is various, for example, outputting in the form of voice, outputting in the form of a road map, and the like.

[0164] When it is required to be output in the form of a road map, after the arrow information of the lane guide sign is determined, a road indication map also needs to be generated. That is, in some embodiments, the lane guide sign recognition method provided by the embodiments of the present application further includes: acquiring position information of the lane guide sign; and generating a road indication map according to the arrow information and the position information. The road indication map refers to a map that integrates road, lane guide sign and other indication information.

[0165] In order to better implement the lane guide sign recognition method in the embodiments of the present application, on the basis of the lane guide sign recognition method, the embodiments of the present application further provide a lane guide sign recognition device, as shown in Figure 8 The lane guide sign recognition device 800 includes:

[0166] An acquisition unit 801 is configured to acquire a state image of a lane guide sign.

[0167] A detection unit 802 is configured to perform sign region detection processing on the state image according to a preset first network model, to obtain a region image of the lane guide sign.

[0168] An identification unit 803 is configured to sequentially perform optical character recognition on characters of the region image according to a preset second network model, to obtain arrow information of the lane guide sign.

[0169] In some embodiments of the present application, the second network model includes a convolutional neural network and a recurrent neural network, and the identification unit 803 is specifically configured to:

[0170] perform feature extraction processing on the region image according to the convolutional neural network, to obtain feature sequence information of the region image;

[0171] perform optical character sequence recognition according to the recurrent neural network and the feature sequence information, to obtain the arrow information.

[0172] In some embodiments of the present application, the lane guide sign recognition device 800 further includes a first training unit (not shown in the figure), and before the step of sequentially recognizing characters of the region image according to the preset second network model to obtain the arrow information of the lane guide sign, the first training unit is specifically configured to:

[0173] acquire a first sample image, wherein the first sample image contains a first sample sign, and the first sample image is labeled with actual arrow sequence information of the first sample sign;

[0174] perform feature extraction processing on the first sample image according to a preset recognition network model, to obtain first image features of the first sample image;

[0175] performing identification processing according to the identification network model and the first image feature, to obtain arrow sequence prediction information of the first sample sign;

[0176] updating model parameters of the identification network model according to the arrow sequence prediction information and the actual arrow sequence information, and taking the identification network model as the second network model when the identification network model converges.

[0177] In some embodiments of the present application, the first training unit is specifically configured to:

[0178] obtain an original image of the first sample sign;

[0179] perform arrow extraction processing on the original image, to obtain a plurality of sample arrow images of the first sample sign;

[0180] perform random combination processing on the plurality of sample arrow images, to obtain an augmented image of the first sample sign;

[0181] take the original image and the augmented image as the first sample image, respectively.

[0182] In some embodiments of the present application, the lane guide sign recognition apparatus 800 further comprises a second training unit (not shown in the figure), and before the step of performing sign region detection processing on the state image according to the preset first network model to obtain a region image of the lane guide sign, the second training unit is specifically configured to:

[0183] obtain a second sample image, wherein the second sample image contains a second sample sign, and the second sample image is labeled with actual region information of the second sample sign;

[0184] perform feature extraction processing on the second sample image according to a preset detection network, to obtain second image features of the second sample image;

[0185] perform prediction processing according to the detection network and the second image features, to obtain region prediction information of the second sample sign;

[0186] update model parameters of the detection network according to the region prediction information and the actual region information, and take the detection network as the first network model when the detection network converges.

[0187] In some embodiments of the present application, the lane guide sign recognition apparatus 800 further comprises a generation unit (not shown in the figure), and the generation unit is specifically configured to:

[0188] obtain position information of the lane guide sign.

[0189] A road sign map is generated based on the arrow information and the location information.

[0190] In some embodiments of this application, the acquisition unit 801 is specifically used for:

[0191] Obtain the video of the lane guidance sign;

[0192] The video is parsed into multiple frames;

[0193] Each frame of the multi-frame image is used as the state image.

[0194] In some embodiments of this application, the identification unit 803 is specifically used for:

[0195] The arrow information of the lane guidance sign is obtained by sequentially performing optical character recognition on the characters in the region image using the CenterNet network.

[0196] This application embodiment first detects and extracts lane guidance signs from images, and then performs optical character recognition on the characters of the lane guidance signs in sequence, so as to output the arrow information of the lane guidance signs in a serialized manner. Since it is not necessary to detect each lane arrow separately, it avoids the problem of missed arrows that are easy to occur in traditional computer vision detection and classification, thereby avoiding errors in the arrow recognition sequence of the entire lane guidance sign and improving the recognition accuracy of lane guidance signs to a certain extent.

[0197] This application embodiment also provides a server that integrates any of the lane guidance sign recognition devices provided in this application embodiment, the server comprising:

[0198] One or more processors;

[0199] Memory; and

[0200] One or more applications, wherein the one or more applications are stored in the memory and configured by the processor to perform the steps of the lane guidance sign recognition method described in any of the embodiments of the lane guidance sign recognition method described above.

[0201] This application also provides a server that integrates any of the lane guidance sign recognition devices provided in this application. For example... Figure 9 As shown, it illustrates a schematic diagram of the server structure involved in an embodiment of this application. Specifically:

[0202] The server can include a processor 901 having one or more processing cores, a memory 902 having one or more computer-readable storage media, a power supply 903, an input unit 904, and the like. Those skilled in the art can understand that Figure 9 The server structure shown in the figure is not a limitation on the server, and can include more or fewer components than shown, or combine certain components, or different component arrangements. Among them:

[0203] The processor 901 is the control center of the server, which connects various parts of the server through various interfaces and lines, performs various functions of the server and processes data by running or executing software programs and / or modules stored in the memory 902, and calling data stored in the memory 902, thereby overall monitoring the server. Optionally, the processor 901 can include one or more processing cores; the processor 901 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like, preferably, the processor 901 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 901.

[0204] The memory 902 can be used to store software programs and modules, and the processor 901 executes various functions and data processing by running the software programs and modules stored in the memory 902. The memory 902 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; the data storage area can store data created according to the use of the server, etc. In addition, the memory 902 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 902 can also include a memory controller to provide the processor 901 with access to the memory 902.

[0205] The server further includes a power supply 903 for supplying power to the various components. Preferably, the power supply 903 can be logically connected to the processor 901 through a power management system, so that the power management system can be used to manage charging, discharging, power consumption management, and the like. The power supply 903 can also include one or more DC or AC power sources, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

[0206] The server can also include an input unit 904 for receiving input digital or character information, and generating keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0207] Although not shown, the server can also include a display unit and the like, which will not be described here. In particular, in the present embodiment, the processor 901 in the server will load one or more executable files corresponding to the processes of one or more application programs into the memory 902 according to the following instructions, and run the application programs stored in the memory 902 by the processor 901, thereby implementing various functions, such as the following:

[0208] obtaining a state image of a lane guide sign;

[0209] performing sign region detection processing on the state image according to a preset first network model, to obtain a region image of the lane guide sign;

[0210] sequentially performing optical character recognition on the characters of the region image according to a preset second network model, to obtain arrow information of the lane guide sign.

[0211] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling relevant hardware by instructions, which can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0212] To this end, an embodiment of the present application provides a computer-readable storage medium, which can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like. A computer program is stored on the storage medium, and the computer program is loaded by a processor to execute the steps in any lane guide sign recognition method provided by an embodiment of the present application. For example, the computer program loaded by the processor can execute the following steps:

[0213] obtaining a state image of a lane guide sign;

[0214] According to a preset first network model, the state image is subjected to label region detection processing, and a region image of the lane guide label is obtained;

[0215] According to a preset second network model, characters in the region image are sequentially subjected to optical character recognition, and arrow information of the lane guide label is obtained.

[0216] In the above embodiments, the description of each embodiment has its own focus. The parts not described in detail in a certain embodiment can be referred to the detailed description of other embodiments above, and will not be described here again.

[0217] In the implementation, the above various units or structures can be implemented as independent entities, or can be combined as the same or several entities. The specific implementation of the above various units or structures can be referred to the method embodiments above, and will not be described here again.

[0218] The specific implementation of the above various operations can be referred to the embodiments above, and will not be described here again.

[0219] The lane guide label recognition method, device, server and storage medium provided by the embodiments of the present application are described in detail above. The principle and implementation mode of the present application are described by applying specific examples. The above embodiment description is only used to help understand the method and its core idea of the present application. Meanwhile, according to the idea of the present application, the specific implementation mode and application range will be changed by the person skilled in the art. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for recognizing lane guidance signs, characterized in that, The lane guide sign recognition method comprises: acquiring a state image of a lane guide sign; performing sign region detection processing on the state image according to a preset first network model to obtain a region image of the lane guide sign; sequentially performing optical character recognition on characters of the region image according to a preset second network model to obtain arrow information of the lane guide sign; wherein, before the sequentially performing optical character recognition on characters of the region image according to a preset second network model to obtain arrow information of the lane guide sign, the method further comprises: acquiring a first sample image, wherein the first sample image contains a first sample sign, and the first sample image is labeled with actual arrow sequence information of the first sample sign; performing feature extraction processing on the first sample image according to a preset recognition network model to obtain first image features of the first sample image; performing recognition processing according to the recognition network model and the first image features to obtain arrow sequence prediction information of the first sample sign; updating model parameters of the recognition network model according to the arrow sequence prediction information and the actual arrow sequence information until the recognition network model converges, and taking the recognition network model as the second network model.

2. The method of claim 1, wherein, The second network model comprises a convolutional neural network and a recurrent neural network, and the sequentially performing optical character recognition on characters of the region image according to a preset second network model to obtain arrow information of the lane guide sign comprises: performing feature extraction processing on the region image according to the convolutional neural network to obtain feature sequence information of the region image; performing optical character sequence recognition according to the recurrent neural network and the feature sequence information to obtain the arrow information.

3. The method of claim 1, wherein, The acquiring of the first sample image comprises: acquiring an original image of the first sample sign; performing arrow extraction processing on the original image to obtain a plurality of sample arrow images of the first sample sign; performing random combination processing on the plurality of sample arrow images to obtain an augmented image of the first sample sign; taking the original image and the augmented image as the first sample image respectively.

4. The method of claim 1, wherein, Before the performing of the sign region detection processing on the state image according to the preset first network model to obtain the region image of the lane guide sign, the method further comprises: acquiring a second sample image, wherein the second sample image contains a second sample sign, and the second sample image is labeled with actual region information of the second sample sign; performing feature extraction processing on the second sample image according to a preset detection network to obtain second image features of the second sample image; performing prediction processing according to the detection network and the second image features to obtain region prediction information of the second sample sign; updating model parameters of the detection network according to the region prediction information and the actual region information until the detection network converges, and taking the detection network as the first network model.

5. The method of claim 1 to 4, wherein, The method further comprises: acquiring position information of the lane guide sign; According to the arrow information and the position information, a road indication map is generated.

6. The method of claim 1 to 4, wherein, The state image of the lane indication sign is obtained by: Obtaining a video of the lane indication sign; Parsing the video into multiple frames of images; Each frame of image in the multiple frames of images is taken as the state image.

7. A lane guide sign recognizing apparatus characterized by comprising: The lane indication sign recognition device comprises: An acquisition unit configured to acquire a state image of a lane indication sign; A detection unit configured to perform sign region detection processing on the state image according to a preset first network model to obtain a region image of the lane indication sign; An identification unit configured to sequentially perform optical character recognition on characters in the region image according to a preset second network model to obtain arrow information of the lane indication sign. The lane indication sign recognition device further comprises a first training unit configured to, before the step of sequentially identifying characters in the region image according to the preset second network model to obtain the arrow information of the lane indication sign, perform the following operations: Obtain a first sample image, wherein the first sample image contains a first sample sign, and the first sample image is labeled with actual arrow sequence information of the first sample sign; Perform feature extraction processing on the first sample image according to a preset recognition network model to obtain first image features of the first sample image; Perform identification processing according to the recognition network model and the first image features to obtain arrow sequence prediction information of the first sample sign; Update model parameters of the recognition network model according to the arrow sequence prediction information and the actual arrow sequence information until the recognition network model converges, and then take the recognition network model as the second network model.

8. A server, characterized by The server comprises: One or more processors; Memory; and One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the lane indication sign recognition method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the lane indication sign recognition method of any one of claims 1 to 6.

Images