Sign recognition method and device, computer device, and storage medium

Abstract

The present application provides a sign recognition method, apparatus, computer device, and storage medium. The method comprises: obtaining an image to be detected; performing sign identification detection on the image to be detected based on a trained sign identification detection model to obtain the sign identification in the image to be detected; obtaining a sign image in the image to be detected based on the sign identification, and processing the sign image to obtain a processed sign image; performing sign information recognition on the processed sign image based on a sign information recognition model to obtain sign information in the image to be detected, wherein the sign information includes a road number and a mileage value. Using this method, not only can the accuracy of sign recognition be improved, thereby obtaining more accurate road numbers and mileage values ​​to improve road sign information and enhance map construction accuracy, but it can also save labor costs.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a label recognition method and device, computer equipment and a storage medium. BACKGROUND

[0002] Artificial intelligence (AI) is a comprehensive technology of computer science, which can make machines have the functions of perception, reasoning and decision-making by studying the design principles and implementation methods of various intelligent machines. With the rapid development of artificial intelligence technology, the proportion of image recognition function based on computer vision technology in the image processing field is becoming larger and larger, and more and more infrastructure in application scenarios begins to use this kind of image recognition function to build, for example, the map in the automatic driving scenario, that is, the above-mentioned image recognition function is needed to identify various traffic signs from massive data to build a high-precision map.

[0003] However, the existing image recognition technology for map construction can only stably identify relatively large-sized traffic elements, and cannot stably identify the information contained in small-sized traffic elements such as road signs, especially the important sign in the highway - the milestone, because it only accounts for less than 0.3% in the whole picture collected, so it is extremely difficult to extract useful information of the sign by using the existing image recognition technology, and the sign recognition accuracy is not high. SUMMARY

[0004] Therefore, it is necessary for the present application to provide a label recognition method, device, computer equipment and storage medium to improve the label recognition accuracy in view of the above technical problems.

[0005] In a first aspect, the present application provides a label recognition method, and the image processing comprises:

[0006] obtaining a to-be-detected image;

[0007] performing label identification detection on the to-be-detected image based on a trained label identification detection model to obtain label identification in the to-be-detected image;

[0008] obtaining a label image in the to-be-detected image according to the label identification, and processing the label image to obtain a processed label image;

[0009] performing label information identification on the processed label image based on a label information identification model to obtain label information in the to-be-detected image, wherein the label information comprises road number and course value.

[0010] In some embodiments of the present application, the step of obtaining a label image in the to-be-detected image according to the label identification and processing the label image to obtain a processed label image comprises:

[0011] extracting a label image in the to-be-detected image according to the label identification in the to-be-detected image;

[0012] performing filter-sharpening processing on the label image to obtain a sharpened label image;

[0013] performing adjustment processing on the sharpened label image to obtain an adjusted label image;

[0014] performing segmentation processing on the adjusted label image to obtain the processed label image.

[0015] In some embodiments of the present application, the step of performing segmentation processing on the adjusted label image to obtain the processed label image comprises:

[0016] determining image segmentation information corresponding to the adjusted label image, the image segmentation information comprising at least one of segmentation number information and segmentation size information;

[0017] performing segmentation processing on the adjusted label image according to the image segmentation information to obtain at least two label segmentation images;

[0018] determining the at least two label segmentation images as the processed label image.

[0019] In some embodiments of the present application, before the step of performing label identification detection on the to-be-detected image based on the trained label identification detection model to obtain the label identification in the to-be-detected image, the method further comprises:

[0020] constructing an initial label identification detection model;

[0021] obtaining a sample image set, the sample image set comprising a plurality of sample images with labeled label identifications, the label identification being a road label identification satisfying a preset condition;

[0022] inputting the sample image set into the initial label identification detection model for training to obtain the trained label identification detection model.

[0023] In some embodiments of the present application, the step of obtaining a sample image set comprises:

[0024] obtaining a driving record video;

[0025] performing frame extraction on the driving record video to obtain a plurality of video images;

[0026] obtaining annotation information of each of the video images operated by a user, the annotation information including the road sign identification;

[0027] determining a video image in which the road sign identification has been annotated as a candidate sample image;

[0028] obtaining the sample image set according to the road sign identification in the candidate sample image.

[0029] In some embodiments of the present application, the step of obtaining the sample image set according to the road sign identification in the candidate sample image comprises:

[0030] extracting a road sign image in the candidate sample image according to the road sign identification in the candidate sample image;

[0031] if pixel information of the road sign image is less than or equal to a preset pixel threshold, determining that the road sign identification satisfies a preset condition, determining the road sign identification as the sign identification, and determining the candidate sample image as the sample image;

[0032] performing transformation processing on the sample image in which the sign identification has been annotated to obtain a first sample transformed image corresponding to the sample image, the transformation processing at least including one of the following: flipping processing, blurring processing, rotating processing, and cutting processing;

[0033] obtaining the sample image and the first sample transformed image to obtain the sample image set.

[0034] In some embodiments of the present application, the step of obtaining the sample image set according to the road sign identification in the candidate sample image comprises:

[0035] extracting a road sign image in the candidate sample image according to the road sign identification in the candidate sample image;

[0036] if pixel information of the road sign image is less than or equal to a preset pixel threshold, determining that the road sign identification satisfies a preset condition, determining the road sign identification as the sign identification, and determining the candidate sample image as the sample image;

[0037] extracting a sign sample image in the sample image according to the sign identification in the sample image;

[0038] performing sign pasting on the sample image and a non-candidate sample image based on the sign sample image in the sample image to obtain the sample image set, the non-candidate sample image being a video image in which the road sign identification has not been annotated.

[0039] In some embodiments of the present application, based on the sign sample image in the sample image, the sample image and the non-candidate sample image are pasted with the sign, and the step of obtaining the sample image set comprises:

[0040] Obtain vanishing point information in the sample image and the non-candidate sample image;

[0041] According to the vanishing point information, the road region in the sample image and the non-candidate sample image is determined respectively;

[0042] Based on the edge blank area corresponding to the road region, the sign sample image is pasted in the sample image and the non-candidate sample image to obtain the second sample transformation image corresponding to the sample image and the processed non-candidate sample image;

[0043] Obtain the sample image, the second sample transformation image and the processed non-candidate sample image to obtain the sample image set.

[0044] In a second aspect, the present application provides a sign recognition device, and the image processing comprises:

[0045] An image acquisition module is configured to acquire a to-be-detected image;

[0046] An image detection module is configured to detect the to-be-detected image based on a trained sign identification detection model to obtain a sign identification in the to-be-detected image;

[0047] An image processing module is configured to acquire a sign image in the to-be-detected image according to the sign identification, and process the sign image to obtain a processed sign image;

[0048] An image recognition module is configured to recognize the processed sign image based on a sign information recognition model to obtain sign information in the to-be-detected image, wherein the sign information comprises a road number and a journey value.

[0049] In a third aspect, the present application further provides a computer device comprising a processor and a memory, wherein the memory stores a plurality of instructions, and the instructions are adapted to be loaded by the processor to execute the steps in the above method.

[0050] In a fourth aspect, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by the processor to execute the steps in the above method.

[0051] The sign identification method, device, computer device and storage medium described above, through the trained sign identification detection model, performs sign identification detection on the to-be-detected image to obtain the sign image in the to-be-detected image, and then processes the sign image, so that the image quality of the sign image is improved before the sign image is input into the sign information identification model to perform sign information identification, and the sign information output by the sign information identification model is more accurate. The method combines the detection model and the identification model to analyze the to-be-detected image, can realize accurate identification of the sign information, and processes the sign image before identifying the sign information, which can not only obtain more accurate road numbers and course values to improve the sign information of the road and improve the map construction accuracy, but also save labor cost and reduce labor consumption of image analysis. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1 is a scene schematic diagram of the sign identification method in the embodiments of the present application;

[0054] Figure 2 is a flowchart of the sign identification method in the embodiments of the present application;

[0055] Figure 3 is a sample schematic diagram of the sample image set in the embodiments of the present application;

[0056] Figure 4 is a specific flowchart of the sign detection and identification step in the embodiments of the present application;

[0057] Figure 5 is a structure schematic diagram of the sign identification device in the embodiments of the present application;

[0058] Figure 6 is a structure schematic diagram of the computer device in the embodiments of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0060] In the description of the present application, the terms "first", "second", etc. are used only for the purpose of description, and should not be understood as indicating or implying relative importance or implying the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0061] In the description of the present application, the word "for example" is used to mean "serving as an example, instance, or illustration". Any embodiment described as "for example" in this application is not necessarily to be construed as preferred or advantageous over other embodiments. The following description is presented to enable any person skilled in the art to make and use the application. In the following description, for the purpose of explanation, details are set forth. It should be understood that one of ordinary skill in the art can realize the application without using these specific details. In other instances, well-known structures and processes are not described in detail in order to avoid obscuring the description of the application. Therefore, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0062] In the embodiments of the present application, the label recognition method mainly involves computer vision technology (CV) and natural language processing technology (NLP) in artificial intelligence (AI). Among them, artificial intelligence is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence.

[0063] Computer vision is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing to make the computer processing become images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish artificial intelligence systems that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common face recognition, fingerprint recognition and other biometric identification technologies. In this application, for the image to be detected, CV mainly realizes target object recognition in image detection and image identification of image semantic understanding (ISU), identifies and predicts the target object in the image and outputs the prediction result. It can be understood that the target object can be any object determined by actual business requirements, such as people, vehicles, license plates, and other people or objects.

[0064] Natural language processing is an important direction in the field of computer science and artificial intelligence, which studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science and mathematics. Therefore, the research in this field will involve natural language, i.e. the language used in daily life, so it has a close relationship with the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph and other technologies.

[0065] In the embodiments of the present application, it should be noted that the sign recognition method provided by the present application is executed in a computer device, and the processing objects of each computer device exist in the form of data or information, such as time, which is essentially time information. It can be understood that in subsequent embodiments, if the size, quantity, position, etc. are mentioned, they are corresponding data, so that the computer device can process, and the specific details are not described here.

[0066] In the embodiments of the present application, it should be noted that the sign recognition method provided by the present application can be applied to, for example Figure 1In the sign recognition system shown. The sign recognition system includes a terminal 100 and a server 200. The terminal 100 can be a device that includes both receiving and transmitting hardware, that is, a device with receiving and transmitting hardware that can perform two-way communication on a two-way communication link. Such a device may include: a cellular or other communication device with a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The terminal 100 can specifically be a desktop terminal or a mobile terminal. The terminal 100 can also specifically be one of a mobile phone, a tablet computer, a laptop computer, etc., or a camera installed at the monitoring site for information collection, storage, and transmission. The server 200 can be an independent server, or a server network or server cluster composed of servers, including but not limited to computers, network hosts, single network servers, multiple network server sets, or cloud servers composed of multiple servers. The cloud server is composed of a large number of computers or network servers based on cloud computing (Cloud Computing).

[0067] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer computer devices as shown in Figure 1 Only one server 200 is shown in FIG. 1 . It is understandable that the sign recognition system may also include one or more other servers, which are not specifically limited here. Figure 1 As shown, the sign recognition system may further include a memory for storing data, such as video surveillance data.

[0068] Finally, it should be noted that Figure 1 The scenario diagram of the sign recognition system shown is only an example. The sign recognition system and scenario described in the embodiment of the present invention are intended to more clearly illustrate the technical solution of the embodiment of the present invention, and do not constitute a limitation on the technical solution provided by the embodiment of the present invention. Ordinary technicians in this field can know that with the evolution of the sign recognition system and the emergence of new business scenarios, the technical solution provided by the embodiment of the present invention is also applicable to similar technical problems.

[0069] See Figure 2 The embodiment of the present application provides a method for identifying a sign, which is mainly applied to the above Figure 1 Taking the server 200 in FIG. 1 as an example, the method includes steps S201 to S204, which are specifically as follows:

[0070] S201: Acquire an image to be detected.

[0071] The to-be-detected image refers to an image that needs to be detected and identified for sign information, and includes but is not limited to a picture, a video frame in a video, and the like. The video includes but is not limited to a short video and a long video. The short video can be a video with a length less than 10 minutes, and the long video can be a video with a length greater than 10 minutes. The short video or the long video can be a video collected by a road monitoring camera, a video collected by a car recorder, or a road scene or highway condition video collected by a drone, a handheld device, or a controllable device. Therefore, the to-be-detected image can be a driving image collected by a car recorder, a road condition image collected by a road monitoring camera, a street scene image or a road condition image collected by another device, and the like.

[0072] Specifically, before the server 200 obtains the to-be-detected image, the server 200 can first obtain a target video through a preset channel. The target video can be a video collected by the machine device described above. The time at which the target video is obtained can be a preset periodic time or a preset specified time. The target video can be obtained in a manner of actively obtaining by sending a request or passively receiving through information security verification. A plurality of video images obtained by frame extraction based on the target video can be used as to-be-detected images for subsequent input to the trained sign identification detection model. The input manner can be frame-by-frame input or combined input in a video frame group. The specific manner can be set according to actual application requirements, and is not limited in the embodiments of the present application.

[0073] S202, performing sign identification detection on the to-be-detected image based on the trained sign identification detection model to obtain a sign identification in the to-be-detected image.

[0074] The sign identification detection model can be a neural network model with a convolutional layer. The sign identification detection model is used to detect and process the sign identification in the to-be-detected image. The convolutional layer is composed of a plurality of convolution units. The parameters of each convolution unit are obtained by a back propagation algorithm. The purpose of the convolution operation is to extract different features of the input. The sign identification can be the identification of a road sign, such as a milestone on a highway, which has road numbers and / or journey values. The sign identification in the to-be-detected image can be a collection of all pixels in a partial region, but does not exclude a pixel set that can include other things in addition to the sign identification in the to-be-detected image.

[0075] Specifically, after the server 200 obtains the to-be-detected image, the to-be-detected image can be input into the trained sign identification detection model. Since the sign identification detection model has been trained by using the sample images pre-labeled with sign identification in the previous step, the sign identification detection model learns the ability to detect sign identification in the training process. Therefore, after the to-be-detected image is input into the trained sign identification detection model, if the to-be-detected image contains sign identification, the trained sign identification detection model will output the to-be-detected image with the sign identification marked. The model labeling type can be a bounding box labeling, that is, the position of the sign identification in the to-be-detected image is marked in the form of a bounding box. The sample image obtaining step and the model training step involved in this embodiment will be described in detail below.

[0076] In an embodiment, in order to perform sign identification detection processing on the to-be-detected image to obtain the sign identification in the to-be-detected image for subsequent analysis processing, it is necessary to pre-train the sign identification detection model to obtain the trained sign identification detection model. At this time, before performing the sign identification detection processing on the to-be-detected image, the step further includes: constructing an initial sign identification detection model; obtaining a sample image set, the sample image set including a plurality of sample images labeled with sign identification, the sign identification being a road sign identification satisfying a preset condition; inputting the sample image set into the initial sign identification detection model for training to obtain the trained sign identification detection model.

[0077] The initial sign identification detection model can be a pre-trained neural network model or an un-pretrained neural network model, and the sign identification detection model has a convolutional layer. The sample image can be an image labeled with sign identification for training the detection model. In this embodiment, the sign identification can be a road sign identification with a size satisfying a preset condition, for example, a milestone identification.

[0078] Specifically, before the server 200 performs sign identification detection processing on the to-be-detected image by using the trained sign identification detection model, an initial sign identification detection model needs to be constructed first. If the initial sign identification detection model is a pre-trained neural network model trained by using a preset image training set, it will have certain initialization parameters, and part of the model training process can be omitted in the subsequent training process, saving the model training time. If the initial sign identification detection model is an un-pretrained neural network model, it needs to be trained completely by using a sample image set until it has the ability to detect sign identification. It can be understood that the above-mentioned preset image training set can be a preset image set with the same properties as the sample image set, that is, the selection of the image training set is not random, but is biased towards the selection of model parameters suitable for subsequent image detection.

[0079] More specifically, the sample image acquired by the server 200 is a sample image including multiple labeled signage logos. The signage logos in each sample image can be marked with a border frame style. The annotation tool can be labellmg written in Python language, which supports running on cross-platforms such as Windows and Linux, and can mark the specified target object with a frame through a visual operation interface.

[0080] It should be noted that the sample images involved in the embodiments of the present application are actually images that have undergone image screening and preprocessing, that is, before obtaining the sample image set, the server 200 must first obtain a plurality of initial images that are not marked with any logos, and then, based on the actual application scenario of the present application, mark the road sign logos (such as milestone logos) in each initial image. At this time, the initial image marked with road sign logos is not necessarily a sample image, but an initial image whose road sign logos meet the preset conditions. The sample image can be obtained by screening out the initial image whose road sign logos meet the preset conditions. At this time, the sign logos in the sample image can be road sign logos that meet the preset pixel conditions. The sample image set with the marked sign logos is input into the initial sign logo detection model. The server 200 will train the model according to the preset command. The trained sign logo detection model can have the sign logo detection capability. For the pictures subsequently input into the model, it can detect whether they contain the sign logos learned in the early stage, and mark the sign logos if they are detected.

[0081] In one embodiment, the step of obtaining a sample image set includes: obtaining a driving record video; extracting frames from the driving record video to obtain multiple video images; obtaining annotation information of each of the video images operated by the user, the annotation information including the road sign logo; determining a video image that has been labeled with the road sign logo as a candidate sample image; and obtaining the sample image set based on the road sign logo in the candidate sample image.

[0082] Specifically, the above embodiments have mentioned that the source of the image to be detected can be a video collected by a road monitoring camera, a video collected by a driving recorder, or a road street scene or highway condition video collected by a drone, handheld or controllable device. It can be understood that the sample images involved in the embodiments of the present application can also be obtained after processing the videos collected by the above-mentioned methods. Although this embodiment only analyzes the video collected by the driving recorder - the driving record video to obtain a sample image set, it does not rule out that other video analyses are used in other embodiments to obtain sample image sets, and the specific embodiments of the present application are not limited.

[0083] More specifically, before the server 200 acquires the sample image set, the server 200 can first extract a plurality of video images from a driving record video acquired periodically or in real time by using an image processing tool to obtain sample images. For example, the video can be extracted by using a tool such as OpenCV (OpenCV is a cross-platform computer vision and machine learning software library) or a tool such as ffmpeg (ffmpeg is a set of open source computer programs that can be used to record, convert digital audio, video, and convert them into streams). The extraction frequency can be once per second or more than once per second, and the number of frames extracted can be one frame or more than one frame per time, which is not limited in the embodiments of the present application. After the server 200 obtains the video images, the server 200 can obtain the labeling information of each video image operated by the user through the terminal 100. The labeling information includes road sign identification. At this time, some or all of the video images labeled with road sign identification can be used as candidate sample images, and some of the video images without road sign identification can be removed. The step of acquiring the sample image set according to the road sign identification in the candidate sample image will be described in detail below.

[0084] In an embodiment, the step of acquiring the sample image set according to the road sign identification in the candidate sample image includes: extracting a road sign image in the candidate sample image according to the road sign identification in the candidate sample image; if the pixel information of the road sign image is less than or equal to a preset pixel threshold, determining that the road sign identification satisfies a preset condition, and determining the road sign identification as the sign identification and the candidate sample image as the sample image; performing transformation processing on the sample image labeled with the sign identification to obtain a first sample transformed image corresponding to the sample image, the transformation processing including at least one of the following: flipping processing, blurring processing, rotating processing, and cutting processing; and obtaining the sample image and the first sample transformed image to obtain the sample image set.

[0085] In the formula, pixel refers to the basic coding of basic primary color and its gray scale, which is the basic unit of digital image. The size of image resolution is usually represented by pixels per inch (PPI).

[0086] Specifically, after the server 200 obtains the candidate sample images, the server 200 can extract the road sign images from the labeled road sign marks in the candidate sample images, that is, extract the pixel set in the bounding box region around the road sign mark, as the road sign image. For each road sign image, the server 200 first needs to obtain the pixel information corresponding to the image, and match the pixel information of each road sign image with the preset pixel threshold. If the pixel information of a road sign image is less than or equal to the preset pixel threshold, it can be determined that the road sign mark corresponding to the road sign image satisfies the preset condition, and the candidate sample image corresponding to the road sign image can be used as a sample image, and the road sign mark in the road sign image can be used as a sign mark.

[0087] More specifically, after the server 200 analyzes the road sign marks in each candidate sample image and determines the sample images in all candidate sample images, the server 200 can perform flipping processing, blurring processing, rotation processing, and cutting processing on each sample image to obtain first sample transformed images processed based on different processing methods. Finally, integrating the first sample transformed images and their corresponding sample images can obtain a sample image set. The above process is called a data augmentation process, and the purpose is to obtain a large number of sample images to train a detection model.

[0088] In one embodiment, the step of obtaining the sample image set according to the road sign marks in the candidate sample images includes: extracting road sign images in the candidate sample images according to the road sign marks in the candidate sample images; if the pixel information of the road sign image is less than or equal to the preset pixel threshold, it is determined that the road sign mark satisfies the preset condition, and the road sign mark is determined as the sign mark, and the candidate sample image is determined as the sample image; extracting the sign sample image in the sample image according to the sign mark in the sample image; based on the sign sample image in the sample image, pasting the sign mark on the sample image and a non-candidate sample image to obtain the sample image set, the non-candidate sample image being a video image that is not labeled with the road sign mark.

[0089] Specifically, the previous embodiment details the first way of obtaining the sample image set, and the present embodiment will explain the second way of obtaining the sample image set, i.e., providing a second data augmentation scheme, which is different from the previous embodiment in that, in the previous embodiment, after the sample images in the candidate sample images are screened out, the sample image with the label mark is transformed to obtain the first sample change image after the sample image is transformed, and the two parts of the image are integrated to obtain the sample image set. However, the present embodiment proposes that after the sample images in the candidate sample images are screened out, the label sample images corresponding to the label marks in each sample image are first extracted, and then the label sample images are used to perform label pasting processing on each sample image with a label mark. At the same time, the label sample images are used to perform label pasting processing on each non-candidate sample image without a road label mark, so that the sample image and the non-candidate sample image both have a label mark with pixel information meeting a preset condition, and the augmentation effect of the training sample data is achieved. The second sampling method explained in the present embodiment and the first sampling method explained in the previous embodiment have higher sampling efficiency and more stable sample quality than the existing simple copy-paste repeated sampling technology, and provide great help to improve the image label recognition accuracy. The label pasting processing steps involved in the present embodiment will be described in detail below.

[0090] In an embodiment, the step of pasting the label on the sample image and the non-candidate sample image based on the label sample image in the sample image to obtain the sample image set comprises: obtaining vanishing point information in the sample image and the non-candidate sample image; determining a road region in the sample image and the non-candidate sample image respectively according to the vanishing point information; based on the edge blank region corresponding to the road region, pasting the label sample image in the sample image and the non-candidate sample image to obtain a second sample change image corresponding to the sample image and a processed non-candidate sample image; obtaining the sample image, the second sample change image and the processed non-candidate sample image to obtain the sample image set.

[0091] The vanishing point is the point at which two or more parallel lines converge in perspective projection, as shown in the image. Figure 3 The vanishing point has been marked in the image.

[0092] Specifically, in addition to obtaining the first sample change image by analyzing the sample image, the server 200 can also analyze the sample image labeled with the sign identification and the video image-non-candidate sample image without labeling the road sign identification, so as to augment the sample image to the sample image set meeting the model training quantity requirement. The sample image set acquisition method includes that the server 200 or the user can first analyze and determine the vanishing point information in each sample image and non-candidate sample image. The determination of the vanishing point information needs to use the perspective imaging technology. If it is parallel perspective, there is only one vanishing point behind the object in the middle. The determination method is to extend the two lines with converging tendency on the left and right depth of the object to the rear. If it is angular perspective, there are two vanishing points behind the object on both sides. The determination method is to extend the four lines with converging tendency on the left and right sides of the object to the rear, and intersect with each other on the left and right sides of the object to form two vanishing points. Among them, any straight line parallel to the picture does not have a vanishing point, and any set of parallel lines with a certain angle with the picture has a vanishing point. However, if the angle is 90 degrees, it is parallel perspective, otherwise it is angular perspective.

[0093] More specifically, after the server 200 determines the vanishing point information in each sample image and non-candidate sample image, the server 200 can further determine the road region in the image based on the vanishing point in each image, for example Figure 3 The triangular region shown is a road region formed by roughly estimating the position of the triangular bottom edge of the collected picture by artificial method, and then dividing the triangular region. Alternatively, a model can be established to more accurately segment the road region. The region segmentation method can be realized by model training. After the server 200 determines the vanishing point information and the road region in each sample image and non-candidate sample image, the sign sample image processed by the operations such as blurring, rotating and scaling is non-overlappingly pasted in the sample image and the non-candidate sample image, and specifically pasted in the image blank area on the edge of the road region in the image, as shown in Figure 3 , so as to obtain the second sample change image after the sample image is pasted, and the processed non-candidate sample image. Finally, the sample image, the second sample change image and the processed non-candidate sample image can be integrated as a sample image set. It can be understood that the above pasting operation can be actually performed by the terminal 100 with an interactive interface, that is, the staff intending to obtain the sign information can perform the image insertion and pasting through the terminal 100, so as to fuse at least two pictures into one.

[0094] S203, obtaining the sign image in the to-be-detected image according to the sign identification, and processing the sign image to obtain a processed sign image.

[0095] Specifically, in the subsequent image recognition process, not all regions of the to-be-detected image are input into the model, but part of the regions, i.e., the label image in the to-be-detected image, is input into the model for recognition. In this embodiment, it will be described that the label image in the to-be-detected image is not directly input into the model, but the label image is first processed to obtain a processed label image which can be input into the model for label information recognition. The label image processing steps involved in this embodiment will be described in detail below.

[0096] In one embodiment, this step includes: extracting a label image in the to-be-detected image according to a label in the to-be-detected image; performing a filter-sharpening processing on the label image to obtain a sharpened label image; performing an adjustment processing on the sharpened label image to obtain an adjusted label image; performing a segmentation processing on the adjusted label image to obtain the processed label image.

[0097] Wherein, the spatial transformation network (STN) is a convolutional neural network architecture model, which reduces the influence of data spatial diversity by transforming the input picture, to improve the classification accuracy of the convolutional network model, instead of changing the network structure. The STN can adaptively perform spatial transformation and alignment on data, so that the model remains invariant to translation, scaling, rotation or other transformations.

[0098] Specifically, in the field of image processing, the main purpose of sharpening processing is to highlight the details in the image, or to enhance the blurred details. From the logical point of view, sharpening processing can be realized by spatial differentiation, and spatial differentiation inevitably needs to select a differential operator, such as Sobel operator, Robers operator, Laplace operator, etc. If the Laolace operator is used, its application emphasizes the mutation of the gray scale in the image and reduces the area with slow gray scale change, which can produce an image that superimposes the light gray lines and mutation points in the image onto the dark background, making the digital edges of the image clearer and improving the label recognition accuracy of the image.

[0099] More specifically, the embodiment of the present application proposes to adopt a filtering method to sharpen the sign image in the to-be-detected image, and the sign image can be cut out from the to-be-detected image. Since the cut-out sign image may have a certain angle tilt, it is necessary to adopt a filtering method to filter and sharpen the sign image to obtain a sharpened sign image. For the sharpened sign image, a spatial transformation network can be used for rotation adjustment to make it angle-neutral as an adjusted sign image. Finally, since the sign image involved in the embodiment of the present application includes a milestone image, the milestone usually contains two kinds of information of road number and mileage value. Therefore, before the sign information identification of the sign image in the to-be-detected image is performed, the embodiment proposes to not only extract the sign image in the to-be-detected image and perform filtering and sharpening and adjustment processing, but also perform cutting processing on the adjusted sign image, that is, cutting according to the information distribution position in the milestone image, and the cutting size and the cutting number are determined by the information distribution position in the milestone image. The specific cutting processing steps will be described in detail below.

[0100] In an embodiment, the step of performing cutting processing on the adjusted sign image to obtain the processed sign image comprises: determining image cutting information corresponding to the adjusted sign image, the image cutting information comprising at least one of cutting number information and cutting size information; performing cutting processing on the adjusted sign image according to the image cutting information to obtain at least two sign cutting images; and determining the at least two sign cutting images as the processed sign image.

[0101] The cutting number information refers to the number of images after the adjusted sign image is cut, for example, 2, 4, 6, etc. The cutting size information refers to the size of the image after the adjusted sign image is cut, for example, 1 inch 25mmx35mm, pixel 2560X1920, etc.

[0102] Specifically, after the sign image including the milestone image is adjusted and cut, at least two sign cutting images can be obtained, including a road number image and a mileage value image, which are used as the processed sign image for subsequent sign information identification processing. It can be understood that although the information distribution position of the milestone image is explained in the embodiment of the present application, the milestone image may also include information other than the road number and the mileage value in other embodiments, that is, the processed sign image is not necessarily two, and the specific embodiment of the present application is not limited.

[0103] S204, performing sign information identification on the processed sign image based on a sign information identification model to obtain sign information in the to-be-detected image, the sign information comprising a road number and a mileage value.

[0104] The sign information recognition model can be a combination of a convolutional neural network (CNN) and a recurrent neural network (RNN), i.e., a convolutional recurrent neural network (CRNN) model. The CRNN model includes a convolutional layer, a recurrent layer, and a transcription layer, and can be used for text processing.

[0105] Specifically, the sign information recognition model provided in the embodiments of the present application does not need to perform character segmentation and horizontal scaling operations when performing sign information recognition processing on the processed sign image, but only needs to be scaled to a fixed length in the vertical direction. The convolutional layer of the CRNN model is actually a common CNN network, which is used to extract convolutional feature maps of the input image, i.e., to convert the image into a convolutional feature matrix; the recurrent layer of the CRNN model is a deep bidirectional long short-term memory (LSTM) network, which continues to extract text sequence features on the basis of the convolutional features and can recognize sequences of any length, and the output of the recurrent layer is a predicted sequence result; and the transcription layer of the CRNN model is used to convert the predicted result into a label sequence, i.e., to find a label sequence with the highest probability according to the label distribution of each frame. For example, “-hh-e-l-ll-oo-” (the “-” represents a blank label) is mapped to “hello”, and the transcription process is to first delete the repeated labels and then delete the blank labels.

[0106] The above embodiments provide a sign recognition method. The trained sign identification detection model is used to perform sign identification detection processing on the to-be-detected image to obtain the sign identification in the to-be-detected image, thereby obtaining the sign image in the to-be-detected image, and the sign image is processed to improve the image quality before being input to the sign information recognition model for sign information recognition processing, so as to make the sign information output by the sign information recognition model more accurate. The method combines the detection model and the recognition model to analyze the to-be-detected image, can realize accurate recognition of the sign information, and processes the sign image before recognizing the sign information, which not only can obtain more accurate road numbers and course values to improve the sign information of the road and improve the accuracy of map construction, but also can save labor costs and reduce labor consumption for image analysis. At the same time, the method also provides two different sampling methods, which can not only save the sampling process and improve the sampling efficiency, but also can obtain sample images that meet the accuracy of actual business needs, so as to make the model training effect better and further improve the sign recognition accuracy.

[0107] It should be understood that, althoughFigure 2 The steps in the flowchart are displayed in sequence according to the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, the execution of the steps is not strictly limited in order, and the steps can be executed in other orders. Moreover, Figure 2 At least part of the steps in the flowchart can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0108] To enable those skilled in the art to fully understand the sign recognition method proposed in the present application, the present application also provides an application scenario applying the above method. Specifically, the method in the application of the application scenario can be referred to Figure 4 , in combination with Figure 4 The flowchart is as follows:

[0109] In this embodiment, small-sized milestone signs are mainly explained and described.

[0110] First, the server 200 needs to pre-acquire and construct a sample image set according to the picture stream before performing sign identification detection and sign information recognition on the to-be-detected image, so as to train the sign identification detection model using the sample image set, so that the sign identification detection model has the sign identification detection capability. The process of constructing the sample image set for the picture stream has been described in detail in the above embodiment, that is, the first mode of repeatedly sampling using sample images, and the second mode of random pasting sampling (Milestone-Paster) using sample images and non-candidate sample images; the to-be-detected image can be an image containing a real scene of a highway, the sign is a milestone sign, and the sign identification detection model can be a neural network model with a convolutional structure.

[0111] Then, the server 200 controls the sign identification detection model to be trained to obtain a trained sign identification detection model, which can be used for sign identification detection on the to-be-detected image obtained at present, so that the model outputs the sign identification in the to-be-detected image. In order to improve the image sign recognition accuracy, the present application proposes that the detection result needs to be pre-processed before being input into the sign information recognition model, that is, the sign image corresponding to the sign identification in the to-be-detected image is sharpened and spatially transformed to obtain a processed sign image, which is then input into the sign information recognition model, so that the sign information recognition model performs sign information recognition on the processed sign image, and outputs the sign information in the to-be-detected image, that is, the road number and the journey value in the milestone image.

[0112] Finally, the road number and the course value in the to-be-detected image are obtained, the highway information is perfected, and a high-precision map is constructed.

[0113] Beneficial effects: The sign identification detection model is used for sign identification detection processing on the to-be-detected image to obtain the sign image in the to-be-detected image, so that the sign image is processed to improve the image quality before being input to the sign information recognition model for sign information recognition processing, so that the sign information output by the sign information recognition model is more accurate. The method combines the detection model and the recognition model to analyze the to-be-detected image, can realize accurate identification of the sign information, and processes the sign image before identifying the sign information, which can not only obtain more accurate road number and course value to improve the sign information of the road and improve the precision of the map construction, but also save the labor cost and reduce the labor consumption of image analysis. At the same time, the method also provides two different sampling methods, which can not only save the sampling process and improve the sampling efficiency, but also obtain sample images meeting the accuracy of actual business requirements, so as to improve the model training effect and further improve the sign recognition accuracy.

[0114] In order to better implement the sign recognition method provided in the embodiments of the present application, based on the sign recognition method, the embodiments of the present application further provide a sign recognition device 500, as shown in Figure 5 The sign recognition device 500 comprises:

[0115] An image acquisition module 502 is configured to acquire a to-be-detected image.

[0116] An image detection module 504 is configured to perform sign identification detection on the to-be-detected image based on a trained sign identification detection model to obtain a sign identification in the to-be-detected image.

[0117] An image processing module 506 is configured to obtain a sign image in the to-be-detected image according to the sign identification, and process the sign image to obtain a processed sign image.

[0118] An image recognition module 508 is configured to perform sign information recognition on the processed sign image based on a sign information recognition model to obtain sign information in the to-be-detected image, wherein the sign information comprises a road number and a course value.

[0119] In an embodiment, the image processing module 506 is further configured to extract a sign image from the to-be-detected image according to the sign identification in the to-be-detected image; perform filter-sharpening processing on the sign image to obtain a sharpened sign image; perform adjustment processing on the sharpened sign image based on a spatial transformation network to obtain an adjusted sign image; and perform segmentation processing on the adjusted sign image to obtain the processed sign image.

[0120] In an embodiment, the image processing module 506 is further configured to determine image segmentation information corresponding to the adjusted sign image, the image segmentation information comprising at least one of segmentation quantity information and segmentation size information; perform segmentation processing on the adjusted sign image according to the image segmentation information to obtain at least two sign segmentation images; and determine the at least two sign segmentation images as the processed sign image.

[0121] In an embodiment, the sign identification apparatus 500 further comprises a model training module 503 configured to construct an initial sign identification detection model; obtain a sample image set, the sample image set comprising a plurality of sample images with annotated sign identification, the sign identification being a road sign identification satisfying a preset condition; input the sample image set to the initial sign identification detection model for training to obtain the trained sign identification detection model.

[0122] In an embodiment, the model training module 503 is further configured to obtain a driving record video; perform frame extraction on the driving record video to obtain a plurality of video images; obtain annotation information of each of the video images operated by a user, the annotation information comprising the road sign identification; determine a video image with the annotated road sign identification as a candidate sample image; and obtain the sample image set according to the road sign identification in the candidate sample image.

[0123] In an embodiment, the model training module 503 is further configured to extract a road sign image from the candidate sample image according to the road sign identification in the candidate sample image; determine that the road sign identification satisfies a preset condition if pixel information of the road sign image is less than or equal to a preset pixel threshold, and determine the road sign identification as the sign identification and the candidate sample image as the sample image; perform transformation processing on the sample image with the sign identification to obtain a first sample transformed image corresponding to the sample image, the transformation processing comprising at least one of the following: flipping processing, blurring processing, rotating processing, and cutting processing; and obtain the sample image set by obtaining the sample image and the first sample transformed image.

[0124] In an embodiment, the model training module 503 is further configured to extract a road sign image in the candidate sample image according to the road sign identification in the candidate sample image; determine that the road sign identification satisfies a preset condition if pixel information of the road sign image is less than or equal to a preset pixel threshold, and determine the road sign identification as the sign identification and the candidate sample image as the sample image; extract a sign sample image in the sample image according to the sign identification in the sample image; and perform sign pasting on the sample image and a non-candidate sample image based on the sign sample image in the sample image, to obtain the sample image set, the non-candidate sample image being a video image without the road sign identification.

[0125] In an embodiment, the model training module 503 is further configured to obtain vanishing point information in the sample image and the non-candidate sample image; determine a road region in the sample image and the non-candidate sample image respectively according to the vanishing point information; cover and paste the sign sample image in the sample image and the non-candidate sample image based on an edge blank region corresponding to the road region, to obtain a second sample transformed image corresponding to the sample image and a processed non-candidate sample image; and obtain the sample image, the second sample transformed image, and the processed non-candidate sample image, to obtain the sample image set.

[0126] In the above embodiments, the to-be-detected image is analyzed in combination with the detection model and the identification model, which not only can obtain more accurate road numbers and course values to improve road sign information and improve map construction accuracy, but also can save labor cost and reduce labor consumption of image analysis. Meanwhile, two different sampling methods are provided, which not only can save sampling process and improve sampling efficiency, but also can meet business requirements in actual applications, promote better model training effect, and further improve sign identification accuracy.

[0127] In some embodiments of the present application, the sign identification apparatus 500 can be implemented in the form of a computer program, which can run on a computer device as shown in Figure 6 The memory of the computer device can store various program modules constituting the sign identification apparatus 500, such as an image acquisition module 502, an image detection module 504, an image processing module 506, and an image identification module 508 as shown in Figure 5 The computer program constituted by various program modules causes the processor to perform the steps in the sign identification method of various embodiments of the present application described in the specification.

[0128] For example, Figure 6 The computer device as shown in Figure 5The image acquisition module 502 in the sign recognition apparatus 500 shown performs step S201. The computer device can perform step S202 through the image detection module 504. The computer device can perform step S203 through the image processing module 506. The computer device can perform step S204 through the image recognition module 508. The computer device comprises a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with external computer devices through a network connection. The computer program is executed by the processor to implement a sign recognition method.

[0129] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] In some embodiments of the present application, a computer device is provided, comprising one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to execute the steps of the sign recognition method described above by the processor. The steps of the sign recognition method can be the steps in the sign recognition method of each of the above embodiments.

[0131] In some embodiments of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is loaded by a processor, so that the processor executes the steps of the sign recognition method described above. The steps of the sign recognition method can be the steps in the sign recognition method of each of the above embodiments.

[0132] The specific implementation of each of the above operations can refer to the previous embodiments, which will not be described here. The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. Due to the instructions stored in the storage medium, the steps in any of the methods provided in the embodiments of the present application can be executed, and thus the beneficial effects of any of the methods provided in the embodiments of the present application can be achieved. Details can be found in the previous embodiments, which will not be described here.

[0133] The above provides a detailed introduction to the label recognition method, device, computer equipment and storage medium provided by the embodiment of the application. The principle and implementation mode of the application are described by applying specific examples. The above embodiment is only used to help understand the method and core idea of the application. Meanwhile, for those skilled in the art, according to the idea of the application, the specific implementation mode and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the application.

Claims

1. A sign recognition method, characterized in that: The method comprises: Build an initial signage detection model; Get driving record video; Extracting frames from the driving record video to obtain multiple video images; Acquire annotation information of each video image operated by a user, wherein the annotation information includes road sign identification; Determine a video image marked with the road sign as a candidate sample image; Acquire a sample image set according to the road sign identification in the candidate sample image, wherein the sample image set includes a plurality of sample images with annotated sign identifications, and the sign identifications are road sign identifications that meet preset conditions; Inputting the sample image set into the initial sign detection model for training to obtain a trained sign detection model; Obtain the image to be detected; Based on the trained sign detection model, perform sign detection on the image to be detected to obtain the sign in the image to be detected; Acquire a signboard image in the image to be detected according to the signboard identifier, and process the signboard image to obtain a processed signboard image; Based on the sign information recognition model, the processed sign image is subjected to sign information recognition to obtain the sign information in the image to be detected, wherein the sign information includes a road number and a mileage value; The step of obtaining a sample image set according to the road sign identification in the candidate sample image includes: Determining the candidate sample image as the sample image according to the road sign identification in the candidate sample image; Extracting a sign sample image from the sample image according to the sign identification in the sample image; Obtaining vanishing point information in the sample image and non-candidate sample images, wherein the non-candidate sample images are video images not marked with the road sign identification; determining, according to the vanishing point information, road areas in the sample image and the non-candidate sample image respectively; Based on the edge blank area corresponding to the road area, the sign sample image is overlaid and pasted on the sample image and the non-candidate sample image to obtain a second sample transformed image corresponding to the sample image and a processed non-candidate sample image; The sample image, the second sample transformed image, and the processed non-candidate sample image are acquired to obtain the sample image set.

2. The sign recognition method according to claim 1, wherein: The step of acquiring the signboard image in the image to be detected according to the signboard identification and processing the signboard image to obtain the processed signboard image includes: Extracting the signage image from the image to be detected according to the signage identification in the image to be detected; Performing filtering and sharpening processing on the sign image to obtain a sharpened sign image; Performing adjustment processing on the sharpened sign image to obtain an adjusted sign image; The adjusted signage image is segmented to obtain the processed signage image.

3. The sign recognition method according to claim 2, wherein: The step of segmenting the adjusted sign image to obtain the processed sign image includes: Determining image segmentation information corresponding to the adjusted sign image, the image segmentation information including at least one of segmentation quantity information and segmentation size information; Performing segmentation processing on the adjusted sign image according to the image segmentation information to obtain at least two sign segmentation images; The at least two signage segmentation images are determined as the processed signage images.

4. The sign recognition method according to claim 1, wherein: The step of determining the candidate sample image as the sample image according to the road sign identification in the candidate sample image includes: Extracting a road sign image from the candidate sample image according to the road sign identification in the candidate sample image; If the pixel information of the road sign image is less than or equal to a preset pixel threshold, it is determined that the road sign identifier meets the preset condition, and the road sign identifier is determined as the sign identifier, and the candidate sample image is determined as the sample image.

5. The sign recognition method according to claim 1, wherein: The step of acquiring a sample image set according to the road sign identification in the candidate sample image further includes: Performing a transformation process on the sample image marked with the sign logo to obtain a first sample transformed image corresponding to the sample image, wherein the transformation process includes at least one of the following: flipping, blurring, rotating, and cutting; The sample image and the first sample transformed image are acquired to obtain the sample image set.

6. A sign recognition device, characterized in that: The device comprises: An image acquisition module, used to acquire an image to be detected; An image detection module is used to perform sign identification detection on the image to be detected based on a trained sign identification detection model to obtain the sign identification in the image to be detected; An image processing module, configured to obtain a signboard image in the image to be detected according to the signboard identification, and process the signboard image to obtain a processed signboard image; An image recognition module is used to perform sign information recognition on the processed sign image based on a sign information recognition model to obtain sign information in the image to be detected, wherein the sign information includes a road number and a mileage value; The image detection module is further configured to construct an initial sign identification detection model; obtain a driving record video; extract frames from the driving record video to obtain a plurality of video images; obtain annotation information of each of the video images operated by a user, the annotation information including road sign identification; determine a video image annotated with the road sign identification as a candidate sample image; obtain a sample image set based on the road sign identification in the candidate sample image, the sample image set including a plurality of sample images annotated with sign identifications, the sign identifications being road sign identifications that meet preset conditions; input the sample image set into the initial sign identification detection model for training to obtain a trained sign identification detection model; The image detection module is further configured to determine the candidate sample image as the sample image based on the road sign identification in the candidate sample image; extract the signboard sample image from the sample image based on the signboard identification in the sample image; obtain vanishing point information from the sample image and non-candidate sample images, where the non-candidate sample image is a video image not labeled with the road sign identification; determine the road area in the sample image and the non-candidate sample image respectively based on the vanishing point information; overwrite and paste the signboard sample image into the sample image and the non-candidate sample image based on the edge blank area corresponding to the road area to obtain a second sample transformed image corresponding to the sample image and a processed non-candidate sample image; and obtain the sample image, the second sample transformed image, and the processed non-candidate sample image to obtain the sample image set.

7. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the signage recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the sign recognition method according to any one of claims 1 to 5.

Images