Image processing method and device, storage medium and electronic equipment

Abstract

The invention discloses an image processing method and device, a storage medium and electronic equipment. The method comprises the steps that a first driving environment image collected by an intelligent driving vehicle is determined; processing the first driving environment image by using an image processing model, and determining feature information of the first traffic sign and a first recognition result; processing the feature information of the first traffic sign by using a classification model, and determining a second identification result of the first traffic sign; and determining a target identification result of the first traffic sign according to the first identification result and the second identification result. Through the method, the server-side equipment does not need to modify an algorithm model, re-train and publish an image processing model of an updated version, the identification of any mark type can be realized through the classification model on the terminal equipment, and the effects of improving the identification accuracy and saving the manpower, material and financial costs are achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent driving technology, and in particular to an image processing method, device, storage medium and electronic device. Background Art

[0002] With the continuous development of computer technology and image processing technology, more and more fields are using image processing to perform target recognition, classification, tracking and other applications. For example, in the field of intelligent driving, intelligent driving vehicles (referred to as vehicles) recognize images collected during driving and assist vehicles in intelligent driving based on the recognized traffic signs.

[0003] At present, image recognition of traffic signs is mainly achieved through neural network models. There are many types of traffic signs, and traffic signs in different regions and scenes are also very different. The neural network model obtained by training based on the limited types of traffic sign samples that can be collected cannot accurately recognize new types of traffic signs after being deployed on the vehicle. Moreover, as new types of traffic signs continue to increase, the recognition accuracy of the neural network model will become lower and lower, affecting the decision-making and control of the intelligent driving system, and further affecting the safe and stable driving of the vehicle. Summary of the invention

[0004] In the prior art, in order to improve the recognition accuracy of the neural network model, when a new type of traffic sign is discovered, the newly added type of data is mined as a new sample, and then a series of tasks such as labeling, modifying the algorithm model, retraining the neural network model, and releasing an updated version are performed, which consumes a lot of manpower, material resources and financial resources.

[0005] In order to solve the technical problem in the prior art that a large amount of manpower, material and financial resources are needed to improve the accuracy of vehicle recognition of traffic signs, the present invention discloses an image processing method, device, storage medium and electronic device to realize the recognition of any sign without releasing a new version, thereby achieving the effect of improving the recognition accuracy and saving manpower, material and financial costs.

[0006] A first aspect of the present disclosure provides an image processing method, including:

[0007] Determine a first driving environment image collected by the intelligent driving vehicle;

[0008] Processing the first driving environment image using an image processing model to determine feature information of a first traffic sign and a first recognition result;

[0009] Processing the feature information of the first traffic sign using a classification model to determine a second recognition result of the first traffic sign;

[0010] A target recognition result of the first traffic sign is determined according to the first recognition result and the second recognition result.

[0011] A second aspect of the present disclosure provides an image processing device, including:

[0012] A first determining module, used to determine a first driving environment image collected by the intelligent driving vehicle;

[0013] A first processing module, configured to process the first driving environment image using an image processing model to determine feature information of a first traffic sign and a first recognition result;

[0014] a second processing module, configured to process the feature information of the first traffic sign using a classification model to determine a second recognition result of the first traffic sign;

[0015] The second determination module is used to determine a target recognition result of the first traffic sign according to the first recognition result and the second recognition result.

[0016] An embodiment of a third aspect of the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the image processing method proposed in the embodiment of the first aspect of the present disclosure.

[0017] In a fourth aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0018] processor;

[0019] a memory for storing instructions executable by the processor;

[0020] The processor is used to read the instruction from the memory and execute the instruction to implement the image processing method proposed in the embodiment of the first aspect of the present disclosure.

[0021] The fifth aspect of the present disclosure provides a computer program product. When an instruction processor in the computer program product executes, the image processing method provided in the first aspect of the present disclosure is executed.

[0022] The image processing method provided by the embodiment of the present disclosure is provided on the vehicle with a classification model determined based on a registered new type of sign. When the vehicle collects a driving environment image, the driving environment image is processed using a pre-deployed image processing model to determine the characteristic information and the first recognition result of the traffic sign, and the characteristic information is processed using a pre-determined classification model to determine the second recognition result of the traffic sign. Finally, based on the first recognition result and the second recognition result, the recognition type with a higher recognition probability is determined as the target recognition result of the traffic sign. Through this method, there is no need for the server device to modify the algorithm model, retrain, or release an updated version of the image processing model. The classification model on the terminal device can realize the recognition of any sign type, thereby achieving the effect of improving the recognition accuracy and saving manpower, material and financial costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of a network architecture of an image processing system provided in the prior art;

[0024] Figure 2 is a schematic diagram of the composition structure of an image processing system provided by an exemplary embodiment of the present disclosure;

[0025] Figure 3 is a flowchart of an image processing method provided by an exemplary embodiment of the present disclosure;

[0026] Figure 4 is a flowchart of an image processing method provided by another exemplary embodiment of the present disclosure;

[0027] Figure 5 is a flowchart of an image processing method provided by yet another exemplary embodiment of the present disclosure;

[0028] Figure 6 is a flowchart of an image processing method provided by another exemplary embodiment of the present disclosure;

[0029] Figure 7 is a flowchart of an image processing method provided by yet another exemplary embodiment of the present disclosure;

[0030] Figure 8 is a flowchart of an image processing method provided by another exemplary embodiment of the present disclosure;

[0031] Fig. 9 is a flowchart of an image processing method provided by yet another exemplary embodiment of the present disclosure;

[0032] Fig.10 is a flowchart of an image processing method provided by another exemplary embodiment of the present disclosure;

[0033] Fig.11is a schematic diagram of the composition structure of an image processing device provided by an exemplary embodiment of the present disclosure;

[0034] Fig.12 is a schematic diagram of the composition structure of an image processing device provided by another exemplary embodiment of the present disclosure;

[0035] Fig.13 is a schematic diagram of the composition structure of an image processing device provided by another exemplary embodiment of the present disclosure;

[0036] Fig.14 It is a schematic diagram of the composition structure of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0038] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0039] Application Overview

[0040] The intelligent driving system is an assisted driving system developed based on automation technology. It senses and analyzes the environmental information around the vehicle, makes decisions and controls autonomously, and realizes automatic driving or semi-automatic driving of the intelligent driving vehicle (hereinafter referred to as the vehicle). Accurately identifying traffic signs is the key to the safe and smooth driving of intelligent driving vehicles. At present, intelligent driving vehicles mainly recognize traffic signs through image perception of the driving environment images collected by the on-board camera through neural network models, and assist the vehicle in intelligent driving based on the identified traffic signs.

[0041] Since there are many types of traffic signs, and traffic signs in different regions and scenes vary greatly, it is obviously unrealistic to collect sample data of all types of traffic signs. In traditional technology, sample images of various types of traffic signs are collected as much as possible, labeled, and obtained as training data. Then, the initial image processing model built based on the neural network is trained according to the training data to obtain a trained image processing model. In order to improve the calculation speed, the trained image processing model can be converted into an integer image processing model, and finally the integer image processing model is deployed to the vehicle.

[0042] During the driving process, the on-board camera on the vehicle collects images of the driving environment, and uses the image processing model to identify and process the images of the driving environment to determine their types. When the collected driving environment images include new types of traffic signs, since they are not in the training data, they are classified as the type with the highest probability when classified based on the image processing model, resulting in recognition errors. With the increasing number of new types of traffic, the recognition accuracy of the image processing model is getting lower and lower, affecting the decision-making and control of the intelligent driving system, and thus affecting the safe and stable driving of the vehicle.

[0043] In the prior art, an image processing method is provided to improve the recognition accuracy of an image processing model. Figure 1 is a schematic diagram of a network architecture of an image processing system provided in the prior art, such as Figure 1 As shown, the image processing system 10 includes a terminal device 100 ( Figure 1 , terminal device 100-1 and terminal device 100-2), network 200 and server device 300 are exemplarily shown. The terminal device 100 is connected to the server device 300 through the network 200. The network 200 can be a wide area network or a local area network, or a combination of the two, and a wired or wireless link is used to realize data transmission. The terminal device 100 and the server device 300 can be directly or indirectly connected through wired or wireless communication, which is not limited here.

[0044] The terminal device 100 may be an electronic device arranged on an intelligent driving vehicle, in which an image processing model is deployed. During the driving process of the vehicle, the on-board camera on the vehicle collects images of the surrounding driving environment, transmits them to the image processing model for recognition processing, and obtains a recognition result, which includes a recognition probability and a recognition type. The recognition probability represents the probability that the traffic sign in the currently recognized driving environment image is of the recognized type. The terminal device 100 determines whether the recognition probability is less than a preset value, which is the minimum probability value for determining that the traffic sign is of the recognized type. If the recognition probability is less than the preset value, it is considered that the current traffic sign is not a sign of the trained type, and it is confirmed that it is a new type of traffic sign. The terminal device 100 generates a report message based on the current driving environment image and its recognition result, and transmits it to the server device 300 through the network 200.

[0045] After receiving the report message transmitted by the terminal device 100, the server device 300 needs to collect more images of the same type as the traffic signs in the current driving environment image as new samples, annotate the new samples, modify the algorithm model, and retrain the image processing model to obtain a new image processing model that can recognize the original type and the newly added type of signs. Then convert the newly trained image processing model into an integer image processing model and republish and deploy it to the vehicle, thus completing the update of the image processing model. The updated image processing model can recognize the original type and the newly added type of signs, thereby achieving the effect of improving the recognition accuracy of the image processing model.

[0046] From the above, it can be seen that in order to improve the recognition accuracy of the image processing model in the prior art, when a new type of traffic sign is found, a series of tasks such as mining the new type of data as a new sample, and then labeling, modifying the algorithm model, retraining the neural network model, and releasing an updated version are performed, which consumes a lot of manpower, material resources and financial resources.

[0047] Under the premise of improving the accuracy of image recognition, in order to save manpower, material and financial costs, an image processing method is provided in the embodiment of the present disclosure. A classification model determined based on a registered new type of sign is set on the vehicle. When the vehicle collects a driving environment image, the driving environment image is processed using a pre-deployed image processing model to determine the characteristic information and the first recognition result of the traffic sign, and the characteristic information is processed using a pre-determined classification model to determine the second recognition result of the traffic sign. Finally, based on the first recognition result and the second recognition result, the recognition type with a higher recognition probability is determined as the target recognition result of the traffic sign. Through this method, there is no need for the server device to modify the algorithm model, retrain, or release an updated version of the image processing model. The classification model on the terminal device can realize the recognition of any sign type, thereby achieving the effect of improving the recognition accuracy and saving manpower, material and financial costs.

[0048] Exemplary Systems

[0049] Figure 2 is a schematic diagram of the composition structure of an image processing system provided by an exemplary embodiment of the present disclosure, such as Figure 2 As shown, the image processing system 20 may include a vehicle-mounted camera 21 , a first processor 22 , a second processor 23 and a fusion device 24 .

[0050] Exemplarily, the second processor 23 may be a brain processor (BPU), on which a trained image processing model is deployed. The image processing model is trained by the server device based on training data of a limited number of types of traffic signs, and is deployed in the second processor 23 after integer processing.

[0051] Exemplarily, the first processor 22 may be a central processing unit (CPU), on which a classification model determined based on the registered new type flag is set.

[0052] When the vehicle is traveling, the vehicle-mounted camera 21 collects images of the driving environment around the vehicle in real time.

[0053] The second processor 23 obtains the current driving environment image from the vehicle-mounted camera 21, processes the current driving environment image using the image processing model, and determines the characteristic information of the current traffic sign and the first recognition result. Then the second processor 23 transmits the characteristic information of the current traffic sign to the first processor 22, and transmits the first recognition result to the fusion unit 24.

[0054] The first processor 22 processes the feature information of the current traffic sign using the classification model to determine a second recognition result of the current traffic sign. The first processor 22 then transmits the second recognition result to the fusion unit 24.

[0055] The fusion device 24 determines the target recognition result of the current traffic sign according to the first recognition result and the second recognition result. Exemplarily, the fusion device 24 may include a comparator for comparing the recognition probability in the first recognition result and the recognition probability in the second recognition result, and taking the recognition type with a greater recognition probability as the target recognition result of the current traffic sign.

[0056] In some embodiments, after the fuser 24 obtains the target recognition result of the current traffic sign, it can transmit it to the decision module of the intelligent driving system so that the decision module performs decision control based on the target recognition result of the current traffic sign.

[0057] The image processing system provided by the disclosed embodiment is provided on the vehicle with a classification model determined based on a registered new type of sign. When the vehicle collects a driving environment image, the driving environment image is processed using a pre-deployed image processing model to determine the characteristic information and the first recognition result of the traffic sign, and the characteristic information is processed using the classification model provided thereon to determine the second recognition result of the traffic sign. Finally, based on the first recognition result and the second recognition result, the recognition type with a higher recognition probability is determined as the target recognition result of the traffic sign. The image processing system does not require the server device to modify the algorithm model, retrain, and release an updated version of the image processing model, nor does it require the redeployment of an updated version of the image processing model. The classification model on the first processor can realize the recognition of any sign type, and achieve the effect of saving manpower, material resources and financial costs while ensuring the accuracy of image recognition.

[0058] Exemplary Methods

[0059] Figure 3 FIG. 1 is a flowchart of an image processing method provided by an exemplary embodiment of the present disclosure. The present disclosure embodiment can be applied to electronic devices, such as Figure 3 As shown, the image processing method comprises the following steps:

[0060] Step S301: determine a first driving environment image collected by the intelligent driving vehicle.

[0061] The embodiments of the present disclosure may be executed by an intelligent driving system in an intelligent driving vehicle, and may be specifically performed by Figure 2 The image processing system shown is used to perform.

[0062] During the driving process of the vehicle, the vehicle-mounted camera in the image processing system on the vehicle collects the driving environment image around the vehicle. Exemplarily, the image processing method provided by the embodiment of the present disclosure is described by taking the first driving environment image collected in real time as an example. The first driving environment image is any frame collected by the vehicle-mounted camera and includes a driving environment image of a traffic sign to be identified.

[0063] Step S302: Process the first driving environment image using an image processing model to determine feature information of the first traffic sign and a first recognition result.

[0064] The image processing model is a model pre-deployed in the image processing system, and the image processing model is obtained by the server device based on neural network training.

[0065] The first driving environment image to be processed in this round is input into the image processing model, and segmentation, feature extraction, classification and other processes are performed to obtain feature information and recognition results of traffic signs in the first driving environment image.

[0066] For the sake of distinction, the traffic sign in the first driving environment image is referred to herein as a first traffic sign, and the recognition result obtained by processing the first driving environment image using the image processing model is referred to as a first recognition result.

[0067] The first recognition result may include a first recognition type and a first recognition probability of the first traffic sign determined according to the image processing model, where the first recognition probability represents the possibility that the first traffic sign is of the first recognition type.

[0068] Step S303: Process the feature information of the first traffic sign using the classification model to determine a second recognition result of the first traffic sign.

[0069] The classification model is a model determined by the image processing system based on the newly added types of traffic signs. The newly added types are types that the image processing model cannot recognize or have low recognition accuracy and need to be customized by the user.

[0070] The feature information of the first traffic sign output by the image processing model is input into the classification model for classification processing to obtain a recognition result of the first traffic sign. For the sake of distinction, the recognition result obtained by processing the feature information of the first traffic sign using the classification model is referred to as the second recognition result.

[0071] The second recognition result includes a second recognition type and a second recognition probability of the first traffic sign determined according to the classification model, and the second recognition probability represents the possibility that the first traffic sign is of the second recognition type.

[0072] In the disclosed embodiment, the classification model on the vehicle can realize the processing of new types that cannot be recognized by the image processing model or have low recognition accuracy, thereby realizing the recognition of any sign type, avoiding the impact of the safe and stable driving of the vehicle due to the inability to recognize or the wrong recognition of traffic signs. Through the classification model on the vehicle, the server-side device does not need to modify the algorithm model, retrain, and release an updated version of the image processing model, nor does it need to redeploy the updated version of the image processing model on the vehicle, thereby achieving the effect of saving manpower, material resources and financial costs while ensuring the accuracy of image recognition.

[0073] Step S304: determining a target recognition result of the first traffic sign according to the first recognition result and the second recognition result.

[0074] The first recognition result and the second recognition result of the first traffic sign are compared, and a target recognition result of the first traffic sign is determined according to the comparison result.

[0075] The image processing method provided by the embodiment of the present disclosure first determines the first driving environment image collected by the intelligent driving vehicle, then processes the first driving environment image using the image processing model to determine the characteristic information and the first recognition result of the first traffic sign, and processes the characteristic information of the first traffic sign using the classification model to determine the second recognition result of the first traffic sign; finally, according to the first recognition result and the second recognition result, the target recognition result of the first traffic sign is determined. Through this method, there is no need for the server device to modify the algorithm model, retrain, or release an updated version of the image processing model. The classification model on the terminal device can realize the recognition of any sign type, thereby achieving the effect of improving the recognition accuracy and saving manpower, material and financial costs.

[0076] In some embodiments, before step S301 in the above embodiment, the image processing method provided by the embodiment of the present disclosure may further include: Figure 4 Step S201 and step S202 shown:

[0077] Step S201, determining an image processing model.

[0078] In some embodiments, step S201 “determine the image processing model” may include the following steps S2011 and S2012:

[0079] Step S2011, obtaining the image processing model sent by the server device.

[0080] The image processing model is trained by the server device. In some embodiments, the server device can train the image processing model through the following steps S101 to S107:

[0081] Step S101, obtaining a training data set.

[0082] In the data collection phase, images of various types of traffic signs are collected as much as possible. The training data set is determined based on the collected images.

[0083] In the disclosed embodiment, the training data set includes a plurality of training data, each of which includes a sample image and a true value type corresponding to the sample image. The sample image is a sample image of various types including traffic signs collected as much as possible, and the true value type is the true type of the traffic sign manually marked in the sample image.

[0084] Step S102, constructing an initial image processing model.

[0085] In view of the strict requirements of vehicle-side operation on model scale and forward propagation calculation speed, the initial image processing model is constructed based on a neural network in the disclosed embodiment. The initial image processing model adopts a "slender" stacked network structure, a large number of network layers, and a network architecture with a small number of channels per layer. While maintaining network accuracy, it greatly reduces network parameters and speeds up forward propagation calculation speed.

[0086] Exemplarily, the constructed initial image processing model may include an initial image segmentation sub-model, an initial feature extraction sub-model and an initial classification sub-model.

[0087] The initial image segmentation sub-model is used to perform segmentation processing on the sample image and predict the sample traffic sign area where the sample traffic sign in the sample image is located.

[0088] The initial feature extraction sub-model is used to extract features from the sample traffic sign area and predict the feature information of the sample traffic sign. In one implementation, the feature extraction sub-model can use a variable group convolutional neural network (VarGNet). VarGNet can enhance the discrimination ability of large-scale sign recognition tasks. When constructing it, the first step of the network downsampling process can be deleted to retain more feature information, which helps to improve the recognition effect.

[0089] The classification sub-model is used to identify and classify the characteristic information of the sample traffic sign to obtain the predicted type and predicted probability of the sample traffic sign. In one implementation, metric learning can be used to train the classification sub-model. For example, the face recognition algorithm (arcface) based on deep learning can use the additive angular margin loss function (Additive Angular Margin Loss), keep all class centers during the training process, randomly extract a small number of negative class centers, and randomly extract several dimensions of feature vectors (embedding) to calculate the marginal normalized exponential function (softmax) loss, instead of using all negative class centers and feature vectors of all dimensions in each iteration, thereby greatly increasing the generalization of the classification sub-model and ensuring the accuracy of the recognition results.

[0090] Step S103: Process the sample images in the training data using the image processing model to obtain the prediction type corresponding to the sample images.

[0091] The sample image is input into the image processing model, and image segmentation, feature extraction and classification recognition processing are performed to obtain the prediction type corresponding to the sample image.

[0092] Step S104, determining the error value according to the true value type and the prediction type corresponding to the sample image.

[0093] The true value type is the actual type of the sample traffic sign in the sample image manually labeled, and the predicted type is the type of the sample traffic sign in the sample image predicted by the image processing model.

[0094] Step S105, adjusting the parameters of the initial image processing model based on the error value to obtain an adjusted image processing model.

[0095] According to the error value determined by the true value type and the predicted type corresponding to the sample image, the parameters of the image processing model are adjusted to reduce the error between the true value type and the predicted type.

[0096] Step S106, determining whether the adjusted image processing model meets the training end condition.

[0097] Exemplarily, the training end condition may be that the model accuracy reaches a preset accuracy, or that the number of training times reaches a preset number, or other conditions may be used, which are not limited in the embodiments of the present disclosure.

[0098] If the adjusted image processing model meets the end training condition, the training process ends, the adjusted image processing model that meets the end training condition is determined as the trained image processing model, and the process proceeds to step S107. If the adjusted image processing model does not meet the end training condition, the process returns to step S103 until the adjusted image processing model meets the end training condition.

[0099] Step S107, performing integer processing on the trained image processing model to obtain an integer image processing model.

[0100] In the training phase, to ensure accuracy, the server device can be trained based on floating-point data, and the trained image processing model is a floating-point model. In practical applications, in order to improve the calculation speed, the model size can be reduced through common methods such as pruning, and the floating-point image processing model can be converted to an integer image processing model through common methods such as quantization, and the integer image processing model can be published. The image processing model deployed in the image processing system is an integer image processing model.

[0101] Through common model compression methods such as pruning and quantization, the model size can be reduced and the calculation speed can be increased. In practical applications, when performing model compression, in addition to pruning and quantization, other common model compression methods can also be used, which are not limited in the implementation of this disclosure.

[0102] Step S2012, deploying the image processing model.

[0103] In some embodiments, the image processing system may deploy the image processing model in a second processor. Exemplarily, the second processor may be a BPU.

[0104] Step S202, determining a classification model.

[0105] The classification model may be a model for classifying characteristic information of traffic signs, which may be constructed and updated by the image processing system based on newly added types of signs registered by users. The classification model may be stored in the first processor, which may be a CPU, for example.

[0106] In some embodiments, based on the above embodiments, step S202 "determine the classification model" in the above embodiments can be performed by Figure 5 Steps S2021 to S2024 shown implement:

[0107] Step S2021, determining a second driving environment image collected by the intelligent driving vehicle.

[0108] During the driving process of the vehicle, the vehicle camera in the image processing system on the vehicle collects the image of the vehicle's surrounding environment to obtain a second driving environment image. The second driving environment image is any frame of the driving environment image collected by the vehicle camera that includes the traffic sign to be identified.

[0109] Step S2022: Process the second driving environment image using the image processing model to determine feature information and recognition results of the second traffic sign.

[0110] The second driving environment image to be processed in this round is input into the image processing model, and segmentation, feature extraction, classification and other processes are performed to obtain feature information and recognition results of traffic signs in the second driving environment image.

[0111] For the sake of distinction, the traffic sign in the second driving environment image is referred to as the second traffic sign here. Without considering the influence of factors such as ambient light and image blur, if the type of the second traffic sign is one of the limited types collected when training the image processing model, that is, the type of the second traffic sign is a trained type, the feature of the second traffic sign will have the highest matching rate with the feature corresponding to a certain type among all the trained types, and this type is the type of the second traffic sign. If the type of the second traffic sign is not a trained type, then the feature of the second traffic sign will have a relatively low matching rate with the features corresponding to all the trained types.

[0112] The recognition result of the second traffic sign may include the recognition type and recognition probability of the second traffic sign determined according to the image processing model. The recognition type is the type with the highest matching rate among all trained types. The highest matching rate is the recognition probability, which represents the possibility that the second traffic sign is of the recognition type.

[0113] Step S2023, in response to the recognition result of the second traffic sign including a recognition probability less than a first preset threshold, determining that the second traffic sign is a newly added type of traffic sign, and determining a target type and an initial classification model of the second traffic sign.

[0114] It is determined whether the recognition probability of the second traffic sign is less than a first preset threshold, wherein the first preset threshold represents a minimum probability value that the traffic sign is of a trained type.

[0115] If the recognition probability of the second traffic sign is less than the first preset threshold, it means that the actual type of the second traffic sign is significantly different from the recognized type, and it is considered that the second traffic sign is not a traffic sign of the trained type, and it is determined that the second traffic sign is a traffic sign of a newly added type.

[0116] After determining that the second traffic sign is a newly added type of traffic sign, the target type of the second traffic sign is obtained. In one implementation, the actual type of the second traffic sign can be determined by the marking information input by the user. Specifically, when the recognition probability of the second traffic sign is less than the first preset threshold, a registration interface can be output, in which a second driving environment image and an input box are displayed. The user determines the actual type of the second traffic sign based on the second driving environment image, and enters the marking information in the input box, and the marking information is used to mark the actual type of the second traffic sign.

[0117] The image processing system determines the actual type input by the user on the registration interface as the target type of the second traffic sign, and initializes the classification model to obtain an initial classification model.

[0118] Step S2024: Adjust the initial classification model according to the feature information and target type of the second traffic sign to obtain a classification model.

[0119] The image processing system adjusts the structure or template data of the initial classification model according to the feature information and target type of the second traffic sign to obtain the classification model.

[0120] In the disclosed embodiment, the classification model can be used to identify new types of traffic signs registered by users other than the trained types. For newly added types, the new types of signs can be registered through the marking information input by the user, and the recognition of limited types of signs can be expanded without retraining the image processing model.

[0121] In some embodiments, based on the above embodiments, step S302 in the above embodiments, "processing the first driving environment image using the image processing model to determine the characteristic information of the first traffic sign and the first recognition result", can be performed by Figure 6 The steps S3021 to S3024 shown are implemented as follows:

[0122] Step S3021: Use the image segmentation sub-model in the image processing model to perform image segmentation processing on the first driving environment image to determine the traffic sign area in the first driving environment image.

[0123] The embodiments of the present disclosure may specifically include Figure 2 The second processor in the image processing system shown in the figure is used for execution. Exemplarily, the second processor may be a brain processor BPU. The image processing model is deployed in the BPU. The image processing model may include an image segmentation sub-model, a feature extraction sub-model and a classification sub-model.

[0124] The BPU obtains the first driving environment image captured by the vehicle-mounted camera, first inputs it into the image processing model, uses the segmentation sub-model to perform image segmentation processing on the first driving environment image, and segments the area where the target is located, that is, the traffic sign area, from the first driving environment image.

[0125] Step S3022: Use the feature extraction sub-model in the image processing model to perform feature extraction processing on the traffic sign area to determine feature information of the first traffic sign.

[0126] Then, the feature extraction sub-model is used to perform feature extraction processing on the segmented traffic sign area to obtain feature information of the first traffic sign.

[0127] Among them, the feature extraction sub-model in the image processing model is trained by the server device based on the Variable Group Convolutional Neural Network (VarGNet), which can enhance the discrimination ability of large-scale traffic sign recognition tasks.

[0128] When the server-side device trains the feature extraction sub-model, the downsampling process of the first step of VarGNet can be deleted for training. This will retain more feature information during feature extraction and help improve the recognition effect.

[0129] Step S3023: Use the classification sub-model in the image processing model to perform recognition and classification processing on the feature information of the first traffic sign to determine the recognition type and recognition probability of the first traffic sign.

[0130] Finally, the classification sub-model is used to perform recognition and classification processing on the feature information of the first traffic sign to obtain the recognition type and recognition probability of the first traffic sign.

[0131] The classification sub-model in the image processing model is trained by the server-side device based on a deep learning algorithm (such as the face recognition algorithm ArcFace). During the training process, all class centers are kept, a small number of negative class centers are randomly extracted, and several dimensions of the feature vector (embedding) are randomly extracted to calculate the marginal-based normalized exponential function (softmax) loss, instead of using all negative class centers and feature vectors of all dimensions in each iteration, thereby greatly increasing the generalization of the classification sub-model and ensuring the accuracy of the recognition results.

[0132] Step S3024: determining a first recognition result of the first traffic sign according to the recognition type and recognition probability of the first traffic sign.

[0133] The recognition type and recognition probability of the first traffic sign are used as a first recognition result.

[0134] In the disclosed embodiment, the first traffic sign is segmented, feature extracted and classified by an image processing model trained by a server device and deployed in an image processing system to obtain a first recognition result of the first traffic sign, thereby providing data support for determining a target recognition result of the first traffic sign.

[0135] In some embodiments, based on the above embodiments, step S304 in the above embodiments, “determining a target recognition result of the first traffic sign according to the first recognition result and the second recognition result”, may include: Figure 7 Step S3041 and step S3042 shown:

[0136] Step S3041, in response to the recognition probability included in the first recognition result being less than the first preset threshold and the recognition probability included in the second recognition result being less than the second preset threshold, determining that the first traffic sign is a newly added type of traffic sign, and determining the target type of the first traffic sign.

[0137] After obtaining the first recognition result and the second recognition result of the first traffic sign, compare the recognition probability in the first recognition result (i.e., the first recognition probability) with the first preset threshold, and compare the recognition probability in the second recognition result (i.e., the second recognition probability) with the second preset threshold. The first preset threshold represents the minimum probability value that the traffic sign is of the trained type, and the second preset threshold represents the minimum probability value that the second traffic sign is of the registered type. The first preset threshold can be an appropriate value between 0 and 1 set when training the image processing model, and the second preset threshold can be an appropriate value between 0 and 1 set when building the classification model. The first preset threshold and the second preset threshold can be equal or unequal.

[0138] When the first recognition probability is less than the first preset threshold and the second recognition probability is less than the second preset threshold, it means that the first traffic sign is neither a trained type nor a registered type, and it is determined that the first traffic sign is a newly added type of traffic sign.

[0139] After determining that the first traffic sign is a newly added type of traffic sign, the target type of the first traffic sign is obtained. In one implementation, the target type of the second traffic sign can be determined by marking information input by the user. Exemplarily, a registration interface displaying a first driving environment image can be output, and the user observes the first traffic sign in the first driving environment image, determines and inputs marking information of the first traffic sign in the registration interface, and the marking information is used to mark the target type of the first traffic sign.

[0140] Step S3042: determining the target recognition result of the first traffic sign according to the target type of the first traffic sign.

[0141] In some embodiments, the target type of the first traffic sign can be transmitted as a target recognition result to a post-processing module of the intelligent driving system, such as a decision module, so that the decision module performs decision control based on the target type of the first traffic sign.

[0142] In some embodiments, in addition to the target type, if other information (such as characteristic information of the first traffic sign, recognition probability, etc.) needs to be transmitted to the post-processing module of the intelligent driving system, the target recognition result of the first traffic sign may include the target type and other information.

[0143] In the disclosed embodiment, when it is determined that the first traffic sign is neither a trained type nor a registered type according to the first recognition probability and the second recognition probability, the newly added type of sign can be registered through the marking information input by the user, and the recognition of limited types of signs can be expanded without retraining the image processing model. In addition, the newly added type of sign registered can be any untrained type of sign, and any type of sign can be recognized.

[0144] In some embodiments, based on the above embodiments, after step S3041 in the above embodiments, the image processing method provided by the embodiments of the present disclosure may further include step S3411: updating the classification model according to the feature information and target type of the first traffic sign to obtain an updated classification model.

[0145] Each time a new type of mark is added, the types that the classification model can recognize can be updated so that the classification model can recognize and classify the newly added type of mark. In this way, when the newly added type of mark needs to be recognized again in the future, it can be recognized and processed according to the updated classification model. In this way, the types that the classification model can recognize are becoming more and more abundant, making the classification model more and more widely used.

[0146] In one implementation, the classification model can be implemented as a comparator. The characteristic information, sign type, and the correspondence between the characteristic information and sign type of all registered signs are stored in a data table as template data. When it is necessary to determine the second recognition result of a certain traffic sign, the similarity between the characteristic information of the traffic sign and the characteristic information of each registered sign in the template data is determined by a comparator, and a registered sign with the highest similarity is selected as the target registered sign. The type of the target registered sign is determined as the second recognition type of the traffic sign, and the similarity between the characteristic information of the traffic sign and the characteristic information of the target registered sign is determined as the second recognition probability of the traffic sign. For each new type of traffic sign, the characteristic information, sign type, and the correspondence between the characteristic information and sign type of a sign are correspondingly stored to update the template data.

[0147] In another implementation, the classification model can be implemented as a simple neural network model, including an input layer, a hidden layer, and an output layer. The input layer is used to receive the input feature information of the traffic sign, the hidden layer is located between the input layer and the output layer, and can be one or more layers, and is used to identify and classify the feature information, and the output layer is used to output the second recognition result of the traffic sign. Among them, the hidden layer can include multiple neurons, and one neuron corresponds to the matching processing operation of one type of traffic sign. For each new type of traffic sign, the number of neurons increases by one.

[0148] In practical applications, the classification model may also be implemented in other ways, which are not limited in the embodiments of the present disclosure.

[0149] In some embodiments, based on the above embodiments, step S304 in the above embodiments, “determining a target recognition result of the first traffic sign according to the first recognition result and the second recognition result”, may include: Figure 8 Step S3043 and step S3044 shown:

[0150] Step S3043, in response to the recognition probability included in the first recognition result being less than the first preset threshold, and the recognition probability included in the second recognition result being greater than or equal to the second preset threshold, determine the target recognition result of the first traffic sign according to the recognition type included in the second recognition result.

[0151] When the recognition probability in the first recognition result (i.e., the first recognition probability) is less than the first preset threshold, and the recognition probability in the second recognition result (i.e., the second recognition probability) is greater than or equal to the second preset threshold, it means that the first traffic sign is not a trained type but a registered type, and the recognition type in the second recognition result is determined as the target recognition result of the first traffic sign.

[0152] Step S3044, in response to the recognition probability included in the first recognition result being greater than or equal to the first preset threshold, and the recognition probability included in the second recognition result being less than the second preset threshold, determine the target recognition result of the first traffic sign according to the recognition type included in the first recognition result.

[0153] When the first recognition probability is greater than or equal to the first preset threshold and the second recognition probability is less than the second preset threshold, it means that the first traffic sign is a trained type but not a registered type, and the recognition type in the first recognition result is determined as the target recognition result of the first traffic sign.

[0154] In some embodiments, based on the above embodiments, step S304 in the above embodiments, “determining a target recognition result of the first traffic sign according to the first recognition result and the second recognition result”, may include: Fig. 9Steps S3045 to S3047 shown:

[0155] Step S3045, in response to the recognition probability included in the first recognition result being greater than or equal to the first preset threshold, and the recognition probability included in the second recognition result being greater than or equal to the second preset threshold, determine the difference between the recognition probability included in the first recognition result and the recognition probability included in the second recognition result.

[0156] When the first recognition probability is greater than or equal to the first preset threshold, and the second recognition probability is also greater than or equal to the second preset threshold, it means that the first traffic sign may be a trained type or a registered type. At this time, the difference between the first recognition probability and the second recognition probability is calculated, and the type of the first traffic sign is determined to be more likely to be a trained type or a registered type based on the difference.

[0157] It is determined whether the difference between the first recognition probability and the second recognition probability is greater than a third preset threshold value. If the difference is greater than the third preset threshold value, the process proceeds to step S3046; if the difference is less than or equal to the third preset threshold value, the process proceeds to step S3047. The third preset threshold value may be 0 or other values, which are not limited in the embodiments of the present disclosure.

[0158] Step S3046: In response to the difference being greater than a third preset threshold, determining a target recognition result of the first traffic sign according to the recognition type included in the first recognition result.

[0159] Taking the third preset threshold as 0 as an example, when the difference is greater than the third preset threshold, that is, the first recognition probability is greater than the second recognition probability, it means that the probability that the first traffic sign is a trained type is greater. At this time, the recognition type in the first recognition result is determined as the target recognition result of the first traffic sign.

[0160] Step S3047, in response to the difference being less than or equal to the third preset threshold, determining the target recognition result of the first traffic sign according to the recognition type included in the second recognition result.

[0161] When the difference is less than or equal to the third preset threshold, that is, the first recognition probability is less than or equal to the second recognition probability, it means that the probability that the first traffic sign is a registered type is greater. At this time, the recognition type in the second recognition result is determined as the target recognition result of the first traffic sign.

[0162] In the disclosed embodiment, by comparing the recognition probability in the first recognition result with the first preset threshold, comparing the recognition probability in the second recognition result with the second preset threshold, and comparing the recognition probability in the first recognition result with the recognition probability in the second recognition result, the target recognition result of the first traffic sign is determined according to the comparison result.

[0163] Based on the above embodiments, the present disclosure further provides an image processing method. Fig.10 is a flowchart of an image processing method provided by another exemplary embodiment of the present disclosure, such as Fig.10 As shown, the method comprises the following steps:

[0164] Step S1001, obtaining the image processing model trained by the server device.

[0165] The image processing model may include an image segmentation sub-model, a feature extraction sub-model and a classification sub-model.

[0166] During the training phase, the server device collects as many sample images of various types of traffic signs as possible, labels them, and obtains training data. Then, the initial image processing model built based on the neural network is trained based on the training data to obtain a trained image processing model.

[0167] In the training phase, to ensure accuracy, the server device can be trained based on floating-point data, and the trained image processing model is floating-point. In practical applications, in order to improve the calculation speed, the server device can convert the floating-point image processing model into an integer image processing model and publish the integer image processing model. The image processing model deployed in the image processing system is an integer model.

[0168] Step S1002: deploy the image processing model in the image processing system.

[0169] Exemplarily, the image processing model can be deployed in a second processor (such as a BPU) in the image processing system that is dedicated to processing image data, which helps to improve the processing speed.

[0170] Step S1003, initializing the classification model.

[0171] When no new type of sign has been registered, the first processor (such as a CPU) in the image processing system initializes the classification model. When the image processing model subsequently recognizes a new type of traffic sign, the initialized classification model is adjusted.

[0172] Step S1004, obtaining the current driving environment image acquired by the image acquisition module in the image processing system.

[0173] The image acquisition module is the vehicle-mounted camera mentioned above. During the driving process of the vehicle, the vehicle-mounted camera collects the environment around the vehicle in real time and transmits each frame of the driving environment image collected to the BPU for processing.

[0174] Step S1005: Process the current driving environment image using the image processing model to obtain feature information of the current traffic sign and a first recognition result.

[0175] The first recognition result includes a first recognition type and a first recognition probability. The first recognition type is a trained type, and among all trained types, the feature corresponding to the first recognition type has the highest matching rate with the feature of the current traffic sign, and the highest matching rate is the first recognition probability.

[0176] Step S1006: Process the feature information of the current traffic sign using the classification model to obtain a second recognition result of the current traffic sign.

[0177] The second recognition result includes a second recognition type and a second recognition probability.

[0178] If the classification model is an initialized model that has not been adjusted, it means that no new type of flag has been registered. In this case, the second recognition type is null, and the second recognition probability is any value calculated based on the initialized classification model.

[0179] If the classification model is not an initialized model, it means that at least one new type of sign has been registered. At this time, the second recognition type is a registered type. Among all registered types, the feature corresponding to the second recognition type has the highest matching rate with the feature of the current traffic sign, and the highest matching rate is the second recognition probability.

[0180] Step S1007, determining whether the first recognition probability is greater than a first threshold.

[0181] When the first recognition probability is greater than the first threshold, it is considered that the actual type of the current traffic sign may be the first recognition type in the trained types, and the process goes to step S1008; when the first recognition probability is less than or equal to the first threshold, it is considered that the actual type of the current traffic sign is not the first recognition type in the trained types, and the process goes to step S1012.

[0182] Step S1008, determining whether the second recognition probability is greater than a second threshold.

[0183] If the second recognition probability is greater than the second threshold, it is considered that the actual type of the current traffic sign may be the second recognition type among the registered types, and step S1009 is entered for further judgment; when the second recognition probability is less than or equal to the second threshold, it is considered that the actual type of the current traffic sign is indeed the first recognition type among the trained types, and step S1010 is entered.

[0184] Step S1009, determining whether the first recognition probability is greater than the second recognition probability.

[0185] If the first recognition probability is greater than the first threshold and the second recognition probability is greater than the second threshold, it means that the actual type of the current traffic sign may be the first recognition type in the trained type or the second recognition type in the registered type. At this time, further judgment is required based on the first recognition probability and the second recognition probability to determine whether the actual type of the current traffic sign is most likely the first recognition type or the second recognition type.

[0186] Exemplarily, the first recognition probability and the second recognition probability are compared. When the first recognition probability is greater than the second recognition probability, it means that the probability that the actual type of the current traffic sign is the first recognition type is greater than the probability of the second recognition type. The type of the current traffic sign is determined to be the first recognition type in the trained types, and the process goes to step S1010.

[0187] When the first recognition probability is less than or equal to the second recognition probability, it means that the probability that the actual type of the current traffic sign is the first recognition type is less than or equal to the probability of the second recognition type, and the type of the current traffic sign is determined to be the second recognition type among the registered types, and the process goes to step S1011.

[0188] Step S1010: determining a target recognition result of the current traffic sign according to the first recognition result.

[0189] The first recognition type in the first recognition result is used as the target recognition result of the current traffic sign.

[0190] Step S1011, determining a target recognition result of the current traffic sign according to the second recognition result.

[0191] The second recognition type in the second recognition result is used as the target recognition result of the current traffic sign.

[0192] Step S1012, determining whether the second recognition probability is greater than a second threshold.

[0193] If the first recognition probability is less than or equal to the first threshold, and it is determined that the actual type of the current traffic sign is not the first recognition type in the trained types, it is further determined whether the current traffic sign is a newly added type of sign based on the second recognition probability.

[0194] Specifically, the second recognition probability can be compared with the second threshold. If the second recognition probability is greater than the second threshold, it is considered that the actual type of the current traffic sign is a registered type, and step S1011 is entered; if the second recognition probability is less than or equal to the second threshold, it is considered that the actual type of the current traffic sign is a newly added type, and step S1013 is entered to register the new type.

[0195] Step S1013, determining the target type of the current traffic sign.

[0196] In one implementation, the actual type of the current traffic sign can be determined by the marking information input by the user. When the current traffic sign is determined to be a newly added type of sign according to the above steps, the image processing system can output a registration interface, which displays the current driving environment image and an input box. The user determines the actual type of the current traffic sign based on the current driving environment image and enters the marking information in the input box, and the marking information is used to mark the type of the current traffic sign. The type entered by the user is the target type of the current traffic sign.

[0197] Step S1014, determining the target recognition result of the current traffic sign according to the target type of the current traffic sign.

[0198] The target type of the current traffic sign is determined as a target recognition result of the current traffic sign.

[0199] Step S1015, updating the classification model.

[0200] After registering the new type of the current traffic sign, the classification model can be updated so that the classification model can recognize and classify the newly added type of sign. In this way, when the current traffic sign included in other images needs to be recognized again in the future, the recognition process can be performed according to the updated classification model, so that the classification model can recognize more and more types.

[0201] After step S1015, the process returns to step S1004 to continue collecting the next driving environment image.

[0202] The method provided by the embodiment of the present disclosure does not require the server-side device to modify the algorithm model, retrain, release an updated version of the image processing model, or redeploy an updated version of the image processing model. The classification model on the first processor can realize the recognition of any type of sign, while ensuring the accuracy of image recognition, achieving the effect of saving manpower, material resources and financial costs.

[0203] Exemplary Devices

[0204] Fig.11 is a schematic diagram of the composition structure of an image processing device provided by an exemplary embodiment of the present disclosure, such as Fig.11 As shown, the image processing device 1100 may include:

[0205] A first determining module 1101 is used to determine a first driving environment image collected by the intelligent driving vehicle;

[0206] A first processing module 1102, configured to process the first driving environment image using an image processing model to determine feature information of a first traffic sign and a first recognition result;

[0207] A second processing module 1103 is used to process the feature information of the first traffic sign using a classification model to determine a second recognition result of the first traffic sign;

[0208] The second determination module 1104 is configured to determine a target recognition result of the first traffic sign according to the first recognition result and the second recognition result.

[0209] See also Fig.12 In some embodiments, the image processing device 1100 may further include:

[0210] The third determination module 1201 is used to determine the image processing model;

[0211] The fourth determination module 1202 is used to determine a classification model.

[0212] See also Fig.13 In some embodiments, the third determining module 1201 may include:

[0213] An acquisition unit 12011 is used to acquire an image processing model sent by a server device;

[0214] The deployment unit 12011 is used to deploy the image processing model.

[0215] See also Fig.13 In some embodiments, the fourth determining module 1202 may include:

[0216] The first determining unit 12021 is used to determine the second driving environment image collected by the intelligent driving vehicle;

[0217] The first processing unit 12022 is used to process the second driving environment image using the image processing model to determine feature information and recognition results of the second traffic sign;

[0218] A second determination unit 12023 is configured to determine, in response to the recognition result of the second traffic sign including a recognition probability less than a first preset threshold, that the second traffic sign is a newly added type of traffic sign, and determine a target type and an initial classification model of the second traffic sign;

[0219] The adjustment unit 12024 is used to adjust the initial classification model according to the feature information and target type of the second traffic sign to obtain a classification model.

[0220] See also Fig.13 In some embodiments, the first processing module 1102 may include:

[0221] The second processing unit 11021 is used to perform image segmentation processing on the first driving environment image by using the image segmentation sub-model in the image processing model to determine the traffic sign area in the first driving environment image;

[0222] The third processing unit 11022 is used to perform feature extraction processing on the traffic sign area by using the feature extraction submodel in the image processing model to determine feature information of the first traffic sign;

[0223] The fourth processing unit 11023 is used to perform recognition and classification processing on the feature information of the first traffic sign by using the classification sub-model in the image processing model to determine the recognition type and recognition probability of the first traffic sign;

[0224] The third determination unit 11024 is configured to determine a first recognition result of the first traffic sign according to the recognition type and recognition probability of the first traffic sign.

[0225] See also Fig.13 In some embodiments, the second determining module 1104 may include:

[0226] a fourth determining unit 11041, configured to determine, in response to the recognition probability included in the first recognition result being less than a first preset threshold and the recognition probability included in the second recognition result being less than a second preset threshold, that the first traffic sign is a newly added type of traffic sign, and determine a target type of the first traffic sign;

[0227] The fifth determining unit 11042 is configured to determine a target recognition result of the first traffic sign according to the target type of the first traffic sign.

[0228] In some embodiments, the adjustment unit 12024 is further used to: update the classification model according to the feature information and target type of the first traffic sign to obtain an updated classification model.

[0229] See also Fig.13 In some embodiments, the second determining module 1104 may further include:

[0230] a sixth determining unit 11043, configured to determine, in response to the recognition probability included in the first recognition result being less than a first preset threshold and the recognition probability included in the second recognition result being greater than or equal to a second preset threshold, a target recognition result of the first traffic sign according to the recognition type included in the second recognition result;

[0231] The seventh determination unit 11044 is used to determine the target recognition result of the first traffic sign according to the recognition type included in the first recognition result in response to the recognition probability included in the first recognition result being greater than or equal to the first preset threshold, and the recognition probability included in the second recognition result being less than the second preset threshold.

[0232] See also Fig.13 In some embodiments, the second determining module 1104 may further include:

[0233] an eighth determining unit 11045, configured to determine a difference between the recognition probability included in the first recognition result and the recognition probability included in the second recognition result in response to the recognition probability included in the first recognition result being greater than or equal to a first preset threshold, and the recognition probability included in the second recognition result being greater than or equal to a second preset threshold;

[0234] a ninth determining unit 11046, configured to determine, in response to the difference being greater than a third preset threshold, a target recognition result of the first traffic sign according to the recognition type included in the first recognition result;

[0235] The tenth determining unit 11047 is configured to determine, in response to the difference being less than or equal to the third preset threshold, a target recognition result of the first traffic sign according to the recognition type included in the second recognition result.

[0236] The beneficial technical effects corresponding to the exemplary embodiments of the device disclosed herein can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.

[0237] Exemplary Electronic Devices

[0238] Fig.14 is a schematic diagram of the composition structure of an electronic device provided by an exemplary embodiment of the present disclosure, such as Fig.14 As shown, the electronic device 1400 includes at least one processor 1401 and a memory 1402 .

[0239] The processor 1401 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1400 to perform desired functions.

[0240] The memory 1402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1401 may execute one or more computer program instructions to implement the image processing method and / or other desired functions of the various embodiments of the present disclosure described above.

[0241] In one example, the electronic device 1400 may further include: an input device 1403 and an output device 1404 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0242] The input device 1403 may also include, for example, a keyboard, a mouse, etc.

[0243] The output device 1404 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0244] Of course, to simplify, Fig.14 Only some of the components related to the present disclosure in the electronic device 1400 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 1400 may also include any other appropriate components.

[0245] Exemplary computer program products and computer-readable storage media

[0246] In addition to the above-mentioned methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the image processing method of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.

[0247] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0248] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the image processing method of various embodiments of the present disclosure described in the above “Exemplary Method” section.

[0249] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0250] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0251] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. An image processing method, include: Determine a first driving environment image collected by the intelligent driving vehicle; Processing the first driving environment image using an image processing model to determine feature information of a first traffic sign and a first recognition result; Processing the feature information of the first traffic sign using a classification model to determine a second recognition result of the first traffic sign; A target recognition result of the first traffic sign is determined according to the first recognition result and the second recognition result.

2. The method according to claim 1, further comprising: include: Determining a second driving environment image collected by the intelligent driving vehicle; Processing the second driving environment image using the image processing model to determine feature information and a recognition result of a second traffic sign; In response to the recognition result of the second traffic sign including a recognition probability less than a first preset threshold, determining that the second traffic sign is a newly added type of traffic sign, and determining a target type and an initial classification model of the second traffic sign; The initial classification model is adjusted according to the characteristic information and target type of the second traffic sign to obtain a classification model.

3. The method according to claim 1, in, The using the image processing model to process the first driving environment image to determine the characteristic information of the first traffic sign and the first recognition result includes: Performing image segmentation processing on the first driving environment image using the image segmentation sub-model in the image processing model to determine a traffic sign area in the first driving environment image; Performing feature extraction processing on the traffic sign area using a feature extraction submodel in the image processing model to determine feature information of the first traffic sign; Using the classification sub-model in the image processing model to perform recognition and classification processing on the feature information of the first traffic sign, and determine the recognition type and recognition probability of the first traffic sign; A first recognition result of the first traffic sign is determined according to the recognition type and the recognition probability of the first traffic sign.

4. The method according to any one of claims 1 to 3, in, The step of determining a target recognition result of the first traffic sign according to the first recognition result and the second recognition result includes: In response to the recognition probability included in the first recognition result being less than a first preset threshold, and the recognition probability included in the second recognition result being less than a second preset threshold, determining that the first traffic sign is a newly added type of traffic sign, and determining a target type of the first traffic sign; An object recognition result of the first traffic sign is determined according to the object type of the first traffic sign.

5. The method according to claim 4, further comprising: include: The classification model is updated according to the feature information and target type of the first traffic sign to obtain an updated classification model.

6. The method according to any one of claims 1 to 3, in, The step of determining a target recognition result of the first traffic sign according to the first recognition result and the second recognition result includes: In response to the recognition probability included in the first recognition result being less than a first preset threshold, and the recognition probability included in the second recognition result being greater than or equal to a second preset threshold, determining a target recognition result of the first traffic sign according to the recognition type included in the second recognition result; In response to the recognition probability included in the first recognition result being greater than or equal to a first preset threshold, and the recognition probability included in the second recognition result being less than a second preset threshold, a target recognition result of the first traffic sign is determined according to the recognition type included in the first recognition result.

7. The method according to any one of claims 1 to 3, in, The step of determining a target recognition result of the first traffic sign according to the first recognition result and the second recognition result includes: In response to the recognition probability included in the first recognition result being greater than or equal to a first preset threshold, and the recognition probability included in the second recognition result being greater than or equal to a second preset threshold, determining a difference between the recognition probability included in the first recognition result and the recognition probability included in the second recognition result; In response to the difference being greater than a third preset threshold, determining a target recognition result of the first traffic sign according to the recognition type included in the first recognition result; In response to the difference being less than or equal to the third preset threshold, a target recognition result of the first traffic sign is determined according to the recognition type included in the second recognition result.

8. An image processing device, include: A first determining module, used to determine a first driving environment image collected by the intelligent driving vehicle; A first processing module, configured to process the first driving environment image using an image processing model to determine feature information of a first traffic sign and a first recognition result; a second processing module, configured to process the feature information of the first traffic sign using a classification model to determine a second recognition result of the first traffic sign; The second determination module is used to determine a target recognition result of the first traffic sign according to the first recognition result and the second recognition result.

9. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the image processing method according to any one of claims 1 to 7.

10. An electronic device, wherein the electronic device include: processor; a memory for storing instructions executable by the processor; The processor is used to read the instructions from the memory and execute the instructions to implement the image processing method described in any one of claims 1-7.

Images