Model training method, object recognition method, device and electronic equipment
The technical solution addresses the technical problem of enhancing the accuracy of the object recognition model by integrating a target subnetwork and the second predicts the clarity of the object and the second predicts the number of recognizable characters, the efficacy of the object recognition model by integrating the object clarity and character count dimensions for supervised training.
Patent Information
- Application Number
- CN202411277658.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-12
AI Technical Summary
Existing object recognition models suffer from low accuracy when trained with unclear or low-quality images due to the introduction of noise from subjective calibration content, leading to poor recognition and classification of such images.
A model training method that utilizes a multi-task learning approach with a global encoder, a target subnetwork, and two auxiliary subnetworks to enhance the object recognition model, where the first predicts the clarity of the object and the second predicts the number of characters, the second predicts the number of recognizable and the second subnetworks to enhance the object recognition task, incorporating object clarity and character count dimensions for supervised training.
Improves the accuracy of object recognition models by integrating object clarity and character count dimensions, enhancing the model's ability to handle low-quality images and reducing errors in recognition tasks.
Smart Images

Figure CN118781471B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of deep learning, and in particular to a model training method, an object recognition method, a device and an electronic device. Background Art
[0002] With the development of science and technology, the technology of object recognition for target objects with fixed format and limited characters such as license plates and certificates in images is becoming more and more advanced based on object recognition models. Among them, the so-called object recognition can at least include recognizing the character string in the object, for example, for a license plate image, recognizing the license plate number of the license plate in the license plate image, and for a certificate image, recognizing the certificate number of the certificate in the certificate image.
[0003] In the related technology, for the object recognition model, during the training process, the image features of the sample images in the sample image set can be generated, and then the image features of the sample images can be recognized and analyzed to obtain the prediction results for the object recognition task, and the model can be trained based on the prediction results and the calibration content of the sample images that matches the object recognition task.
[0004] Affected by motion, lighting, imaging, etc., the character string in the sample image may not be clear, that is, there are characters that are not recognizable to the human eye. Therefore, for sample images where the character string may not be clear, calibration is difficult and the calibration content is highly subjective, and incorrect calibration content will undoubtedly introduce noise to model training.
[0005] However, when conducting model training, the related technology only relies on the calibration content of the sample image that matches the object recognition task for supervision, which undoubtedly leads to low accuracy of the trained object recognition model. Summary of the invention
[0006] The purpose of the embodiments of the present application is to provide a model training method, an object recognition method, a device and an electronic device to improve the accuracy of the object recognition model. The specific technical solution is as follows:
[0007] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:
[0008] Acquire a sample image set; wherein each sample image in the sample image set is an image containing objects of the same type;
[0009] Input each sample image in the sample image set into a preset object recognition model to be trained for performing an object recognition task, so that the global encoder in the preset object recognition model performs feature encoding on each sample image to obtain the initial image feature of each sample image, and input the initial image feature of each sample image into the target subnetwork, the first subnetwork and the second subnetwork in the preset object recognition model respectively, to obtain the output result of the target subnetwork for each sample image, the output result of the first subnetwork for each sample image and the output result of the second subnetwork for each sample image; wherein the first subnetwork is used to generate a first auxiliary feature for characterizing the object clarity of each sample image, the second subnetwork is used to generate a second auxiliary feature for characterizing the number of characters of each sample image, and based on the second auxiliary feature of each sample image, respectively predict the number of recognizable characters and the number of unrecognizable characters of each sample image, and the target subnetwork is used to predict the result of the object recognition task based on the initial image feature, the first auxiliary feature and the second auxiliary feature of each sample image;
[0010] For each of the target subnetwork, the first subnetwork, and the second subnetwork, based on an output result of the subnetwork for each sample image and a calibration true value of each sample image that matches the subnetwork, determine a network loss of the subnetwork;
[0011] Determining a model loss of the preset object recognition model based on the obtained network loss of each sub-network;
[0012] When it is determined based on the model loss that the preset object recognition model has not converged, the network parameters of the preset object recognition model are adjusted, and the process returns to the step of obtaining the sample image set.
[0013] Optionally, the object recognition task includes: a first subtask for recognizing a character string and a second subtask for recognizing an object type of a specified dimension;
[0014] The target subnetwork is used to perform feature fusion on the initial image features and the first auxiliary features of each sample image, and perform feature encoding on the fused image features to obtain target image features of each sample image;
[0015] Based on the target image features of each sample image and the second auxiliary features of each sample image, predict the recognition result of the first subtask and obtain the output result corresponding to the first subtask; and, based on the target image features of each sample image, predict the recognition result of the second subtask and obtain the output result corresponding to the second subtask.
[0016] Optionally, the target subnetwork includes: a target feature encoder, a first classifier, and a second classifier;
[0017] The target feature encoder is used to perform feature fusion on the initial image feature and the first auxiliary feature of each sample image, and perform feature encoding on the fused image feature to obtain the target image feature of each sample image;
[0018] The first classifier is used to predict the recognition result of the first subtask based on the target image feature of each sample image and the second auxiliary feature of each sample image, and obtain the output result corresponding to the first subtask;
[0019] The second classifier is used to predict the recognition result of the second subtask based on the target image features of each sample image, and obtain the output result corresponding to the second subtask.
[0020] Optionally, the first sub-network includes: a first feature encoder; the second sub-network includes: a second feature encoder, a third classifier and a fourth classifier;
[0021] The first feature encoder is used to generate a first auxiliary feature of each sample image based on the initial image feature of each sample image;
[0022] The second feature encoder is used to perform feature fusion on the initial image feature of each sample image and the first auxiliary feature of each sample image, and perform feature encoding on the fused auxiliary feature to obtain the second auxiliary feature of each sample image;
[0023] The third classifier is used to predict the first number of recognizable characters for each sample image based on the second auxiliary feature of each sample image;
[0024] The fourth classifier is used to predict the number of unrecognizable characters for each sample image based on the second auxiliary feature of each sample image.
[0025] Optionally, determining the model loss of the preset object recognition model based on the obtained network loss of each sub-network includes:
[0026] The network loss of each sub-network obtained is subjected to a specified operation process to obtain the model loss of the preset object recognition model.
[0027] Optionally, the calibration truth value of each sample image matched with the first sub-network is: a specified category label used to characterize the clarity of the object;
[0028] The determining of the network loss of the sub-network based on the output result of the sub-network for each sample image and the calibration truth value of each sample image that matches the sub-network includes:
[0029] In the case where the sub-network is the first sub-network, based on the first auxiliary feature of each sample image and the designated category label of each sample image, and using a predetermined contrast loss function, determining the network loss of the sub-network;
[0030] The predetermined contrast loss function is a function for calculating the loss based on the difference between sample images with the same specified category label regarding the first auxiliary feature, and the difference between sample images with different specified category labels regarding the first auxiliary feature.
[0031] Optionally, the number of recognizable characters and the number of unrecognizable characters of each sample image predicted by the second sub-network are both corresponding to confidence levels;
[0032] The determining the network loss of the sub-network based on the first auxiliary feature of each sample image and the designated category label of each sample image and using a predetermined contrast loss function includes:
[0033] Determine the network loss of the sub-network based on the first auxiliary feature of each sample image, the designated category label of each sample image, the first confidence and the second confidence corresponding to each sample image, and using a predetermined contrast loss function;
[0034] The first confidence level corresponding to each sample image is the confidence level corresponding to the number of recognizable characters, and the second confidence level corresponding to each sample image is the confidence level corresponding to the number of unrecognizable characters.
[0035] Optionally, determining the network loss of the sub-network based on the first auxiliary feature of each sample image, the designated category label of each sample image, the first confidence and the second confidence corresponding to each sample image, and using a predetermined contrast loss function includes:
[0036] For each sample image, based on the first auxiliary feature of the sample image, the first auxiliary feature of the sample image having the same designated category label as the sample image, and the first auxiliary feature of the sample image having a different designated category label from the sample image, an initial loss corresponding to the sample image is calculated according to a predetermined contrast loss function; and based on the first confidence and the second confidence corresponding to the sample image, a weight value is calculated according to a weight calculation method that matches the designated category label of the sample image; the initial loss corresponding to the sample image is multiplied by the calculated weight value to obtain a target loss of the sample image;
[0037] Add the target losses of each sample image to get the network loss of the sub-network;
[0038] The weight calculation method that matches the specified category label of the sample image includes:
[0039] If the designated category label of the sample label is the label with the highest degree of clarity, the ratio of the first confidence level corresponding to the sample image to the target value corresponding to the sample image is calculated as the weight; wherein the target value of the sample image is the value obtained by summing the first confidence level and the second confidence level corresponding to the sample image;
[0040] Otherwise, the ratio of the second confidence level corresponding to the sample image to the target value is calculated as the weight.
[0041] Optionally, the second auxiliary feature of each sample image is a feature map having a specified number of channels; the specified number is the maximum number of predicted characters.
[0042] In a second aspect, an embodiment of the present application provides an object recognition method, the method comprising:
[0043] Get the target image;
[0044] The target image is input into a preset object recognition model for performing an object recognition task to obtain an object recognition result of the target image; wherein the preset object recognition model is trained based on the model training method.
[0045] In a third aspect, an embodiment of the present application provides a model training device, the device comprising:
[0046] A first acquisition module is used to acquire a sample image set; wherein each sample image in the sample image set is an image containing objects of the same type;
[0047] A first input module is used to input each sample image in the sample image set into a preset object recognition model to be trained for performing an object recognition task, so that the global encoder in the preset object recognition model performs feature encoding on each sample image to obtain the initial image feature of each sample image, and the initial image feature of each sample image is respectively input into the target sub-network, the first sub-network and the second sub-network in the preset object recognition model to obtain the output result of the target sub-network for each sample image, the output result of the first sub-network for each sample image and the output result of the second sub-network for each sample image; wherein the first sub-network is used to generate a first auxiliary feature for characterizing the object clarity of each sample image, the second sub-network is used to generate a second auxiliary feature for characterizing the number of characters of each sample image, and based on the second auxiliary feature of each sample image, respectively predict the number of recognizable characters and the number of unrecognizable characters of each sample image, and the target sub-network is used to predict the result of the object recognition task based on the initial image feature, the first auxiliary feature and the second auxiliary feature of each sample image;
[0048] A first determination module is used to determine, for each sub-network among the target sub-network, the first sub-network and the second sub-network, a network loss of the sub-network based on an output result of the sub-network for each sample image and a calibration true value of each sample image that matches the sub-network;
[0049] A second determination module, configured to determine a model loss of the preset object recognition model based on the obtained network loss of each sub-network;
[0050] An adjustment module is used to adjust the network parameters of the preset object recognition model when it is determined based on the model loss that the preset object recognition model has not converged, and return to the step of obtaining the sample image set.
[0051] In a fourth aspect, an embodiment of the present application provides an object recognition device, the device comprising:
[0052] A second acquisition module is used to acquire a target image;
[0053] The second input module is used to input the target image into a preset object recognition model for performing an object recognition task to obtain an object recognition result of the target image; wherein the preset object recognition model is trained based on the model training method.
[0054] In a fifth aspect, an embodiment of the present application provides an electronic device, including:
[0055] Memory, used to store computer programs;
[0056] The processor is used to implement any of the model training methods and / or object recognition methods when executing a program stored in the memory.
[0057] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described model training methods and / or object recognition methods.
[0058] Beneficial effects of the embodiments of the present application:
[0059] The model training method provided in the embodiment of the present application first obtains a sample image set, inputs each sample image in the sample image set into a preset object recognition model to be trained for performing an object recognition task, and the global encoder in the object recognition model performs feature encoding on each sample image to obtain the initial image feature of each sample image; in order to improve the accuracy of the object recognition model, the object recognition model includes multiple sub-networks, and the multiple sub-networks have correlation with the object recognition task, and the initial image feature of each sample image is respectively input into the target sub-network, the first sub-network and the second sub-network in the object recognition model, and the output results of each sub-network for each sample image are respectively obtained; the first sub-network can generate a first auxiliary feature for characterizing the clarity of the object of each sample image, and the second sub-network can generate a second auxiliary feature for characterizing the number of characters of each sample image. The first auxiliary feature and the second auxiliary feature of each sample image are used to predict the number of recognizable characters and the number of unrecognizable characters of each sample image respectively based on the second auxiliary feature of each sample image; the first auxiliary feature and the second auxiliary feature of each sample image are auxiliary features used by the target sub-network, and combined with the initial image features of each sample image, the result of the object recognition task can be predicted; after obtaining the output results of each sub-network, for each sub-network, based on the output results of the sub-network for each sample image and the calibration true value corresponding to each sample image, the network loss of the sub-network is determined, and based on the network loss of each sub-network, the model loss of the preset object recognition model is determined; after obtaining the model loss, whether the preset object recognition model has converged is judged based on the model loss. If not, the network parameters of the preset object recognition model can be adjusted, and the step of obtaining the sample image set is returned to continue training the preset object recognition model. It can be seen that in the embodiments of the present application, the target subnetwork can combine the object clarity dimension, the character number dimension and the initial image feature dimension to predict the result of the object recognition task, so that the preset object recognition model can rely on the object clarity, the number of characters and the calibration content matching the object recognition task for supervised training during training. Compared with the related art, the supervised training method that only relies on the calibration content of the sample image matching the object recognition task can improve the accuracy of the trained object recognition model.
[0060] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.
[0062] Figure 1 A schematic diagram of the classification results of license plates classified according to clear attributes;
[0063] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;
[0064] Figure 3 A schematic diagram of a network structure of a preset object recognition model provided in an embodiment of the present application;
[0065] Figure 4 A characteristic schematic diagram of a second auxiliary feature provided in an embodiment of the present application;
[0066] Figure 5 A flowchart of an object recognition method provided in an embodiment of the present application;
[0067] Figure 6 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;
[0068] Figure 7 A schematic diagram of the structure of an object recognition device provided in an embodiment of the present application;
[0069] Figure 8 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.
[0071] With the continuous development of modern manufacturing and technology, more and more vehicles are appearing in society, and the importance of vehicle automation management is self-evident. License plates are an inherent attribute of vehicles and play a very important role in public security, vehicle monitoring and management.
[0072] License plates usually have a fixed format and limited character content. With the support of deep learning technology, the accuracy of clear license plate recognition using object recognition models has reached a very high level. However, the recognition of unclear (low-quality) license plates is greatly affected by the influence of lighting, imaging, object occlusion, etc. For example, the classification results of license plates classified according to the clarity attribute are as follows: Figure 1 As shown: the clarity attributes can include clarity, defect, blur (partially recognizable), reflection and extremely severe blur (unrecognizable). The black filling in the figure is privacy masking; (1) For example, the second license plate in the defective category is blocked by the car in front and the last two characters are unrecognizable; (2) For example, the third license plate in the blurry category is affected by the lighting and all the characters are almost unrecognizable to the human eye; (3) For example, the first license plate in the reflective category is affected by the lighting and the first two characters are almost unrecognizable to the human eye; for example, the first license plate in the extremely severe blur category is affected by motion blur and all the characters are unrecognizable to the human eye.
[0073] For sample images with unclear character strings, for example, containing characters that cannot be recognized by the human eye, calibration is very difficult and will inevitably introduce noise. If the object recognition model is trained with noise in a strong supervised manner, the trained object recognition model will forcibly predict the character results for unrecognizable characters with high confidence, resulting in incorrect recognition of low-quality license plates. In other words, the supervision of the calibration content of the sample image that matches the object recognition task will undoubtedly lead to the low accuracy of the trained object recognition model.
[0074] There is also a solution for deblurring unclear sample images in the related technology:
[0075] In one method, a predetermined blur discriminator is used to determine whether the sample image is a blurry image; if it is a blurry image, a predetermined deblurring network module is used to deblur the sample image. However, the blurry image has seriously lost the character texture details, and the deblurring result is uncontrollable; the multi-stage deblurring (blur judgment, deblurring, license plate detection, license plate recognition) method will introduce cumulative errors, resulting in the subsequent use of the deblurred sample image for model training. The accuracy of the trained object recognition model is still not high.
[0076] In another method, the sample image of the tilted blurred license plate (the license plate blurred due to tilt) is transformed by manual intervention to obtain a corrected sample image; the corrected sample image is de-blurred by using a de-motion blur filter; the de-motion blurred sample image is segmented to perform license plate character recognition. Since the premise of performing character segmentation on the license plate is the perspective transformation that needs to be performed by manual intervention, it requires a lot of manpower; and this solution only uses the de-motion blur filter, so it is only applicable to a specific blur form (not applicable to low-quality sample images such as reflections, occlusions, etc.), and this method is not universal.
[0077] Based on this, in order to accurately identify target objects with fixed formats and limited characters such as license plates and ID cards in images, the embodiments of the present application provide a model training method, object recognition method, device and electronic device to improve the accuracy of the object recognition model.
[0078] The following first introduces a model training method provided in an embodiment of the present application.
[0079] Among them, a model training method provided in an embodiment of the present application can be applied to an electronic device, and the electronic device is used to train a preset object recognition model. In a specific application, the electronic device can be a mobile phone, a computer, etc. In addition, the preset object recognition model in the present application is a neural network model of multi-task learning, including: a target subnetwork, a first subnetwork, and a second subnetwork, and the tasks between the subnetworks have correlation with object recognition.
[0080] A model training method provided in an embodiment of the present application may include the following steps:
[0081] Acquire a sample image set; wherein each sample image in the sample image set is an image containing objects of the same type;
[0082] Input each sample image in the sample image set into a preset object recognition model to be trained for performing an object recognition task, so that the global encoder in the preset object recognition model performs feature encoding on each sample image to obtain the initial image feature of each sample image, and input the initial image feature of each sample image into the target subnetwork, the first subnetwork and the second subnetwork in the preset object recognition model respectively, to obtain the output result of the target subnetwork for each sample image, the output result of the first subnetwork for each sample image and the output result of the second subnetwork for each sample image; wherein the first subnetwork is used to generate a first auxiliary feature for characterizing the object clarity of each sample image, the second subnetwork is used to generate a second auxiliary feature for characterizing the number of characters of each sample image, and based on the second auxiliary feature of each sample image, respectively predict the number of recognizable characters and the number of unrecognizable characters of each sample image, and the target subnetwork is used to predict the result of the object recognition task based on the initial image feature, the first auxiliary feature and the second auxiliary feature of each sample image;
[0083] For each of the target subnetwork, the first subnetwork, and the second subnetwork, based on an output result of the subnetwork for each sample image and a calibration true value of each sample image that matches the subnetwork, determine a network loss of the subnetwork;
[0084] Determining a model loss of the preset object recognition model based on the obtained network loss of each sub-network;
[0085] When it is determined based on the model loss that the preset object recognition model has not converged, the network parameters of the preset object recognition model are adjusted, and the process returns to the step of obtaining the sample image set.
[0086] The model training method provided in the embodiment of the present application first obtains a sample image set, inputs each sample image in the sample image set into a preset object recognition model to be trained for performing an object recognition task, and the global encoder in the object recognition model performs feature encoding on each sample image to obtain the initial image feature of each sample image; in order to improve the accuracy of the object recognition model, the object recognition model includes multiple sub-networks, and the multiple sub-networks have correlation with the object recognition task, and the initial image feature of each sample image is respectively input into the target sub-network, the first sub-network and the second sub-network in the object recognition model, and the output results of each sub-network for each sample image are respectively obtained; the first sub-network can generate a first auxiliary feature for characterizing the clarity of the object of each sample image, and the second sub-network can generate a second auxiliary feature for characterizing the number of characters of each sample image. The first auxiliary feature and the second auxiliary feature of each sample image are used to predict the number of recognizable characters and the number of unrecognizable characters of each sample image respectively based on the second auxiliary feature of each sample image; the first auxiliary feature and the second auxiliary feature of each sample image are auxiliary features used by the target sub-network, and combined with the initial image features of each sample image, the result of the object recognition task can be predicted; after obtaining the output results of each sub-network, for each sub-network, based on the output results of the sub-network for each sample image and the calibration true value corresponding to each sample image, the network loss of the sub-network is determined, and based on the network loss of each sub-network, the model loss of the preset object recognition model is determined; after obtaining the model loss, whether the preset object recognition model has converged is judged based on the model loss. If not, the network parameters of the preset object recognition model can be adjusted, and the step of obtaining the sample image set is returned to continue training the preset object recognition model. It can be seen that in the embodiments of the present application, the target subnetwork can combine the object clarity dimension, the character number dimension and the initial image feature dimension to predict the result of the object recognition task, so that the preset object recognition model can rely on the object clarity, the number of characters and the calibration content matching the object recognition task for supervised training during training. Compared with the related art, the supervised training method that only relies on the calibration content of the sample image matching the object recognition task can improve the accuracy of the trained object recognition model.
[0087] A model training method provided in an embodiment of the present application is introduced below in conjunction with the accompanying drawings.
[0088] like Figure 2 As shown, a model training method provided in an embodiment of the present application may include the following steps:
[0089] S201: Acquire a sample image set;
[0090] Wherein, each sample image in the sample image set is an image containing objects of the same type;
[0091] The embodiment of the present application is to train a preset object recognition model to be trained. The preset object recognition model can perform object recognition tasks for images of the same type of objects, thereby identifying target objects with fixed layouts and limited characters in the images. Therefore, a sample image set can be obtained first, and each sample image in the sample image set is an image containing the same type of objects. For example, each sample image in the sample image set is a sample image containing a license plate, or each sample image in the sample image set is a sample image containing an ID card, or each sample image in the sample image set is a sample image containing a bank card, etc., and the present application does not limit this. In addition, the same type of objects contained in each sample image in the sample image set are target objects with fixed layouts and limited characters. The target object can be a license plate, or an ID card, work permit, student ID card and other documents.
[0092] It can be understood that the preset object recognition model is a model that performs object recognition tasks for a class of objects. If each sample image in the sample image set is a sample image containing a license plate, then the preset object recognition model trained subsequently can be used for the license plate recognition task; if each sample image in the sample image set is a sample image containing an ID card, then the preset object recognition model trained subsequently can be used for the ID card recognition task; that is, the objects that can be recognized by the preset object recognition model are matched with the same class of objects contained in each sample image in the sample image set.
[0093] In addition, the clarity of each sample image in the sample image set may be any clarity, and the sample image set may include sample images of one or more clarity levels, which is not limited in the present application.
[0094] S202: Input each sample image in the sample image set into a preset object recognition model to be trained for performing an object recognition task, so that a global encoder in the preset object recognition model performs feature encoding on each sample image to obtain an initial image feature of each sample image, and input the initial image feature of each sample image into a target sub-network, a first sub-network, and a second sub-network in the preset object recognition model, respectively, to obtain an output result of the target sub-network for each sample image, an output result of the first sub-network for each sample image, and an output result of the second sub-network for each sample image;
[0095] Among them, the first subnetwork is used to generate a first auxiliary feature for characterizing the clarity of the object for each sample image, the second subnetwork is used to generate a second auxiliary feature for characterizing the number of characters for each sample image, and based on the second auxiliary feature of each sample image, predict the number of recognizable characters and the number of unrecognizable characters of each sample image respectively, and the target subnetwork is used to predict the result of the object recognition task based on the initial image feature, the first auxiliary feature and the second auxiliary feature of each sample image.
[0096] It should be noted that the first sub-network can specifically generate a first auxiliary feature for characterizing the object clarity of each sample image based on the initial image features of each sample image, and the second sub-network can generate a second auxiliary feature for characterizing the number of characters in each sample image based at least on the initial image features of each sample image.
[0097] After obtaining the sample image set, each sample image in the sample image set can be input into a preset object recognition model to be trained for performing the object recognition task to obtain the output result of the model. Exemplarily, the object recognition task may include at least one subtask, and the at least one subtask may include at least one of a first subtask for character string recognition and a second subtask for object type recognition in a specified dimension, wherein, considering that several objects belonging to the same category can be divided into types from a certain dimension, for example, for license plates, several license plates can be divided into white license plates, blue license plates, green license plates and other license plate types from the perspective of color dimension as the specified dimension, therefore, the object recognition task may also include the above-mentioned second subtask.
[0098] Among them, the object recognition model preset in the present application includes a global encoder, a target subnetwork, a first subnetwork and a second subnetwork. After each sample image is input into the preset object recognition model, the global encoder first performs feature encoding on each sample image to obtain the initial image features of each sample image. The initial image features of a sample image can be understood as the image features of the sample image as a whole. In order to improve the accuracy of the trained preset object recognition model, the target subnetwork used to perform the object recognition task can utilize the first auxiliary features of each sample image in the dimension of object clarity generated by the first subnetwork, the second auxiliary features of each sample image in the dimension of character quantity generated by the second subnetwork, and combine the initial image features of each sample image to obtain more accurate results of the object recognition task.
[0099] It should be noted that the object recognition model preset in the present application is different from the object recognition model in the prior art. The object recognition model preset in the present application is a neural network model for multi-task learning. The multiple tasks are tasks performed by the target subnetwork, the first subnetwork and the second subnetwork. Among them, the first subnetwork and the second subnetwork serve as auxiliary tasks to assist the target subnetwork in object recognition to improve the accuracy of object recognition.
[0100] In addition, the output result of the target subnetwork is the result of the predicted object recognition task. The result of the object recognition task is a result that includes confidence, and the result of the object recognition task is the result with the highest confidence among all the predicted results. Specifically, the result of the object recognition task can be a specific string of the identified object, and each character in the string has a corresponding confidence, and the confidence corresponding to each character can characterize whether the character is a recognizable character or an unrecognizable character. Compared with the related art, the confidence of unrecognizable characters in the present application is more reasonable.
[0101] Exemplarily, if the sample image is an image containing a license plate, the result of the object recognition task can be the license plate number (of course, it can also include the license plate type, etc., which is not limited in this application), and the result of the object recognition task is: Beijing A×××××, where each "×" in the license plate is only used as a character example. In actual applications, each "×" can be replaced by a character suitable for a license plate, where the confidence of Beijing can be 90%, the confidence of A can be 90%, the confidence of the first "×" can be 88%, the confidence of the second "×" can be 78%, the confidence of the third "×" can be 82%, the confidence of the fourth "×" can be 30%, and the confidence of the fifth "×" can be 26%; where the confidence threshold can be 50%, when the confidence of a character is greater than 50%, the character is a recognizable character, and when the confidence of a character is less than or equal to 50%, the character is an unrecognizable character. Of course, when the target subnetwork predicts the result of each character, there may be multiple possible prediction results for the character, and each prediction result has a confidence level. The prediction result with the highest confidence level can be used as the final prediction result of the character. For example, for the second "×", the multiple possible prediction results of the character are: 1, with a confidence level of 78%; 7, with a confidence level of 20%; 9, with a confidence level of 2%; then the prediction result of the character is 1, with a confidence level of 78%.
[0102] In the present application, the target sub-network can identify the string of the object and the object type of the specified dimension (i.e., classify the object according to the specified dimension). Exemplarily, the object recognition task includes: a first sub-task for identifying the string and a second sub-task for identifying the object type of the specified dimension;
[0103] The target subnetwork is used to perform feature fusion on the initial image features and the first auxiliary features of each sample image, and perform feature encoding on the fused image features to obtain target image features of each sample image;
[0104] Based on the target image features of each sample image and the second auxiliary features of each sample image, predict the recognition result of the first subtask and obtain the output result corresponding to the first subtask; and, based on the target image features of each sample image, predict the recognition result of the second subtask and obtain the output result corresponding to the second subtask.
[0105] The object recognition task may include a first subtask for recognizing a character string and a second subtask for recognizing an object type of a specified dimension, wherein the first subtask is used to recognize the specific content of each character in the character string; the second subtask is used to classify the entire character string according to the specified dimension. For example, if the sample image is an image containing a license plate, the specified dimension is the license plate type dimension, and the types may include: motor vehicle type, new energy type, and large motor vehicle type. Moreover, since the colors of the license plates of different types are different, the motor vehicle type is blue, the new energy type is green, and the large motor vehicle type is yellow, the object type can be recognized according to color, and the result of the second subtask may be that the license plate belongs to the motor vehicle type.
[0106] Specifically, the target sub-network can first fuse the initial image features and the first auxiliary features of each sample image, and encode the fused image features to obtain the target image features of each sample image, which are the features fused by the two dimensions of clarity and the overall image; based on the target image features of each sample image and the second auxiliary features representing the character count of each sample image, the recognition results of the first sub-task can be predicted, that is, each character in the string of each sample image is predicted to obtain the output result corresponding to the first sub-task, which is equivalent to combining the features of the clarity dimension, the overall image dimension, and the character count dimension to predict the characters, and obtain the output result corresponding to the first sub-task with higher accuracy. The second sub-task does not need to consider the second auxiliary features of the character quantity dimension, and can directly predict the recognition results of the second sub-task based on the target image features of each sample image to obtain the output result corresponding to the second sub-task.
[0107] In addition, there may be multiple ways to fuse the initial image features and the first auxiliary features of each sample image. For example, the fusion may be achieved through feature point multiplication, or based on an attention mechanism or pooling. The specific implementation method may be similar to the prior art and will not be elaborated here.
[0108] It should be noted that the above description of the object recognition task is only an exemplary description. The object recognition task may include at least a first subtask for recognizing a character string, and other recognition tasks required by the user, for example, the object recognition task includes a first subtask, a second subtask, and a third subtask for recognizing the geographical area represented by the characters in the character string, etc., and this application does not limit this. Among them, the content of the characters in the specified position in the character string can represent the geographical area to which the character string belongs, for example, for the character string of the license plate object, usually the first character can represent the geographical area to which the character string belongs; for the character string of the ID card object, usually the first 6 characters can represent the geographical area to which the character string belongs; based on the target image features of each sample image, the recognition result of the third subtask can be predicted to obtain the output result corresponding to the third subtask.
[0109] Exemplarily, in one implementation, the target subnetwork includes: a target feature encoder, a first classifier, and a second classifier;
[0110] The target feature encoder is used to perform feature fusion on the initial image feature and the first auxiliary feature of each sample image, and perform feature encoding on the fused image feature to obtain the target image feature of each sample image;
[0111] The first classifier is used to predict the recognition result of the first subtask based on the target image feature of each sample image and the second auxiliary feature of each sample image, and obtain the output result corresponding to the first subtask;
[0112] The second classifier is used to predict the recognition result of the second subtask based on the target image features of each sample image, and obtain the output result corresponding to the second subtask.
[0113] The target subnetwork can be decoupled into a target feature encoder, a first classifier and a second classifier. The target feature encoder can fuse the initial image features and the first auxiliary features of each sample image, feature encode the fused image features, and obtain the target image features of each sample image; the first classifier can predict the recognition results of the first subtask based on the target image features of each sample image and the second auxiliary features of each sample image; the second classifier can predict the recognition results of the second subtask based on the target image features of each sample image.
[0114] By decoupling the target sub-network, the target feature encoder, the first classifier and the second classifier can only perform their respective tasks, so as to simplify the object recognition task of the target sub-network and improve the efficiency of outputting the corresponding results.
[0115] Optionally, the first sub-network includes: a first feature encoder; the second sub-network includes: a second feature encoder, a third classifier and a fourth classifier;
[0116] The first feature encoder is used to generate a first auxiliary feature of each sample image based on the initial image feature of each sample image;
[0117] The second feature encoder is used to perform feature fusion on the initial image feature of each sample image and the first auxiliary feature of each sample image, and perform feature encoding on the fused auxiliary feature to obtain the second auxiliary feature of each sample image;
[0118] The third classifier is used to predict the first number of recognizable characters for each sample image based on the second auxiliary feature of each sample image;
[0119] The fourth classifier is used to predict the number of unrecognizable characters for each sample image based on the second auxiliary feature of each sample image.
[0120] The first feature encoder in the first subnetwork can generate a first auxiliary feature of each sample image based on the initial image feature of each sample image. Exemplarily, the first feature encoder can encode the initial image feature of each sample image with respect to clarity to generate a first auxiliary feature of each sample image.
[0121] The second feature encoder in the second subnetwork can perform feature fusion on the initial image features of each sample image and the first auxiliary features of each sample image. The fusion method can be similar to the method of fusing the initial image features and the first auxiliary features of each sample image in the above-mentioned target subnetwork. The fused auxiliary features are encoded for character counts to obtain the second auxiliary features of each sample image; the third classifier can predict the first number of recognizable characters for each sample image based on the second auxiliary features of each sample image; the fourth classifier can predict the second number of unrecognizable characters for each sample image based on the second auxiliary features of each sample image.
[0122] In addition, after obtaining the first number and the second number of each sample image, the sample images can be classified based on the first number and the second number of each sample image. For example, each sample image can be divided into 10 categories according to the first number of recognizable characters in each sample image (of course, each sample image can also be classified according to the second number, or according to the first number and the second number, which is not limited), the number of recognizable characters of the sample image in the first category is 10, the number of recognizable characters of the sample image in the second category is 9, ... the number of recognizable characters of the sample image in the tenth category is 0, etc. Thus, the clarity of the sample image is divided again by the dimensions of the number of recognizable characters and the number of unrecognizable characters to improve the accuracy of the preset object recognition model.
[0123] In addition, the prediction of the first number of recognizable characters for any sample image can be understood as a classification process, where each category represents a quantity (it can also be understood that a quantity can be used as a category), different categories represent different quantities, and the predicted first quantity is the quantity represented by the category with the highest confidence. Similarly, the prediction of the second number of unrecognizable characters for any sample image can also be understood as a classification process.
[0124] Exemplarily, in one implementation, the second auxiliary feature of each sample image is a feature map having a specified number of channels; the specified number is the maximum number of predicted characters.
[0125] The second auxiliary feature of each sample image can be a feature map with a specified number of channels, where the specified number is the maximum number of predicted characters. The specified number can be flexibly set according to the different objects in the sample image. For example, if the object in the sample image is a license plate, and due to differences in geographical areas, the number of characters in the license plate is different, then the specified number can be 8 or 10, etc., and this application does not limit this.
[0126] It should be noted that the feature map of each sample image may implicitly contain the length and width of the sample image, the specified number of the maximum predicted number of characters, the maximum predicted number of classification types, etc., and each channel implicitly contains whether the channel represents a character, and if the channel represents a character, whether the character is a recognizable character, etc., and the present application does not limit this.
[0127] It should be noted that, on the premise that each sub-network can realize the corresponding function, the specific structure of each sub-network can be set according to the actual situation, and this application does not make specific limitations on this. That is to say, for any sub-network, any network structure that can realize the corresponding function can be used as the network structure of the sub-network in this application.
[0128] S203: for each sub-network among the target sub-network, the first sub-network and the second sub-network, based on the output result of the sub-network for each sample image and the calibration true value of each sample image that matches the sub-network, determine the network loss of the sub-network;
[0129] After obtaining the output results of each sub-network for each sample image, for each sub-network, the network loss of the sub-network can be determined based on the output results of the sub-network for each sample image and the calibration true value matching each sample image with the sub-network.
[0130] Exemplarily, the calibration truth value that matches each sample image with the target subnetwork may be: the character content of each calibrated sample image, and the classification result of the object type of the specified dimension; the calibration truth value that matches each sample image with the first subnetwork may be: the clarity of each calibrated sample image, for example: clear, blurred or defective, etc.; the calibration truth value that matches each sample image with the second subnetwork may be: the number of recognizable characters and the number of unrecognizable characters of each calibrated sample image.
[0131] For the target subnetwork and the second subnetwork, the network loss of the subnetwork can be determined according to their respective corresponding classification loss functions, and the specific implementation method can be similar to the prior art; for the first subnetwork, the network loss of the first subnetwork is the contrast loss of each sample image for sample images of the same type of clarity and sample images of different types of clarity. The specific calculation method will be introduced in detail in the subsequent embodiments and will not be repeated here.
[0132] S204: Determine the model loss of the preset object recognition model based on the obtained network loss of each sub-network;
[0133] After the network loss of each sub-network is obtained, the overall model loss of the preset object recognition model can be determined based on the obtained network loss of each sub-network. Exemplarily, in one implementation, the model loss of the preset object recognition model is determined based on the obtained network loss of each sub-network, including:
[0134] The network loss of each sub-network obtained is subjected to a specified operation process to obtain the model loss of the preset object recognition model.
[0135] In the present application, the network loss of each sub-network obtained can be subjected to a specified operation process to obtain the model loss of the preset object recognition model; wherein the specified operation process can be understood as assigning weights to the network loss of each sub-network and summing them up. Without considering the importance, the weight of the network loss of each sub-network can be 1; when considering the importance, the importance of the target sub-network is higher than that of the first sub-network and the second sub-network, and the importance of the first sub-network and the second sub-network is the same, then the weight of the network loss of the target sub-network can be 2, and the weights of the network loss of the first sub-network and the network loss of the second sub-network can be 1.
[0136] It should be noted that the above-mentioned method for determining the model loss is only an example, and any method that can determine the model loss based on the network loss of each sub-network is applicable to the present application, and the present application does not limit this.
[0137] S205: When it is determined based on the model loss that the preset object recognition model has not converged, adjust the network parameters of the preset object recognition model, and return to the step of obtaining the sample image set.
[0138] After determining the model loss of the preset object recognition model, it can be judged whether the preset object recognition model converges based on the model loss. If it converges, such as: the model loss is less than the preset threshold, the trained preset object recognition model is directly obtained; if it does not converge, such as: the model loss is not less than the preset threshold, it is necessary to continue training the preset object recognition model. The network parameters of each subnetwork in the preset object recognition model can be adjusted first, and the sample image set can be re-acquired for training.
[0139] Among them, for each sub-network, the network parameters of the sub-network can be adjusted so that the output result of the sample image based on the adjusted network parameters is closer to the calibrated true value of the sample image that matches the sub-network. The specific adjustment method of the network parameters can be similar to the existing technology and will not be repeated here.
[0140] In the technical solution of the present application, the sample image set involved, as well as the acquisition, storage, use, processing, transmission, provision and disclosure of the initial image features, first auxiliary features, second auxiliary features, etc. of each sample image, are all carried out with the user's authorization.
[0141] The model training method provided in the embodiment of the present application first obtains a sample image set, inputs each sample image in the sample image set into a preset object recognition model to be trained for performing an object recognition task, and the global encoder in the object recognition model performs feature encoding on each sample image to obtain the initial image feature of each sample image; in order to improve the accuracy of the object recognition model, the object recognition model includes multiple sub-networks, and the multiple sub-networks have correlation with the object recognition task, and the initial image feature of each sample image is respectively input into the target sub-network, the first sub-network and the second sub-network in the object recognition model, and the output results of each sub-network for each sample image are respectively obtained; the first sub-network can generate a first auxiliary feature for characterizing the clarity of the object of each sample image, and the second sub-network can generate a second auxiliary feature for characterizing the number of characters of each sample image. The first auxiliary feature and the second auxiliary feature of each sample image are used to predict the number of recognizable characters and the number of unrecognizable characters of each sample image respectively based on the second auxiliary feature of each sample image; the first auxiliary feature and the second auxiliary feature of each sample image are auxiliary features used by the target sub-network, and combined with the initial image features of each sample image, the result of the object recognition task can be predicted; after obtaining the output results of each sub-network, for each sub-network, based on the output results of the sub-network for each sample image and the calibration true value corresponding to each sample image, the network loss of the sub-network is determined, and based on the network loss of each sub-network, the model loss of the preset object recognition model is determined; after obtaining the model loss, whether the preset object recognition model has converged is judged based on the model loss. If not, the network parameters of the preset object recognition model can be adjusted, and the step of obtaining the sample image set is returned to continue training the preset object recognition model. It can be seen that in the embodiments of the present application, the target subnetwork can combine the object clarity dimension, the character number dimension and the initial image feature dimension to predict the result of the object recognition task, so that the preset object recognition model can rely on the object clarity, the number of characters and the calibration content matching the object recognition task for supervised training during training. Compared with the related art, the supervised training method that only relies on the calibration content of the sample image matching the object recognition task can improve the accuracy of the trained object recognition model.
[0142] Optionally, in another embodiment of the present application, in order to further distinguish the clarity within the class of sample images of the same clarity, the calibration true value of each sample image matched with the first sub-network is: a specified category label for characterizing the clarity of the object;
[0143] The determining of the network loss of the sub-network based on the output result of the sub-network for each sample image and the calibration truth value of each sample image that matches the sub-network includes:
[0144] In the case where the sub-network is the first sub-network, based on the first auxiliary feature of each sample image and the designated category label of each sample image, and using a predetermined contrast loss function, determining the network loss of the sub-network;
[0145] The predetermined contrast loss function is a function for calculating the loss based on the difference between sample images with the same specified category label regarding the first auxiliary feature, and the difference between sample images with different specified category labels regarding the first auxiliary feature.
[0146] The first sub-network can generate a first auxiliary feature for characterizing the object clarity of each sample image, that is, the first sub-network predicts the sample image through the object clarity dimension. Correspondingly, the calibration true value of each sample image matched with the first sub-network is a specified category label characterizing the object clarity of each sample image, such as: clear, blurred or defective, etc.
[0147] For the first sub-network, based on the contrast loss, the feature distance between each sample image and the sample images of different categories of the sample image can be enlarged, and the feature distance between the sample image and the sample images of the same category of the sample image can be shortened, so as to accurately predict the quality score of the object clarity of each sample image. The present application uses a predetermined contrast loss function, the first auxiliary feature of each sample image, and the designated category label of each sample image to determine the network loss of the first sub-network. Among them, the predetermined contrast loss function is a function for calculating the loss based on the difference between the first auxiliary features of the sample images with the same designated category label, and the difference between the first auxiliary features of the sample images with different designated category labels; that is, for each sample image, the predetermined contrast loss function can calculate the difference between the first auxiliary features of the sample image and the sample images of the same category, and the difference between the first auxiliary features of the sample image and the sample images of different categories; the sample images of the same category and the sample images of different categories of each sample image can be determined based on the designated category labels of each sample image, and the sample images of the same category and the sample images of different categories have corresponding first auxiliary features, so that the network loss of the first sub-network is determined by the predetermined contrast loss function, the first auxiliary feature of each sample image, and the designated category label of each sample image.
[0148] Sample images of different categories can be accurately distinguished only by calculating contrast loss, but for each sample image of the same category, the object clarity of each sample image of the same category cannot be further distinguished. For example, for multiple sample images containing license plates with a specified category label of clear, it is impossible to distinguish the object clarity of each clear sample image, that is, it is impossible to distinguish which sample image has a higher object clarity, etc.; that is, the network loss of the first subnetwork calculated only using the predetermined contrast loss function, the first auxiliary feature of each sample image, and the specified category label of each sample image may not be accurate. In the present application, the number of recognizable characters and the number of unrecognizable characters predicted by the second subnetwork for each sample image are both corresponding to confidence levels;
[0149] The determining the network loss of the sub-network based on the first auxiliary feature of each sample image and the designated category label of each sample image and using a predetermined contrast loss function includes:
[0150] Determine the network loss of the sub-network based on the first auxiliary feature of each sample image, the designated category label of each sample image, the first confidence and the second confidence corresponding to each sample image, and using a predetermined contrast loss function;
[0151] The first confidence level corresponding to each sample image is the confidence level corresponding to the number of recognizable characters, and the second confidence level corresponding to each sample image is the confidence level corresponding to the number of unrecognizable characters.
[0152] In an embodiment of the present application, the second sub-network can respectively predict the number of recognizable characters and the number of unrecognizable characters of each sample image, and the obtained number of recognizable characters and the number of unrecognizable characters both have corresponding confidence levels; the number of recognizable characters and the number of unrecognizable characters can also characterize the object clarity of the sample image, for example: the more recognizable characters, the higher the object clarity of the sample image; the present application also applies the first confidence level corresponding to the number of recognizable characters of each sample image, and the second confidence level corresponding to the number of unrecognizable characters, to the calculation of the network loss of the first sub-network, so as to obtain a more accurate network loss of the first sub-network.
[0153] Exemplarily in one implementation, the determining the network loss of the sub-network based on the first auxiliary feature of each sample image, the designated category label of each sample image, the first confidence and the second confidence corresponding to each sample image, and using a predetermined contrast loss function includes:
[0154] For each sample image, based on the first auxiliary feature of the sample image, the first auxiliary feature of the sample image having the same designated category label as the sample image, and the first auxiliary feature of the sample image having a different designated category label from the sample image, an initial loss corresponding to the sample image is calculated according to a predetermined contrast loss function; and based on the first confidence and the second confidence corresponding to the sample image, a weight value is calculated according to a weight calculation method that matches the designated category label of the sample image; the initial loss corresponding to the sample image is multiplied by the calculated weight value to obtain a target loss of the sample image;
[0155] Add the target losses of each sample image to get the network loss of the sub-network;
[0156] The weight calculation method that matches the specified category label of the sample image includes:
[0157] If the designated category label of the sample label is the label with the highest degree of clarity, the ratio of the first confidence level corresponding to the sample image to the target value corresponding to the sample image is calculated as the weight; wherein the target value of the sample image is the value obtained by summing the first confidence level and the second confidence level corresponding to the sample image;
[0158] Otherwise, the ratio of the second confidence level corresponding to the sample image to the target value is calculated as the weight.
[0159] When the first confidence and the second confidence of each sample image are applied to the calculation of the first sub-network, the initial loss corresponding to each sample image can be calculated first; since the clarity of objects of different categories is different, the weight value corresponding to each category may also be different, and the weight value can be calculated based on the first confidence and the second confidence of each sample image in a weight calculation method that matches the specified category label of each sample image; after obtaining the weight value corresponding to each sample image, the initial loss corresponding to each sample image is multiplied by the corresponding weight value to obtain the target loss of the sample image; then the target losses of each sample image can be added to obtain the network loss of the first sub-network. Among them, due to the different number of sample images, the network loss of the first sub-network obtained is different, so the preset threshold of the corresponding model loss can be related to the number of sample images. The more sample images there are, the higher the preset threshold can be, and this application does not limit this.
[0160] Exemplarily, the first auxiliary feature of the sample image, the first auxiliary feature of the sample image having the same specified category label as the sample image, and the first auxiliary feature of the sample image having a different specified category label from the sample image can be substituted into the formula of a predetermined contrast loss function to calculate the initial loss corresponding to the sample image. The formula of the predetermined contrast loss function will be introduced in detail in subsequent embodiments and will not be elaborated here.
[0161] Among them, in the embodiment of the present application, the calculation method of the weight value of the sample image is matched with the designated category label of the sample image, so that the weight value of the sample image is matched with the designated category label of the sample image, that is, the weight value corresponding to the sample images of the same category is the same. For the category with the highest clarity, that is, the designated category label is the label representing the highest clarity, the ratio of the first confidence corresponding to the sample image and the target value corresponding to the sample image is calculated as the weight value; otherwise, that is, the designated category label is a label representing other clarity, the ratio of the second confidence corresponding to the sample image and the target value is calculated as the weight value. That is, in the present application, if the designated category label is the label representing the highest clarity, the ratio of the first confidence of the number of recognizable characters and the target value is used as the weight value of the sample image; if the designated category label is a label representing other clarity, the object clarity of the sample image is considered to be unclear, and the ratio of the second confidence of the number of unrecognizable characters and the target value is used as the weight value of the sample image.
[0162] In an embodiment of the present application, the network loss of the first subnetwork can be calculated based on the first auxiliary feature of each sample image and the designated category label (i.e., the calibrated true value) of each sample image in the first subnetwork, and using a predetermined contrast loss function, so that the first subnetwork can accurately predict the object clarity of the sample image; in order to further distinguish the object clarity of each sample image of the same category and improve the accuracy of the network loss of the first subnetwork, the first confidence of the number of recognizable characters and the second confidence of the number of unrecognizable characters predicted by the second subnetwork for each sample image can also be used in the calculation process of the network loss of the first subnetwork; for each sample image, the initial loss corresponding to the sample image can be calculated first, and then a weight value is calculated based on the first confidence and second confidence corresponding to the sample image, and a weight calculation method matching the designated category label of the sample image, and the weight value is multiplied by the initial loss to obtain the target loss of the sample image, and the sum of the target losses of each sample image is the network loss of the first subnetwork. It can be seen that the present application can apply the first confidence and the second confidence predicted by the second sub-network to the calculation of the network loss of the first sub-network, so that the dimension of the number of recognizable characters and the number of unrecognizable characters can be additionally utilized to calculate a more accurate network loss of the first sub-network.
[0163] A model training method provided by the present application is introduced in detail below in conjunction with a specific embodiment.
[0164] The present application provides a model training method, wherein the trained preset object recognition model can be applied to the recognition of license plates (taking the target object as a license plate as an example, but not limited to it) in low-quality images. The network structure of the preset object recognition model provided by the present application is as follows: Figure 3 As shown, from left to right are the first sub-network, the second sub-network and the target sub-network. The overall process is as follows:
[0165] After each sample image is input into the preset object recognition model, the global encoder in the preset object recognition model performs feature encoding on each sample image to obtain the initial image features of each sample image;
[0166] After obtaining the initial image features of each sample image, the initial image features of each sample image are respectively shunted to the first sub-network, the second sub-network and the target sub-network; wherein each sub-network is adapted to the corresponding input and output features, and can output the corresponding prediction results according to the input content; the encoder of each sub-network can be similar to the existing one, and the input and output can be adjusted according to the corresponding task;
[0167] In the first sub-network, the first feature encoder generates a clarity feature for the object clarity of each sample image based on the initial image feature of each sample image, that is, the first auxiliary feature mentioned above, and the clarity of each sample image can be distinguished and classified through the clarity feature of each sample image. The first sub-network can also compare the clarity of each sample image based on the number of recognizable characters and the number of unrecognizable characters of each sample image obtained by the second sub-network, combined with the clarity feature of the first sub-network, and calculate the contrast loss of each sample image, that is, the target loss of each sample image, so as to obtain the network loss of the first sub-network based on the contrast loss, etc.
[0168] In the second subnetwork, the second feature encoder generates a second auxiliary feature that characterizes the number of characters in each sample image based on the initial image feature and clarity feature of each sample image; the second auxiliary feature of each sample image is then input into the third classifier and the fourth classifier, the third classifier outputs the number of recognizable characters in each sample image based on the second auxiliary feature of each sample image, and the fourth classifier outputs the number of unrecognizable characters in each sample image based on the second auxiliary feature of each sample image, and the sum of the number of recognizable characters and the number of unrecognizable characters is the total number of characters in the license plate in the sample image. Of course, based on the number of recognizable characters and the number of unrecognizable characters of each sample image, each sample image can be classified into the number of recognizable characters and the number of unrecognizable characters, so as to characterize the clarity of the license plate from the dimension of the number of recognizable and unrecognizable characters.
[0169] In the target subnetwork, the target feature encoder generates the target image feature of each sample image based on the initial image feature and the clarity feature of each sample image; the target image feature of each sample image is then input into the first classifier and the second classifier; the first classifier predicts the recognition result of the first subtask about character string recognition based on the target image feature of each sample image and the second auxiliary feature of each sample image obtained by the second feature encoder, and obtains the output result corresponding to the first subtask; the second classifier predicts the recognition result of the second subtask about the object type of the specified dimension based on the target image feature of each sample image, and obtains the output result corresponding to the second subtask. Among them, the output result of the first subtask is the result of license plate character recognition, and the output result of the second subtask is the result of license plate type classification, which can be motor vehicle type (blue license plate), new energy type (green license plate) and large motor vehicle type (yellow license plate), etc.
[0170] It can be seen that the object recognition model preset in this application is a neural network model of multi-task learning. Through multiple related tasks, the recognition and prediction of objects such as license plates in multiple dimensions can improve the accuracy of the preset object recognition model. In addition, through weakly supervised auxiliary learning of the number of recognizable characters and the number of unrecognizable characters, a more reasonable low confidence can be assigned to unrecognizable characters, thereby improving the accuracy of the results of the object recognition task.
[0171] The purpose of the first sub-network is to enable the preset object recognition model to have intra-class distinction capabilities for different clarity levels, as follows:
[0172] During the calibration process, the calibrator needs to clarify the clarity of the license plate. The following three types are exemplary (only as examples, other types may also be included, not limited):
[0173] Clear: All characters on the complete license plate are readable by human eyes;
[0174] Blurred: At least one character on the complete license plate cannot be recognized by human eyes;
[0175] Defective: license plate with incomplete characters;
[0176] Since image clarity is subjective and it is impossible to obtain objective image clarity through manual calibration, this application uses contrast loss to shorten the feature distance within each sample image of the same clarity category and to lengthen the feature distance between each sample image of different clarity categories. Even so, it is still impossible to distinguish between each sample image of the same clarity category, for example: multiple license plates are clear, but it is unknown which one of these license plates is clearer.
[0177] Therefore, with the help of the confidence of the number of recognizable characters and the number of unrecognizable characters obtained by the third classifier and the fourth classifier, combined with the manually calibrated clarity category, the sample images of the same category are distinguished within the category; wherein, the network loss of the first sub-network can be calculated by the contrast loss function:
[0178] Exemplarily, the contrast loss function may be: Specific ;in represents the feature distance, Represents the current sample (anchor). Each sample image can be used as the current sample. Indicates "samples of the same category as the current sample" (positive), Indicates "samples of different categories from the current sample" (negative), The thresholds representing the “feature distance between samples of the same category” and “feature distance between samples of different categories” are preset parameters. It is used to ensure that the feature distance between sample pairs of the same category (positive example pairs) and the feature distance between sample pairs of different categories (negative example pairs) have a certain degree of separation when training a preset object recognition model, so as to achieve category separation of samples of different categories; specifically, is a hyperparameter (a hyperparameter is a parameter whose value is set before the training process begins, rather than a parameter data obtained through training) that controls the minimum distance difference between positive and negative pairs in the feature space (the minimum distance difference is ), through this loss function The loss calculation prompts the preset object recognition model to adjust the network parameters to increase the similarity between the two samples of the positive pair (i.e., reduce the feature distance between the two samples of the positive pair) and reduce the similarity between the two samples of the negative pair (i.e., increase the feature distance between the two samples of the negative pair), thereby widening the feature distance between samples of different categories in the feature space and achieving classification of samples of different categories. It can be understood that each sample image in the sample image set can be used as the current sample to calculate using the contrast loss function, so that a loss value will be obtained for each sample image; and the network loss of the first subnetwork is the comprehensive loss obtained by summing or weighted summing the losses obtained for each sample image in the sample image set.
[0179] Among them, the confidence of the number of recognizable characters and the number of unrecognizable characters is reflected in the weight of the contrast loss function. , the specific calculation method of the network loss of the first sub-network is as follows:
[0180] Network loss of the first subnetwork Affected by "confidence of the number of recognizable characters / number of unrecognizable characters" and "clear attribute contrast loss", for:
[0181] ;
[0182] in, is the initial loss corresponding to the sample image whose clarity category is clear; is the initial loss corresponding to the sample image whose clarity category is blurred; is the initial loss corresponding to the sample image whose clarity category is defective; is a first auxiliary feature of a sample image whose clarity category is clear; is the first auxiliary feature of the sample image whose clarity category is blurred; is the first auxiliary feature of the sample image whose clarity category is defective; A first auxiliary feature of a clear sample image of the same type as the sample image; A first auxiliary feature of a blurred sample image of the same type as the sample image; A first auxiliary feature of a defective sample image of the same type as the sample image; A first auxiliary feature of a non-clear sample image that is different from the clear sample image; is a first auxiliary feature of a non-blurred sample image of a different class from the blurred sample image; is a first auxiliary feature of a non-defective sample image of a different class from the defective sample image; is the confidence level of the number of recognizable characters in the sample image (i.e., the first confidence level mentioned above); is the confidence level of the number of unrecognizable characters in the sample image (ie, the second confidence level mentioned above).
[0183] Among them, the first sub-network does not output a score of the sample image regarding clarity, but influences the prediction of other sub-networks for the sample image through the obtained clarity features and contrast loss, etc.
[0184] In addition, other sub-networks can calculate the network loss of the sub-network through classification loss. For example, taking the target sub-network as an example, the classification loss function of the second classifier is as follows: ;in represents the output result of the second classifier (i.e., the output content of the i-th sample image for the j-th category), represents the calibration truth value corresponding to the output result of the second classifier (the calibration content of the i-th sample image for the j-th category), represents the number of sample images, Indicates the number of categorizable categories, is the classification loss of the second subtask of the second classifier.
[0185] The "clarity feature" of the first sub-network characterizes the overall clarity of the input sample image, and its feature dimension is the same as the image feature output by the global encoder. This application combines the clarity feature with the second sub-network and the target sub-network by directly using dot multiplication, that is, by fusing the clarity feature with other features by dot multiplication.
[0186] The feature diagram of the second auxiliary feature output by the second feature encoder of the second sub-network is as follows Figure 4 As shown, the number of channels is the maximum number of predicted characters, i.e. , where each channel can represent: recognizable characters, unrecognizable characters and non-characters (shown in different grayscales in the figure); where the feature map of the second auxiliary feature does not directly include the recognizable characters / unrecognizable characters / non-characters represented by each channel, and it can be understood that these features are implicit in the feature map. Of course, the feature map can also implicitly contain the height, width and other dimensional data of the sample image, as well as the number of classifiable categories.
[0187] The purpose of the second sub-network is to enable the preset object recognition model to distinguish between recognizable characters and unrecognizable characters. It is assumed that the content of a single character has three states: recognizable, unrecognizable, and non-character. The sum of all characters is the maximum predicted number of characters; the sum of the number of recognizable characters and unrecognizable characters is the total number of calibrated characters; but during calibration, recognizable characters and unrecognizable characters are not distinguished, and only the total number of calibrated characters is used as supervisory information. Therefore, the second sub-network is a weakly supervised training method, and its loss is the classification loss.
[0188] The license plate character counting task of the second sub-network is simpler than the license plate character recognition task, and the license plate character counting task converges faster. The simple auxiliary task can help the preset object recognition model learn more general and effective features, while providing useful supplementary information to help the preset object recognition model better understand and solve the main license plate recognition task. Therefore, the license plate character counting task is very suitable as an auxiliary task for multi-task learning.
[0189] For the target sub-network, on the basis of the original recognition (character-by-character classification), a second auxiliary feature is introduced. The number of channels of this feature is the maximum number of predicted characters, which corresponds to the physical meaning of the character position of the first classifier. This enables the first classifier to distinguish which characters are recognizable and which are unrecognizable, and outputs high confidence for recognizable characters and low confidence for unrecognizable characters, thereby avoiding the first classifier from strongly learning the appearance of unrecognizable characters.
[0190] The model loss of the preset object recognition model is:
[0191] ;
[0192] in: represents the network loss of the first subnetwork, represents the network loss of the second sub-network, represents the network loss for the license plate character recognition task of the first classifier, Represents the network loss for the license plate type classification task for the second classifier.
[0193] The object recognition model preset in this application includes weakly supervised auxiliary tasks of counting recognizable characters and unrecognizable characters, i.e., the tasks of the third classifier and the fourth classifier. The second auxiliary feature is used for the first classifier, so that the first classifier assigns high confidence to recognizable characters and low confidence to unrecognizable characters, and distinguishes the confidence of recognizable characters from that of unrecognizable characters, thereby reducing the high confidence false alarms of unrecognizable characters in the prior art. In addition, this application proposes a weakly supervised auxiliary task of intra-class comparison of image clarity, which combines the loss weights obtained by the confidence of the number of recognizable characters and the number of unrecognizable characters, so that sample images of clarity categories such as clear, blurred, and defective have intra-class distinction.
[0194] Based on the above embodiments, the present application also provides an object recognition method, such as Figure 5 As shown, the method includes:
[0195] S501: Acquire a target image;
[0196] S502: Inputting the target image into a preset object recognition model for performing an object recognition task to obtain an object recognition result of the target image;
[0197] Wherein, the preset object recognition model is obtained by training based on the model training method.
[0198] It can be understood that the objects included in the target image and the objects included in the sample image used when training the preset object recognition model are of the same type of objects.
[0199] The object recognition method provided in the embodiment of the present application is to perform object recognition based on the object recognition model obtained by the model training method. The target subnetwork can combine the object clarity dimension, the character number dimension and the initial image feature dimension to predict the result of the object recognition task, so that the preset object recognition model can rely on the object clarity, the character number and the calibration content matching the object recognition task for supervised training during training. Compared with the related art, the supervised training method that only relies on the calibration content matching the object recognition task of the sample image can improve the accuracy of the trained object recognition model. Therefore, the object recognition method provided in the embodiment of the present application can use a more accurate, trained object recognition model to recognize the target image and obtain a more accurate object recognition result of the target object.
[0200] Based on the above method embodiment, the present application also provides a model training device, such as Figure 6 As shown, the device comprises:
[0201] A first acquisition module 610 is used to acquire a sample image set; wherein each sample image in the sample image set is an image containing objects of the same type;
[0202] The first input module 620 is used to input each sample image in the sample image set into a preset object recognition model to be trained for performing an object recognition task, so that the global encoder in the preset object recognition model performs feature encoding on each sample image to obtain the initial image feature of each sample image, and the initial image feature of each sample image is respectively input into the target sub-network, the first sub-network and the second sub-network in the preset object recognition model to obtain the output result of the target sub-network for each sample image, the output result of the first sub-network for each sample image and the output result of the second sub-network for each sample image; wherein the first sub-network is used to generate a first auxiliary feature for characterizing the object clarity of each sample image, the second sub-network is used to generate a second auxiliary feature for characterizing the number of characters in each sample image, and based on the second auxiliary feature of each sample image, respectively predict the number of recognizable characters and the number of unrecognizable characters of each sample image, and the target sub-network is used to predict the result of the object recognition task based on the initial image feature, the first auxiliary feature and the second auxiliary feature of each sample image;
[0203] A first determination module 630 is used to determine, for each sub-network among the target sub-network, the first sub-network and the second sub-network, a network loss of the sub-network based on an output result of the sub-network for each sample image and a calibration true value of each sample image that matches the sub-network;
[0204] A second determination module 640 is used to determine the model loss of the preset object recognition model based on the obtained network loss of each sub-network;
[0205] The adjustment module 650 is used to adjust the network parameters of the preset object recognition model when it is determined based on the model loss that the preset object recognition model has not converged, and return to the step of obtaining the sample image set.
[0206] The model training device provided in the embodiment of the present application first obtains a sample image set, and inputs each sample image in the sample image set into a preset object recognition model to be trained for performing an object recognition task. The global encoder in the object recognition model performs feature encoding on each sample image to obtain the initial image feature of each sample image. In order to improve the accuracy of the object recognition model, the object recognition model includes multiple sub-networks, and the multiple sub-networks have correlation with the object recognition task. The initial image feature of each sample image is respectively input into the target sub-network, the first sub-network and the second sub-network in the object recognition model, and the output results of each sub-network for each sample image are respectively obtained. The first sub-network can generate a first auxiliary feature for characterizing the clarity of the object of each sample image, and the second sub-network can generate a second auxiliary feature for characterizing the number of characters of each sample image. The first auxiliary feature and the second auxiliary feature of each sample image are used to predict the number of recognizable characters and the number of unrecognizable characters of each sample image respectively based on the second auxiliary feature of each sample image; the first auxiliary feature and the second auxiliary feature of each sample image are auxiliary features used by the target sub-network, and combined with the initial image features of each sample image, the result of the object recognition task can be predicted; after obtaining the output results of each sub-network, for each sub-network, based on the output results of the sub-network for each sample image and the calibration true value corresponding to each sample image, the network loss of the sub-network is determined, and based on the network loss of each sub-network, the model loss of the preset object recognition model is determined; after obtaining the model loss, whether the preset object recognition model has converged is judged based on the model loss. If not, the network parameters of the preset object recognition model can be adjusted, and the step of obtaining the sample image set is returned to continue training the preset object recognition model. It can be seen that in the embodiments of the present application, the target subnetwork can combine the object clarity dimension, the character number dimension and the initial image feature dimension to predict the result of the object recognition task, so that the preset object recognition model can rely on the object clarity, the number of characters and the calibration content matching the object recognition task for supervised training during training. Compared with the related art, the supervised training method that only relies on the calibration content of the sample image matching the object recognition task can improve the accuracy of the trained object recognition model.
[0207] Optionally, the object recognition task includes: a first subtask for recognizing a character string and a second subtask for recognizing an object type of a specified dimension;
[0208] The target subnetwork is used to perform feature fusion on the initial image features and the first auxiliary features of each sample image, and perform feature encoding on the fused image features to obtain target image features of each sample image;
[0209] Based on the target image features of each sample image and the second auxiliary features of each sample image, predict the recognition result of the first subtask and obtain the output result corresponding to the first subtask; and, based on the target image features of each sample image, predict the recognition result of the second subtask and obtain the output result corresponding to the second subtask.
[0210] Optionally, the target subnetwork includes: a target feature encoder, a first classifier, and a second classifier;
[0211] The target feature encoder is used to perform feature fusion on the initial image feature and the first auxiliary feature of each sample image, and perform feature encoding on the fused image feature to obtain the target image feature of each sample image;
[0212] The first classifier is used to predict the recognition result of the first subtask based on the target image feature of each sample image and the second auxiliary feature of each sample image, and obtain the output result corresponding to the first subtask;
[0213] The second classifier is used to predict the recognition result of the second subtask based on the target image features of each sample image, and obtain the output result corresponding to the second subtask.
[0214] Optionally, the first sub-network includes: a first feature encoder; the second sub-network includes: a second feature encoder, a third classifier and a fourth classifier;
[0215] The first feature encoder is used to generate a first auxiliary feature of each sample image based on the initial image feature of each sample image;
[0216] The second feature encoder is used to perform feature fusion on the initial image feature of each sample image and the first auxiliary feature of each sample image, and perform feature encoding on the fused auxiliary feature to obtain the second auxiliary feature of each sample image;
[0217] The third classifier is used to predict the first number of recognizable characters for each sample image based on the second auxiliary feature of each sample image;
[0218] The fourth classifier is used to predict the number of unrecognizable characters for each sample image based on the second auxiliary feature of each sample image.
[0219] Optionally, the second determining module is specifically configured to:
[0220] The network loss of each sub-network obtained is subjected to a specified operation process to obtain the model loss of the preset object recognition model.
[0221] Optionally, the calibration truth value of each sample image matched with the first sub-network is: a specified category label used to characterize the clarity of the object;
[0222] The first determining module includes:
[0223] A determination submodule, configured to determine the network loss of the subnetwork based on the first auxiliary feature of each sample image and the designated category label of each sample image of the subnetwork and using a predetermined contrast loss function when the subnetwork is the first subnetwork;
[0224] The predetermined contrast loss function is a function for calculating the loss based on the difference between sample images with the same specified category label regarding the first auxiliary feature, and the difference between sample images with different specified category labels regarding the first auxiliary feature.
[0225] Optionally, the number of recognizable characters and the number of unrecognizable characters of each sample image predicted by the second sub-network are both corresponding to confidence levels;
[0226] The determining submodule includes a determining unit, which is used to:
[0227] Determine the network loss of the sub-network based on the first auxiliary feature of each sample image, the designated category label of each sample image, the first confidence and the second confidence corresponding to each sample image, and using a predetermined contrast loss function;
[0228] The first confidence level corresponding to each sample image is the confidence level corresponding to the number of recognizable characters, and the second confidence level corresponding to each sample image is the confidence level corresponding to the number of unrecognizable characters.
[0229] Optionally, the determining unit is specifically configured to:
[0230] For each sample image, based on the first auxiliary feature of the sample image, the first auxiliary feature of the sample image having the same designated category label as the sample image, and the first auxiliary feature of the sample image having a different designated category label from the sample image, an initial loss corresponding to the sample image is calculated according to a predetermined contrast loss function; and based on the first confidence and the second confidence corresponding to the sample image, a weight value is calculated according to a weight calculation method that matches the designated category label of the sample image; the initial loss corresponding to the sample image is multiplied by the calculated weight value to obtain a target loss of the sample image;
[0231] Add the target losses of each sample image to get the network loss of the sub-network;
[0232] The weight calculation method that matches the specified category label of the sample image includes:
[0233] If the designated category label of the sample label is the label with the highest degree of clarity, the ratio of the first confidence level corresponding to the sample image to the target value corresponding to the sample image is calculated as the weight; wherein the target value of the sample image is the value obtained by summing the first confidence level and the second confidence level corresponding to the sample image;
[0234] Otherwise, the ratio of the second confidence level corresponding to the sample image to the target value is calculated as the weight.
[0235] Optionally, the second auxiliary feature of each sample image is a feature map having a specified number of channels; the specified number is the maximum number of predicted characters.
[0236] Based on the above method embodiment, the present application also provides an object recognition device, such as Figure 7 As shown, the device comprises:
[0237] The second acquisition module 710 is used to acquire a target image;
[0238] The second input module 720 is used to input the target image into a preset object recognition model for performing an object recognition task to obtain an object recognition result of the target image; wherein the preset object recognition model is trained based on the model training method.
[0239] The object recognition device provided in the embodiment of the present application performs object recognition based on the object recognition model obtained by the model training method. The target subnetwork can combine the object clarity dimension, the character number dimension and the initial image feature dimension to predict the result of the object recognition task, so that the preset object recognition model can rely on the object clarity, the character number and the calibration content matching the object recognition task for supervised training during training. Compared with the related art, which only relies on the sample image calibration content matching the object recognition task for supervised training, the accuracy of the trained object recognition model can be improved. Therefore, the object recognition device provided in the embodiment of the present application can use a more accurate trained object recognition model to recognize the target image and obtain a more accurate object recognition result of the target object.
[0240] The present application also provides an electronic device, such as Figure 8 As shown, including:
[0241] Memory 801, used for storing computer programs;
[0242] The processor 802 is used to implement any of the model training methods and / or object recognition methods when executing the program stored in the memory 801.
[0243] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 802, the communication interface, and the memory 801 communicate with each other via the communication bus.
[0244] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0245] The communication interface is used for communication between the above electronic device and other devices.
[0246] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0247] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0248] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model training methods and / or object recognition methods are implemented.
[0249] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the model training methods and / or object recognition methods in the above-mentioned embodiments.
[0250] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.
[0251] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0252] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0253] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A model training method, characterized in that: The method comprises: Acquire a sample image set; wherein each sample image in the sample image set is an image containing objects of the same type; Input each sample image in the sample image set into a preset object recognition model to be trained for performing an object recognition task, so that the global encoder in the preset object recognition model performs feature encoding on each sample image to obtain the initial image feature of each sample image, and input the initial image feature of each sample image into the target subnetwork, the first subnetwork and the second subnetwork in the preset object recognition model respectively, to obtain the output result of the target subnetwork for each sample image, the output result of the first subnetwork for each sample image and the output result of the second subnetwork for each sample image; wherein the first subnetwork is used to generate a first auxiliary feature for characterizing the object clarity of each sample image, the second subnetwork is used to generate a second auxiliary feature for characterizing the number of characters of each sample image, and based on the second auxiliary feature of each sample image, respectively predict the number of recognizable characters and the number of unrecognizable characters of each sample image, and the target subnetwork is used to predict the result of the object recognition task based on the initial image feature, the first auxiliary feature and the second auxiliary feature of each sample image; For each of the target subnetwork, the first subnetwork, and the second subnetwork, based on an output result of the subnetwork for each sample image and a calibration true value of each sample image that matches the subnetwork, determine a network loss of the subnetwork; Determining a model loss of the preset object recognition model based on the obtained network loss of each sub-network; When it is determined based on the model loss that the preset object recognition model has not converged, the network parameters of the preset object recognition model are adjusted, and the process returns to the step of obtaining the sample image set.
2. The method according to claim 1, characterized in that The object recognition task includes: a first subtask for character string recognition and a second subtask for object type recognition of a specified dimension; The target subnetwork is used to perform feature fusion on the initial image features and the first auxiliary features of each sample image, and perform feature encoding on the fused image features to obtain target image features of each sample image; Based on the target image features of each sample image and the second auxiliary features of each sample image, predict the recognition result of the first subtask and obtain the output result corresponding to the first subtask; and, based on the target image features of each sample image, predict the recognition result of the second subtask and obtain the output result corresponding to the second subtask.
3. The method according to claim 2, characterized in that The target sub-network includes: a target feature encoder, a first classifier and a second classifier; The target feature encoder is used to perform feature fusion on the initial image feature and the first auxiliary feature of each sample image, and perform feature encoding on the fused image feature to obtain the target image feature of each sample image; The first classifier is used to predict the recognition result of the first subtask based on the target image feature of each sample image and the second auxiliary feature of each sample image, and obtain the output result corresponding to the first subtask; The second classifier is used to predict the recognition result of the second subtask based on the target image features of each sample image, and obtain the output result corresponding to the second subtask.
4. The method according to claim 1, characterized in that: The first sub-network includes: a first feature encoder; the second sub-network includes: a second feature encoder, a third classifier and a fourth classifier; The first feature encoder is used to generate a first auxiliary feature of each sample image based on the initial image feature of each sample image; The second feature encoder is used to perform feature fusion on the initial image feature of each sample image and the first auxiliary feature of each sample image, and perform feature encoding on the fused auxiliary feature to obtain the second auxiliary feature of each sample image; The third classifier is used to predict the first number of recognizable characters for each sample image based on the second auxiliary feature of each sample image; The fourth classifier is used to predict the number of unrecognizable characters for each sample image based on the second auxiliary feature of each sample image.
5. The method according to any one of claims 1 to 4, characterized in that: The determining the model loss of the preset object recognition model based on the obtained network loss of each sub-network includes: The network loss of each sub-network obtained is subjected to a specified operation process to obtain the model loss of the preset object recognition model.
6. The method according to any one of claims 1 to 4, characterized in that: The calibration truth value of each sample image matched with the first sub-network is: a specified category label used to characterize the clarity of the object; The determining of the network loss of the sub-network based on the output result of the sub-network for each sample image and the calibration truth value of each sample image that matches the sub-network includes: In the case where the sub-network is the first sub-network, based on the first auxiliary feature of each sample image and the designated category label of each sample image, and using a predetermined contrast loss function, determining the network loss of the sub-network; The predetermined contrast loss function is a function for calculating the loss based on the difference between sample images with the same specified category label regarding the first auxiliary feature, and the difference between sample images with different specified category labels regarding the first auxiliary feature.
7. The method according to claim 6, characterized in that The number of recognizable characters and the number of unrecognizable characters of each sample image predicted by the second sub-network are both corresponding to confidence levels; The determining the network loss of the sub-network based on the first auxiliary feature of each sample image and the designated category label of each sample image and using a predetermined contrast loss function includes: Determine the network loss of the sub-network based on the first auxiliary feature of each sample image, the designated category label of each sample image, the first confidence and the second confidence corresponding to each sample image, and using a predetermined contrast loss function; The first confidence level corresponding to each sample image is the confidence level corresponding to the number of recognizable characters, and the second confidence level corresponding to each sample image is the confidence level corresponding to the number of unrecognizable characters.
8. The method according to claim 7, characterized in that The determining the network loss of the sub-network based on the first auxiliary feature of each sample image, the designated category label of each sample image, the first confidence and the second confidence corresponding to each sample image, and using a predetermined contrast loss function includes: For each sample image, based on the first auxiliary feature of the sample image, the first auxiliary feature of the sample image having the same designated category label as the sample image, and the first auxiliary feature of the sample image having a different designated category label from the sample image, an initial loss corresponding to the sample image is calculated according to a predetermined contrast loss function; and based on the first confidence and the second confidence corresponding to the sample image, a weight value is calculated according to a weight calculation method that matches the designated category label of the sample image; the initial loss corresponding to the sample image is multiplied by the calculated weight value to obtain a target loss of the sample image; Add the target losses of each sample image to get the network loss of the sub-network; The weight calculation method that matches the specified category label of the sample image includes: If the designated category label of the sample label is the label with the highest degree of clarity, the ratio of the first confidence level corresponding to the sample image to the target value corresponding to the sample image is calculated as the weight; wherein the target value of the sample image is the value obtained by summing the first confidence level and the second confidence level corresponding to the sample image; Otherwise, the ratio of the second confidence level corresponding to the sample image to the target value is calculated as the weight.
9. The method according to any one of claims 1 to 4, characterized in that: The second auxiliary feature of each sample image is a feature map having a specified number of channels; the specified number is the maximum number of predicted characters.
10. An object recognition method, characterized in that: The method comprises: Get the target image; The target image is input into a preset object recognition model for performing an object recognition task to obtain an object recognition result of the target image; wherein the preset object recognition model is trained based on the model training method described in any one of claims 1-9.
11. A model training device, characterized in that: The device comprises: A first acquisition module is used to acquire a sample image set; wherein each sample image in the sample image set is an image containing objects of the same type; A first input module is used to input each sample image in the sample image set into a preset object recognition model to be trained for performing an object recognition task, so that the global encoder in the preset object recognition model performs feature encoding on each sample image to obtain the initial image feature of each sample image, and the initial image feature of each sample image is respectively input into the target sub-network, the first sub-network and the second sub-network in the preset object recognition model to obtain the output result of the target sub-network for each sample image, the output result of the first sub-network for each sample image and the output result of the second sub-network for each sample image; wherein the first sub-network is used to generate a first auxiliary feature for characterizing the object clarity of each sample image, the second sub-network is used to generate a second auxiliary feature for characterizing the number of characters of each sample image, and based on the second auxiliary feature of each sample image, respectively predict the number of recognizable characters and the number of unrecognizable characters of each sample image, and the target sub-network is used to predict the result of the object recognition task based on the initial image feature, the first auxiliary feature and the second auxiliary feature of each sample image; A first determination module is used to determine, for each sub-network among the target sub-network, the first sub-network and the second sub-network, a network loss of the sub-network based on an output result of the sub-network for each sample image and a calibration true value of each sample image that matches the sub-network; A second determination module, configured to determine a model loss of the preset object recognition model based on the obtained network loss of each sub-network; An adjustment module is used to adjust the network parameters of the preset object recognition model when it is determined based on the model loss that the preset object recognition model has not converged, and return to the step of obtaining the sample image set.
12. An object recognition device, characterized in that: The device comprises: A second acquisition module is used to acquire a target image; The second input module is used to input the target image into a preset object recognition model for performing an object recognition task to obtain an object recognition result of the target image; wherein the preset object recognition model is trained based on the model training method described in any one of claims 1-9.
13. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-10 when executing a program stored in a memory.
Citation Information
Patent Citations
Text recognition model training method and device and text recognition method and device
CN115546798A
Paper medicine box steel seal character recognition method based on improved YOLOV5 model
CN116486228A
