Model training method, image recognition method, device, equipment and storage medium

Abstract

The embodiments of the present application disclose a model training method, an image recognition method, an apparatus, a device and a storage medium. A sample set including a plurality of sample images with labeling information is obtained, a first feature extraction operation is performed on the sample images in the sample set to obtain a first image feature; a second feature extraction operation is performed on the sample images belonging to the same category to obtain second image features corresponding to the sample images of each category; a multi-task learning model is trained based on the first image feature, each second image feature and the labeling information to obtain a target multi-task learning model. The target multi-task learning model obtained by training according to the scheme of the present application can have high accuracy while taking into account the computing power of the hardware.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and specifically, to a model training method, image recognition method, device, equipment and storage medium. Background Art

[0002] With major breakthroughs in deep learning technology research, massive big data support, and the emergence of factors such as reduced computing costs, the requirements for multi-task information recognition technology in scenarios where neural networks are densely deployed are becoming increasingly higher. Most of the current multi-task models use the same feature extraction network to extract features in the scene, and then input the same feature map into the network of each subtask to meet the needs of performing multiple tasks on the same inference hardware. However, due to the diversity of tasks and different dependencies on different channels in the feature map, the feature extraction performance of some tasks is low when sharing the same feature extraction network. Summary of the invention

[0003] The embodiments of the present application provide a model training method, an image recognition method, an apparatus, a device, and a storage medium, which are used to improve the problem in the prior art that multi-task models share the same feature extraction network and have low performance of some tasks.

[0004] In a first aspect, an embodiment of the present application provides a model training method, the method comprising: obtaining a sample set, the sample set comprising a plurality of sample images with labeling information, the plurality of sample images respectively belonging to a plurality of image categories; performing a first feature extraction operation on the sample images in the sample set to obtain a first image feature, the first image feature being a common feature of sample images of a plurality of different categories; performing a second feature extraction operation on the sample images belonging to the same category to obtain second image features respectively corresponding to the sample images of each category; training a multi-task learning model based on the first image feature, each of the second image features and the labeling information to obtain a target multi-task learning model.

[0005] In a second aspect, an embodiment of the present application provides a method for image recognition, the method comprising: obtaining an image to be recognized; recognizing the image to be recognized using a target multi-task learning model to obtain a recognition result corresponding to the image to be recognized, wherein the target multi-task learning model is generated by the model training method described in any of the above embodiments.

[0006] In a third aspect, an embodiment of the present application provides a model training device, the device comprising: a sample set acquisition unit, a first feature extraction unit, a second feature extraction unit and a model generation unit. The sample set acquisition unit is used to acquire a sample set, the sample set includes a plurality of sample images with label information, and the plurality of sample images belong to a plurality of image categories respectively; the first feature extraction unit is used to perform a first feature extraction operation on the sample images in the sample set to obtain a first image feature, the first image feature being a common feature of sample images of a plurality of different categories; the second feature extraction unit is used to perform a second feature extraction operation on sample images belonging to the same category to obtain second image features corresponding to sample images of each category respectively; the model generation unit is used to train a multi-task learning model based on the first image feature, each of the second image features and the label information to obtain a target multi-task learning model.

[0007] In a fourth aspect, an embodiment of the present application provides an image recognition device, the device comprising: an image acquisition unit and a recognition unit. The image acquisition unit is used to acquire an image to be recognized; the recognition unit is used to recognize the image to be recognized using a target multi-task learning model to obtain a recognition result corresponding to the image to be recognized, and the target multi-task learning model is generated by the model training method described in any of the above embodiments.

[0008] In a fifth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: one or more processors and a storage device, wherein one or more programs are stored on the storage device. When the one or more programs are executed by the one or more processors, the one or more processors implement the model training method described in any of the above embodiments or the image recognition method described in any of the above embodiments.

[0009] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the model training method described in any of the above embodiments or the image recognition method described in any of the above embodiments.

[0010] The technical solution provided by the present invention obtains multiple sample images with marking information, performs a first feature extraction operation on the multiple sample images, and obtains a first image feature, which is a common feature of sample images of multiple different categories; performs a second feature extraction operation on sample images belonging to the same category to obtain second image features corresponding to sample images of each category; trains a multi-task learning model based on the first image feature, each of the second image features and the marking information to obtain a target multi-task learning model. Through the method, similar tasks are jointly trained while taking into account the computing power of the hardware to generate a target multi-task learning model with higher accuracy, thereby improving the performance of each subtask in the target multi-task learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown;

[0013] Figure 2 A schematic diagram of the structure of a target multi-task learning model provided in an embodiment of the present application is shown;

[0014] Figure 3 A flowchart of an image recognition method provided by an embodiment of the present application is shown;

[0015] Figure 4 A schematic diagram showing an execution of an embodiment of an image recognition method provided by an embodiment of the present application is shown;

[0016] Figure 5 A structural block diagram of a model training device provided in an embodiment of the present application is shown;

[0017] Figure 6 A structural block diagram of a pattern recognition device provided by an embodiment of the present application is shown;

[0018] Figure 7 A schematic diagram of the structure of an electronic device for executing a model training method or an image recognition method provided in an embodiment of the present application is shown;

[0019] Figure 8 A schematic diagram of the structure of a computer storage medium for executing a model training method or an image recognition method provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] In the scenario where neural networks are densely deployed for autonomous driving vehicles, there are multiple downstream subtasks such as lane detection, pedestrian detection, and scene recognition. The ideal technical solution is that each downstream subtask has a separate neural network to infer scene information and obtain signal outputs for executing multiple tasks. However, the ideal technical solution cannot be achieved when the network computing overhead is limited by hardware configuration. The multi-task model of the prior art usually enables multiple tasks to share the same feature extraction network, and then inputs the feature map extracted by the same feature extraction network into the information integration network of each subtask to adapt to the hardware computing power. However, due to the diversity of tasks and the different dependencies of each subtask on different channels in the feature map, the multi-task model of the prior art that shares the same feature extraction network has low accuracy, and the performance of some tasks cannot be taken into account in a multi-task scenario.

[0022] In order to alleviate the above problems, the inventors of the present application have proposed a model training method, image recognition method, device, equipment and storage medium provided by the present application. By obtaining a sample set of multiple sample images with label information, a first feature extraction operation is performed on the sample images in the sample set to obtain a first image feature, which is a common feature of sample images of multiple different categories; a second feature extraction operation is performed on sample images belonging to the same category to obtain second image features corresponding to sample images of each category; a multi-task learning model is trained based on the first image feature, each of the second image features and the label information to obtain a target multi-task learning model. Therefore, the target multi-task learning model generated by the above model training method uses the first feature extraction operation to obtain the common features of sample images of multiple different categories, and similar tasks share the corresponding second feature extraction operation to obtain the second image features corresponding to similar tasks, so that when training based on the first image feature, the second image feature and the label information, it can be achieved to improve the performance indicators of each subtask in the trained target multi-task learning model while adapting the hardware computing power.

[0023] The following will describe the various embodiments of the present application in detail with reference to the accompanying drawings. Figure 1, an embodiment of the present application provides a model training method, which includes steps S110 to S140.

[0024] Step S110: Acquire a sample set, wherein the sample set includes a plurality of sample images with label information, and the plurality of sample images respectively belong to a plurality of image categories.

[0025] It should be understood that the sample set is a sample image set for training a multi-task model, and the sample set includes a plurality of sample images with label information, wherein the plurality of image categories include at least two of dynamic object images, color feature images, and geometric shape images.

[0026] In some embodiments, the sample set may be a data set of an autonomous driving vehicle obtained from a third party, and the data set is decomposed into single-frame images as sample images, the sample images are semantically segmented, the objects to be identified in the sample images are marked, and a sample set for model training is constructed. The marking information may be a mark for marking the objects to be identified in the form of text content, digital codes, boxes, lines, dots, etc.

[0027] For example, in the application scenario of autonomous driving vehicles, the sample set obtained contains multiple sample images with information such as pedestrians, bicycles, vehicles, traffic signs, signal lights, lane lines, and road edges. Among them, pedestrians, bicycles, and vehicles are all dynamic objects, so the sample images marked with pedestrians, bicycles, and vehicles in the sample set belong to the dynamic object class image; traffic signs and signal lights have more significant features in color to distinguish them from background objects, so the sample images marked with traffic signs and signal lights in the sample set belong to the color feature class image; lane lines and road edges have more significant features in geometric shape to distinguish them from background objects, so the sample images marked with lane lines and road edges in the sample set belong to the geometric shape class image.

[0028] The sample set may be sample images acquired from different environments, such as sample images acquired under environments with different illumination, background, viewing angle, weather, etc.

[0029] Step S120: performing a first feature extraction operation on the sample images in the sample set to obtain a first image feature, where the first image feature is a common feature of sample images of multiple different categories.

[0030] In an embodiment proposed in the present application, the first feature extraction operation is performed on the sample images in the sample set to obtain the first image feature. The first feature extraction operation can be performed on the sample images in the sample set using HOG (histogram of oriented gradients), SIFT (scale-invariant feature transform), SURF (speeded up robust features), DOG (difference of Gaussian functions), LBP (local binary patterns), HAAR (haar-like features) and other methods to obtain the first image feature; or the first feature extraction operation can be performed on the sample images in the sample set using different neural networks to obtain the first image feature.

[0031] In one embodiment of the present application, the first image feature is a common feature of sample images of multiple different categories, which can be a low-level visual feature of the image including texture, color, and shape. Among them, the texture feature extraction algorithms include: Tamura texture feature, autoregressive texture model, Gabor transform, wavelet transform, MPEG7 edge histogram, gray level co-occurrence matrix method, Fourier power spectrum method, etc.; the color feature extraction algorithms include: color histogram method, color set, color moment, color aggregation vector, cumulative histogram method, color clustering method, etc.; the shape feature extraction algorithms include: canny algorithm, Fourier shape descriptor, invariant moment, wavelet contour description, etc.

[0032] Step S130: performing a second feature extraction operation on sample images belonging to the same category to obtain second image features corresponding to sample images of each category.

[0033] In an embodiment of the present application, the sample images belonging to the same category can be divided into dynamic object images, color feature images and geometric shape images; they can also be divided according to the depth information and contour information of the image.

[0034] In one embodiment proposed in the present application, the second feature extraction operation is performed on sample images belonging to the same category, and one or more networks that can extract image features, such as convolutional neural networks, deep residual networks, deep residual shrinkage networks, forward feedback networks, Hopfield networks, inverse image networks, AlexNet networks, vgg networks, googlenet networks, resnet networks, etc., are used to perform the second feature extraction operation on sample images belonging to the same category to obtain second image features corresponding to sample images of each category.

[0035] In an embodiment of the present application, the second image feature is a common feature of sample images of the same category, and can be a high-level semantic feature of the image. High-level semantic features are abstract features, and the high-level semantic features of the image can include the object layer and the concept layer of image semantics. Among them, the object layer usually includes image attribute features, etc., which can be understood as the state of an object at a certain moment; the concept layer is what the image expresses that is closest to human understanding. For example, if there are sand, blue sky, sea water, etc. on a picture, the visual layer is a distinction between pieces, the object layer is sand, blue sky and sea water, and the concept layer is the beach. High-level semantic feature extraction methods include: semantic network, mathematical logic, framework and other methods.

[0036] In some embodiments, the high-level semantic feature is feature information obtained after several convolutions (feature extractions). Its receptive field is large, and the extracted features are increasingly abstract, which is beneficial for object classification. However, it will also lose detail information, which is not conducive to accurate segmentation.

[0037] Step S140: training a multi-task learning model based on the first image feature, each of the second image features and the label information to obtain a target multi-task learning model.

[0038] In an embodiment of the present application, the multi-task model may refer to a neural network model or a variation of a neural network model, such as a convolutional neural network model, YoloV3, YoloV4, YoloV5, SSD, Mobilenet, RetinaNet, CenterNet, CornerNet, FastRCNN, FasterRCNN, MaskRCNN, CascadeRCNN, etc.

[0039] See also Figure 2 In some embodiments, the multi-task learning model includes a first feature extraction network and multiple second feature extraction networks, wherein each second feature extraction network corresponds to an image category.

[0040] In some embodiments, the multi-task learning model further includes a plurality of information integration networks, and each second feature extraction network corresponds to one information integration network. The multi-task learning model is trained based on the first image feature, each of the second image features, and the label information to obtain a target multi-task learning model, which can be understood as using each of the information integration networks to process the first image feature and the second image feature extracted by the second feature extraction network corresponding to the information integration network to obtain a prediction result; and the model parameters of the multi-task learning model are adjusted according to the prediction results corresponding to each of the information integration networks and the label information in the sample image to obtain a target multi-task learning model.

[0041] In some embodiments, the first feature extraction network, the second feature extraction network, and the information integration network can be one or more neural networks such as a feedforward network, a Hopfield network, a convolutional neural network, an inverse image network, an AlexNet network, a vgg network, a googlenet network, and a resnet network that can extract image features.

[0042] In the embodiment of the present application, the performing of a first feature extraction operation on the sample images in the sample set to obtain a first image feature can be understood as using a first feature extraction network to perform a first feature extraction operation on the sample images in the sample set to obtain the first image feature; the performing of a second feature extraction operation on sample images belonging to the same category to obtain second image features corresponding to sample images of each category can be understood as using each second feature extraction network based on the first image feature to perform a second feature extraction operation on sample images in the image category corresponding to the second feature extraction network to obtain second image features corresponding to sample images of each category.

[0043] Among them, the use of each of the information integration networks to process the first image feature and the second image feature extracted by the second feature extraction network corresponding to the information integration network to obtain a prediction result can be to use each of the information integration networks to fuse the first image feature and the second image feature extracted by the second feature extraction network corresponding to the information integration network to obtain a fused feature, and obtain a prediction result based on the fused feature.

[0044] In some embodiments, the information integration networks are used to fuse the first image features and the second image features extracted by the second feature extraction network corresponding to the information integration network. The fusion levels can be divided into three parts: pixel level, feature level and decision level. Pixel-level image fusion is directly fused on the original data layer. This fusion has the highest accuracy and has more accurate, richer and more reliable detail information, which is conducive to further understanding and analysis of the image. Pixel-level image fusion is the basis of feature-level fusion and decision-level fusion; feature-level image fusion belongs to the middle level and prepares for decision-level image fusion. It retains important information and compresses the information; decision-level image fusion refers to classifying, identifying and other processing of the feature information of each image to form corresponding results, and further fusion engineering is carried out. The final decision result is the global optimal decision. Decision-level image fusion is a higher-level information fusion, and its results will provide a basis for various controls or decisions.

[0045] It should be understood that the fused feature includes the first image feature and the second image feature extracted by the second feature extraction network corresponding to the information integration network, and the fused feature performs the extraction of the information integration network and outputs the prediction result of the task execution corresponding to the information integration network. The multi-task learning model is trained based on the first image feature, each of the second image features and the label information to obtain a target multi-task learning model.

[0046] In some embodiments, the prediction result may be a recognition result of a dynamic object in a dynamic object class image in the sample set, a recognition result of an object with color features in a color feature class image in the sample set, or a recognition result of an object with geometric features in a geometric shape class image in the sample set.

[0047] Exemplarily, the sample set is a sample image set of pedestrians, vehicles, bicycles, traffic lights, traffic signs, lane lines, and road edges in a certain autonomous driving scene. The first feature extraction network neural network is used to perform a first feature extraction operation on the sample images in the sample set to obtain a first image feature, and the first image feature includes the common features of the sample images in the autonomous driving scene, such as the contours, positions, and color information of objects such as roads, pedestrians, vehicles, bicycles, traffic lights, traffic signs, lane lines, and road edges. Among them, pedestrians, vehicles, and bicycles in the sample images are dynamic objects, and they are highly dependent on the depth information and contour information of the image, and have certain commonalities. Sample images belonging to the same category correspond to one second feature extraction network; traffic signs and signal lights are both artificially designed, and have more significant features in color that are different from background objects, and have certain commonalities. Sample images belonging to the same category correspond to one second feature extraction network; lane lines and road edges are both recognition of the geometric shape (straight line) of the edge of an object, and have certain commonalities. Sample images belonging to the same category correspond to one second feature extraction network. Based on the first image feature, each second feature extraction network is used to perform a second feature extraction operation on the sample image in the image category corresponding to the second feature extraction network, and obtain the second image features corresponding to each category of sample images. For example, feature outputs of pedestrians, vehicles, bicycles, traffic signs, signal lights, lane lines, and road edges are obtained respectively. The information integration network performs feature-level and decision-level fusion on the second image features obtained by the first feature extraction network and the second feature extraction network corresponding to the information integration network to obtain fused features, and outputs prediction results based on the fused features. The prediction results are output as recognition results corresponding to each task in the target multi-task learning model, such as the position, color, and contour of pedestrians, traffic signs, and lane lines.

[0048] In some embodiments, the model parameters of the multi-task learning model are adjusted according to the prediction results corresponding to each of the information integration networks and the label information in the sample image to obtain a target multi-task learning model. The organizational form of the second feature extraction network can be a parallel structure or a branch bypass structure, and adjusting the model parameters of the task learning model includes adjusting the parameters of the second feature extraction network and adjusting the organizational form of the second feature extraction network.

[0049] In some embodiments, the training of the multi-task learning model may be adjusting the parameters of the neural network included in the multi-task learning model, replacing the type of the neural network, or adjusting the structure of the multi-task learning model.

[0050] In some embodiments, after the target multi-task learning model is designed using the model training method, similar subtasks are jointly trained under the same sample set to give full play to the advantages of similar subtasks jointly improving task performance.

[0051] A model training method proposed in an embodiment of the present application obtains a sample set of multiple sample images with labeling information, performs a first feature extraction operation on the sample images in the sample set, and obtains a first image feature, which is a common feature of sample images of multiple different categories; performs a second feature extraction operation on sample images belonging to the same category, and obtains second image features corresponding to sample images of each category; trains a multi-task learning model based on the first image feature, each of the second image features, and the labeling information to obtain a target multi-task learning model. The target multi-task learning model generated by the model training method enables each subtask in the multi-task learning model to share the same feature extraction network to reduce hardware computing overhead and to adapt to the computing power of the hardware; enables similar subtasks to share the same depth of feature extraction network, and jointly trains similar subtasks; each subtask corresponds to an independently designed information integration network to obtain different outputs for all tasks and improve the performance of each subtask.

[0052] See also Figure 4 , an image recognition method proposed in an embodiment of the present application includes steps S210 to S220.

[0053] Step S210: Acquire an image to be recognized.

[0054] In some embodiments, the image to be identified is an image collected by a vehicle, and the image includes at least one of a dynamic object image, an object image filled with color features, and an object image with geometric features.

[0055] Exemplarily, the image to be identified is an image collected by a vehicle in an autonomous driving scenario, and the image includes: pedestrians, vehicles, bicycles, traffic lights, traffic signs, lane lines, road edges, etc., among which pedestrians, vehicles, and bicycles are dynamic objects; traffic lights and traffic signs are objects filled with color features; lane lines and road edges are objects with geometric features.

[0056] Step S220: using a target multi-task learning model to identify the image to be identified, and obtaining a recognition result corresponding to the image to be identified, wherein the target multi-task learning model is generated by the model training method described in any of the above embodiments.

[0057] In an embodiment of the present application, the target multi-task learning model includes a first feature extraction network, multiple second feature extraction networks and multiple information integration networks, and each second feature extraction network corresponds to an information integration network.

[0058] In some embodiments, the first feature extraction network, the multiple second feature extraction networks, and the multiple information integration networks can be any one or more networks that can extract image features, such as a feedforward network, a Hopfield network, a convolutional neural network, an inverse image network, an AlexNet network, a vgg network, a googlenet network, a resnet network, etc.

[0059] In some embodiments, the use of the target multi-task learning model to identify the image to be identified and obtain a recognition result corresponding to the image to be identified may be to use the first feature extraction network to perform a first feature extraction operation on the image to be identified to obtain a first target image feature; use each second feature extraction network to extract features from the first target image feature respectively to obtain second target image features corresponding to each second feature extraction network respectively; use each information integration network to process the first target image feature and the second target image features extracted by the second feature extraction network corresponding to the information integration network to obtain image recognition results corresponding to each information integration network respectively.

[0060] For the generation process of the target multi-task learning model, please refer to the detailed description of steps S110 to S140 above, which will not be repeated here.

[0061] In some embodiments, the first target image feature can be understood as the basic visual feature of the image to be recognized, and the second target image feature can be understood as the high-level semantic feature of the image to be recognized. Figure 4 ,like Figure 4The image to be identified shown includes: pedestrians, vehicles, bicycles, signal lights, traffic signs, lane lines, road edges and other contents. The first feature extraction operation in the target multi-task learning model is performed on the image to be identified to obtain the first target image feature, which includes the contour, position, color and other visual features of pedestrians, vehicles, bicycles, signal lights, traffic signs, lane lines, and road edges; the second feature extraction networks are used to extract features from the image to be identified to obtain the second target image features corresponding to the second feature extraction networks. Among them, there are second target image features including dynamic objects such as pedestrians, vehicles, and bicycles; there are second target image features including objects filled with color features such as signal lights and traffic signs; there are second image features including objects with geometric features such as lane lines and road edges.

[0062] In some embodiments, each of the information integration networks is used to process the first target image features and the second target image features extracted by the second feature extraction network corresponding to the information integration network to obtain image recognition results corresponding to each of the feature extraction networks. The information integration networks may be used to fuse the first target image features and the second target image features extracted by the second feature extraction network corresponding to the information integration network.

[0063] Among them, the levels of image fusion can be divided into three parts: pixel-level image fusion that is directly fused on the original data layer; feature-level image fusion that prepares for decision-level image fusion, retains important information, and compresses the information; and decision-level image fusion that classifies, identifies, and processes the feature information of each image to form corresponding results and conduct further fusion engineering. In the embodiment of the present application, the information integration network mainly performs feature-level and decision-level image fusion, and uses the temporal and spatial correlation of images and the complementarity of information to make the fusion features obtained after fusion have a more comprehensive and clearer description of the scene, which is more conducive to human eye recognition or machine automatic detection. Finally, the information integration network converts the fusion features into electrical signals or other forms of signal output as the recognition result of the image to be recognized.

[0064] Exemplarily, the first target image features include image features such as the outline, position, and color of objects such as pedestrians, vehicles, bicycles, signal lights, traffic signs, lane lines, and road edges; each of the second feature extraction networks corresponds to second target image features of dynamic objects such as pedestrians, vehicles, and bicycles, second target image features of objects filled with color features such as signal lights and traffic signs, and second image features of objects with geometric features such as lane lines and road edges. Each of the information integration networks fuses the second target image features corresponding to the information integration network with the first target image features, and outputs recognition results of pedestrians, vehicles, bicycles, signal lights, traffic signs, lane lines, road edges, and other objects, respectively, and the recognition results are electrical signals or signals in other forms.

[0065] See also Figure 5 , which shows a model training device provided by an embodiment of the present invention, the model training device 300 includes: a sample set acquisition unit 310, a first feature extraction unit 320, a second feature extraction unit 330 and a model generation unit 340. Specifically, the sample set acquisition unit 310 is used to acquire a sample set, the sample set includes a plurality of sample images with label information, and the plurality of sample images belong to a plurality of image categories respectively; the first feature extraction unit 320 is used to perform a first feature extraction operation on the sample images in the sample set to obtain a first image feature, the first image feature is a common feature of sample images of a plurality of different categories; the second feature extraction unit 330 is used to perform a second feature extraction operation on sample images belonging to the same category to obtain second image features corresponding to sample images of each category respectively; the model generation unit 340 is used to train a multi-task learning model based on the first image feature, each of the second image features and the label information to obtain a target multi-task learning model.

[0066] As an implementation manner, the sample set acquisition unit 310 is used to acquire a sample set, which is a sample image set for training a multi-task learning model. The sample set includes multiple sample images with labeling information. The sample set may be an autonomous driving data set acquired from a third party, and the data set is decomposed into single-frame images, and semantic segmentation is performed on the single-frame images to label the objects to be identified in the single-frame images, and a sample set for model training is constructed. The labeling information may be a mark for marking the objects to be identified in the form of text content, digital code, box, line, point, etc. Among them, the multiple image categories include at least two of dynamic object images, color feature images, and geometric shape images.

[0067] In some embodiments, the first feature extraction unit 320 may perform a first feature extraction operation on the sample images in the sample set using methods such as HOG (Histogram of Oriented Gradients), SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), DOG (Differential Gaussian Function), LBP (Local Binary Pattern), HAAR (Haar-like Features), etc. to obtain the first image feature; or may perform a first feature extraction operation on the sample images in the sample set using a neural network to obtain the first image feature. The neural network may be one or more neural networks that can extract image features, such as a convolutional neural network, a deep residual network, a deep residual shrinkage network, a forward feedback network, a Hopfield network, an inverse image network, an AlexNet network, a vgg network, a googlenet network, a resnet network, etc. The first image feature may be a low-level visual feature of an image, including texture, color, and shape.

[0068] In some embodiments, the second feature extraction unit 330 may be a neural network that performs a second feature extraction operation on the sample images in the sample set to obtain a second image feature. The neural network may be a convolutional neural network, a deep residual network, a deep residual shrinkage network, a forward feedback network, a Hopfield network, an inverse image network, an AlexNet network, a vgg network, a googlenet network, a resnet network, or one or more neural networks that can extract image features. The second image feature is a common feature of sample images of the same category that can be extracted by utilizing the temporal and spatial correlation and information complementarity of sample images of the same category to more comprehensively and clearly describe the scene. It is more abstract and may be a high-level semantic feature of the image.

[0069] In some embodiments, the model generation unit 340 trains a multi-task learning model based on the first image feature, each of the second image features, and the label information. The multi-task learning model includes a first feature extraction network, multiple second feature extraction networks, and multiple information integration networks, each second feature extraction network corresponds to an image category, and each second feature extraction network corresponds to an information integration network. The training of the multi-task learning model may be to adjust the parameters of the first feature extraction network, the second feature extraction network, and the information integration network, or to adjust the organizational form of the multi-task model.

[0070] For the specific execution process of the model training device 300, please refer to the specific description of steps S110 to S140 above, which will not be repeated here.

[0071] See also Figure 6, which shows an image recognition device provided by an embodiment of the present invention, the image recognition device 400 includes: an image acquisition unit 410 and a recognition unit 420. Specifically, the image acquisition unit 410 is used to acquire an image to be recognized; the recognition unit 420 is used to recognize the image to be recognized using a target multi-task learning model to obtain a recognition result corresponding to the image to be recognized, and the target multi-task learning model is generated by the model training method of any of the above embodiments.

[0072] In some embodiments, the target multi-task learning model includes a first feature extraction network, multiple second feature extraction networks and multiple information integration networks, and each second feature extraction network corresponds to an information integration network respectively. The use of the target multi-task learning model to identify the image to be identified and obtain a recognition result corresponding to the image to be identified can be to use the first feature extraction network to perform a first feature extraction operation on the image to be identified to obtain a first target image feature; use each second feature extraction network to perform feature extraction on the first target image feature respectively to obtain a second target image feature corresponding to each second feature extraction network; use each information integration network to process the first target image feature and the second target image feature extracted by the second feature extraction network corresponding to the information integration network to obtain an image recognition result corresponding to each information integration network.

[0073] The specific execution process of the image recognition device 400 can refer to the specific description of step S210 to step S220 above, which will not be repeated here.

[0074] It should be noted that each embodiment of this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Any processing method described in the method embodiment can be implemented by the corresponding processing module in the device embodiment, and will not be repeated one by one in the device embodiment.

[0075] See also Figure 7Based on the above model training method or image recognition method, the present application also provides another electronic device 500 that can execute the above model training method or image recognition method, and the electronic device 500 includes: one or more processors 510, a memory 520, and one or more application programs. Among them, the memory 520 stores a program that can execute the content of the above embodiment, and the processor 510 can execute the program stored in the memory 520. Among them, the electronic device 500 can be an intelligent control panel, a smart phone, a smart wearable device, an intelligent voice navigation device, an intelligent robot, a tablet computer, a personal computer, a vehicle, etc.

[0076] Among them, the processor 510 may include one or more cores for processing data and a message matrix unit. The processor 510 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions and processes data of the electronic device 500 by running or executing instructions, programs, code sets or instruction sets stored in the memory 520, and calling data stored in the memory 520. Optionally, the processor 510 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 510 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a communication chip.

[0077] The memory 520 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). The memory 520 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 520 may include a program storage area and a data storage area, wherein the program storage area may store the code of the model training method or image recognition method described in the above method embodiment, and instructions for implementing at least one function (such as obtaining an image to be recognized, etc.). The data storage area may also store a target multi-task learning model created by the terminal during use, etc.

[0078] See also Figure 8 , which shows a structural block diagram of a computer-readable storage medium 600 provided in an embodiment of the present application. The computer-readable storage medium 600 stores a program code 610, which can be called by a processor to execute the model training method or image recognition method described in the above method embodiment.

[0079] The computer readable storage medium 600 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 600 includes a non-transitory computer-readable storage medium. The computer readable storage medium 600 has storage space for program codes 610 that perform any method steps in the above method. These program codes 610 can be read from or written to one or more computer program products. The program codes 610 can be compressed, for example, in an appropriate form.

[0080] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a sample set, wherein the sample set includes a plurality of sample images with label information, and the plurality of sample images respectively belong to a plurality of image categories; Performing a first feature extraction operation on the sample images in the sample set to obtain a first image feature, where the first image feature is a common feature of sample images of multiple different categories; Performing a second feature extraction operation on sample images belonging to the same category to obtain second image features corresponding to sample images of each category; Training a multi-task learning model based on the first image feature, each of the second image features, and the label information to obtain a target multi-task learning model; The target multi-task learning model includes a first feature extraction network, multiple second feature extraction networks and multiple information integration networks, and each second feature extraction network corresponds to an information integration network. The multi-task learning model is trained based on the first image feature, each of the second image features and the label information to obtain the target multi-task learning model, including: Using each of the information integration networks to process the first image features and the second image features extracted by the second feature extraction network corresponding to the information integration network to obtain a prediction result; The model parameters of the multi-task learning model are adjusted according to the prediction results corresponding to each of the information integration networks and the label information in the sample image to obtain a target multi-task learning model.

2. The model training method according to claim 1, characterized in that: Each second feature extraction network corresponds to an image category, and performing a first feature extraction operation on a sample image in the sample set to obtain a first image feature includes: Using a first feature extraction network to perform a first feature extraction operation on a sample image in the sample set to obtain a first image feature; The performing of the second feature extraction operation on the sample images belonging to the same category to obtain the second image features corresponding to the sample images of each category respectively includes: Based on the first image features, each second feature extraction network is used to perform a second feature extraction operation on the sample images in the image category corresponding to the second feature extraction network to obtain second image features corresponding to the sample images of each category.

3. The model training method according to claim 1, characterized in that: The using each of the information integration networks to process the first image feature and the second image feature extracted by the second feature extraction network corresponding to the information integration network to obtain a prediction result includes: Each of the information integration networks is used to fuse the first image feature and the second image feature extracted by the second feature extraction network corresponding to the information integration network to obtain a fused feature, and a prediction result is obtained based on the fused feature.

4. The model training method according to claim 1, characterized in that: The multiple image categories include at least two of dynamic object images, color feature images, and geometric shape images.

5. An image recognition method, characterized in that: The method comprises: Obtain an image to be recognized; The image to be identified is identified using a target multi-task learning model to obtain a recognition result corresponding to the image to be identified, wherein the target multi-task learning model is trained based on the method described in any one of claims 1 to 4.

6. The image recognition method according to claim 5, characterized in that: The target multi-task learning model includes a first feature extraction network, multiple second feature extraction networks and multiple information integration networks, and each second feature extraction network corresponds to an information integration network. The target multi-task learning model is used to identify the image to be identified, and a recognition result corresponding to the image to be identified is obtained, including: Using the first feature extraction network to perform a first feature extraction operation on the image to be identified, to obtain a first target image feature; Using each second feature extraction network to extract features of the image to be identified, respectively, to obtain second target image features corresponding to each second feature extraction network; The first target image features and the second target image features extracted by the second feature extraction network corresponding to the information integration network are processed by each of the information integration networks to obtain image recognition results corresponding to each of the feature extraction networks.

7. The image recognition method according to claim 6, characterized in that: The image to be identified is an image collected by a vehicle, and the image includes at least one of a dynamic object image, an object image filled with color features, and an object image with geometric features.

8. A model training device, characterized in that: The device comprises: A sample set acquisition unit, used to acquire a sample set, wherein the sample set includes a plurality of sample images with label information, and the plurality of sample images respectively belong to a plurality of image categories; A first feature extraction unit, configured to perform a first feature extraction operation on the sample images in the sample set to obtain a first image feature, where the first image feature is a common feature of sample images of multiple different categories; A second feature extraction unit, used to perform a second feature extraction operation on sample images belonging to the same category, to obtain second image features corresponding to the sample images of each category; A model generation unit is used to train a multi-task learning model based on the first image feature, each of the second image features and the labeling information to obtain a target multi-task learning model; wherein the target multi-task learning model includes a first feature extraction network, multiple second feature extraction networks and multiple information integration networks, and each second feature extraction network corresponds to an information integration network respectively, and the multi-task learning model is trained based on the first image feature, each of the second image features and the labeling information to obtain the target multi-task learning model, including: using each of the information integration networks to process the first image feature and the second image feature extracted by the second feature extraction network corresponding to the information integration network to obtain a prediction result; adjusting the model parameters of the multi-task learning model according to the prediction results corresponding to each of the information integration networks and the labeling information in the sample image to obtain the target multi-task learning model.

9. An image recognition device, characterized in that: The device comprises: An image acquisition unit, used for acquiring an image to be recognized; A recognition unit is used to recognize the image to be recognized by using a target multi-task learning model to obtain a recognition result corresponding to the image to be recognized, wherein the target multi-task learning model is trained based on any one of the methods described in claims 1-4.

10. An electronic device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images