Vehicle parts segmentation method and related equipment based on knowledge distillation
By using knowledge distillation technology in vehicle component segmentation, using labeling and unlabeled data to train the teacher network and supervise the student network, the problem of poor segmentation effect of traditional models on the mobile terminal is solved, and more efficient vehicle component identification is achieved.
Patent Information
- Application Number
- CN202210791176.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-20
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-06-20
AI Technical Summary
When traditional deep learning models are used for vehicle parts segmentation in mobile scenarios, the results are poor, mainly due to the large amount of parameters, resulting in insufficient speed and accuracy.
Using knowledge distillation method, by building an initial teacher network and student network, double training is performed using labeled data and unlabeled data, dark knowledge is extracted to supervise the training of the student network, and a second student network with small parameters and high segmentation accuracy is obtained. , and embed it into the mobile device.
The speed and accuracy of vehicle component segmentation in mobile scenarios are improved, and more efficient vehicle component identification is achieved.
Smart Images

Figure CN115063589B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a vehicle component segmentation method, device, electronic device and storage medium based on knowledge distillation. Background Art
[0002] Knowledge distillation is a model compression method. The knowledge distillation system includes a teacher network with good performance but large parameters and a student network with small parameters. Knowledge is extracted from the teacher network to obtain supervision information to train the student network, so that the student network can achieve better performance.
[0003] Vehicle part segmentation is widely needed in scenarios such as intelligent damage assessment and vehicle acceptance. Deep learning models are usually used to segment the collected vehicle images to obtain the segmentation results of vehicle parts. However, due to factors such as mobile terminal performance and the large number of deep learning model parameters, traditional deep learning models often do not work well in mobile scenarios. Therefore, it is necessary to design the structure and training method of the deep learning model for vehicle part segmentation scenarios to improve the speed and accuracy of vehicle part segmentation in mobile scenarios. Summary of the invention
[0004] In view of the above content, it is necessary to propose a vehicle parts segmentation method and related equipment based on knowledge distillation to solve the technical problem of how to improve the speed and accuracy of vehicle parts segmentation in mobile scenarios. The related equipment includes a vehicle parts segmentation device based on knowledge distillation, an electronic device and a storage medium.
[0005] The present application provides a vehicle parts segmentation method based on knowledge distillation, the method comprising:
[0006] Collecting vehicle images with labeled data as a labeled data set, collecting vehicle images without labeled data as an unlabeled data set, and using the labeled data set and the unlabeled data set as training sets, wherein the labeled data includes the type of vehicle parts for each pixel in the vehicle image;
[0007] Building an initial teacher network, and training the initial teacher network based on the training set to obtain a second teacher network;
[0008] Building an initial student network;
[0009] Extracting dark knowledge of each vehicle image in the training set based on the second teacher network, and training the initial student network based on the dark knowledge to obtain a second student network, wherein the dark knowledge reflects feature similarity and position correlation between different types of vehicle parts;
[0010] A component segmentation result of a real-time vehicle image is obtained based on the second student network.
[0011] In some embodiments, the training the initial teacher network based on the training set to obtain a second teacher network comprises:
[0012] Training the initial teacher network based on the labeled data set and the cross entropy loss function to obtain a first teacher network;
[0013] Based on the first teacher network, a component segmentation result of each vehicle image in the unlabeled data set is obtained, wherein the component segmentation result includes a category vector of each pixel in the vehicle image, and the category vector includes a probability value of the pixel belonging to each type of vehicle component;
[0014] Calculate the credibility index of each component segmentation result according to the credibility calculation model;
[0015] Screening the vehicle images in the unlabeled data set based on the credibility index and a preset credibility threshold to obtain abnormal images;
[0016] The label data of all abnormal images are obtained, and the first teacher network is trained based on the abnormal images and the label data of the abnormal images to obtain a second teacher network.
[0017] In some embodiments, the credibility calculation model satisfies the relationship:
[0018]
[0019] Where N is the number of all vehicle parts, W×H is the width and height of the part segmentation result k, is the probability value of the category vector of the pixel point (i, j) in the component segmentation result k belonging to the vehicle component type c, α k is the credibility index of the component segmentation result k, and its value range is (0,1]. The larger the value, the greater the credibility of the component segmentation result k and the higher the accuracy of vehicle component segmentation.
[0020] In some embodiments, screening the vehicle images in the unlabeled dataset based on the credibility index and the preset credibility threshold to obtain abnormal images includes:
[0021] comparing the credibility indicator with a preset credibility threshold;
[0022] If the credibility index is greater than the preset credibility threshold, it means that the accuracy of the component segmentation result corresponding to the credibility index meets the requirement, and the vehicle image corresponding to the component segmentation result is marked as a normal image;
[0023] If the credibility index is not greater than the preset credibility threshold, it means that the accuracy of the component segmentation result corresponding to the credibility index does not meet the requirement, and the vehicle image corresponding to the component segmentation result is marked as an abnormal image.
[0024] In some embodiments, extracting dark knowledge of each vehicle image in the training set based on the second teacher network, and training the initial student network based on the dark knowledge to obtain a second student network, wherein the dark knowledge reflects feature similarity and position correlation between different types of vehicle parts, includes:
[0025] Inputting a target vehicle image into the second teacher network and the initial student network at the same time, the target vehicle image is any one of all vehicle images in the training set, using the output result of the second teacher network as the teacher component segmentation result, and using the output result of the initial student network as the student component segmentation result, and both the teacher component segmentation result and the student component segmentation result include a category vector for each pixel in the target vehicle image;
[0026] Constructing a position correlation matrix of the target vehicle image based on the teacher component segmentation result;
[0027] Using the category vector of the teacher component segmentation result and the position correlation matrix as dark knowledge of the target vehicle image, and constructing a first preset loss function based on the dark knowledge;
[0028] Determine whether the target vehicle image has label data to obtain a judgment parameter, if the target vehicle image has label data, construct a cross entropy loss function based on the label data and the student component segmentation result, and the value of the judgment parameter is 1; if the target vehicle image does not have label data, the value of the judgment parameter is 0;
[0029] Constructing a preset loss function based on the judgment parameter and the first preset loss function;
[0030] The initial student network is trained based on the preset loss function and all vehicle images in the training set to obtain a second student network.
[0031] In some embodiments, constructing a position correlation matrix of the target vehicle image based on the teacher component segmentation result includes:
[0032] Selecting the vehicle component type corresponding to the maximum probability value in the category vector of the teacher component segmentation result as the vehicle component type of each pixel;
[0033] The pixel values of the pixels of the same vehicle component type are set to 1, and the pixel values of other regions are set to 0 to obtain the regional image of the vehicle component, and all the vehicle component types in the teacher component segmentation result are traversed to obtain the regional image of each vehicle component;
[0034] Arranging pixel values in the regional image of the vehicle parts in a fixed order along the row direction to obtain a position vector of each vehicle part in the target vehicle image;
[0035] Based on the position vector, the position correlation of different vehicle components is calculated to construct a position correlation matrix of the target vehicle image, and the position correlation satisfies the relationship:
[0036] x mn =h m (h n ) T
[0037] Among them, h m is the position vector of the vehicle component m in the target vehicle image, (h n ) T is the transpose of the position vector of vehicle component n in the target vehicle image, x mn is the position correlation between vehicle component m and vehicle component n.
[0038] In some embodiments, in constructing a preset loss function based on the judgment parameter and the first preset loss function, the preset loss function satisfies the relationship:
[0039]
[0040] Wherein, Loss1 is the first preset loss function, Loss2 is the cross entropy loss function constructed based on the label data and the student component segmentation result, β is a judgment parameter, the value is 0 or 1, and Loss is the preset loss function;
[0041] The first preset loss function satisfies the relationship:
[0042]
[0043] Among them, X s is the position correlation matrix of the student component segmentation result, X is the position correlation matrix of the target vehicle image, ‖X s -X‖2 means X s -L2 distance of X; P s (i, j) is the category vector of the pixel point (i, j) in the student component segmentation result, P(i, j) is the category vector of the pixel point (i, j) in the teacher component segmentation result, KL(P s(i,j),P(i,j)) is used to calculate P s (i,j), KL divergence between P(i,j); W, H are the width and height of the target vehicle image.
[0044] The embodiment of the present application also provides a vehicle parts segmentation device based on knowledge distillation, the device comprising:
[0045] A collection unit, used to collect vehicle images with labeled data as a labeled data set, collect vehicle images without labeled data as an unlabeled data set, and use the labeled data set and the unlabeled data set as training sets, wherein the labeled data includes the type of vehicle parts for each pixel in the vehicle image;
[0046] A training unit, configured to build an initial teacher network, and train the initial teacher network based on the training set to obtain a second teacher network;
[0047] Building unit, used to build the initial student network;
[0048] an extraction unit, configured to extract dark knowledge of each vehicle image in the training set based on the second teacher network, and train the initial student network based on the dark knowledge to obtain a second student network, wherein the dark knowledge reflects feature similarity and position correlation between different types of vehicle components;
[0049] A segmentation unit is used to obtain a component segmentation result of a real-time vehicle image based on the second student network.
[0050] The present application also provides an electronic device, the electronic device comprising:
[0051] A memory storing at least one instruction;
[0052] A processor executes instructions stored in the memory to implement the vehicle parts segmentation method based on knowledge distillation.
[0053] An embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the vehicle component segmentation method based on knowledge distillation.
[0054] In summary, this application trains the initial teacher network twice based on labeled data and unlabeled data to obtain a teacher network, and uses the teacher network to extract the dark knowledge of each vehicle image in the labeled data and the unlabeled data to supervise the training of the lightweight initial student network, thereby obtaining a second student network with a small number of parameters and high segmentation accuracy. Embedding the second student network into a mobile device can improve the speed and accuracy of vehicle parts segmentation in mobile scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is a flowchart of a preferred embodiment of the vehicle parts segmentation method based on knowledge distillation involved in the present application.
[0056] Figure 2 It is a schematic diagram of the training process of the initial student network involved in this application.
[0057] Figure 3 It is a functional module diagram of a preferred embodiment of the vehicle component segmentation device based on knowledge distillation involved in the present application.
[0058] Figure 4 It is a schematic diagram of the structure of an electronic device of a preferred embodiment of the vehicle component segmentation method based on knowledge distillation involved in the present application. DETAILED DESCRIPTION
[0059] In order to more clearly understand the purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, many specific details are set forth to facilitate a full understanding of the present application, and the embodiments described are only a part of the embodiments of the present application, rather than all of the embodiments.
[0060] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.
[0062] The embodiment of the present application provides a vehicle component segmentation method based on knowledge distillation, which can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0063] An electronic device can be any electronic product that can interact with a customer, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.
[0064] The electronic device may also include a network device and / or a client device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.
[0065] The network where the electronic device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.
[0066] like Figure 1 FIG. 1 is a flowchart of a preferred embodiment of the vehicle component segmentation method based on knowledge distillation of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.
[0067] S10, collecting vehicle images with labeled data as a labeled data set, collecting vehicle images without labeled data as an unlabeled data set, and using the labeled data set and the unlabeled data set as training sets, wherein the labeled data includes the type of vehicle parts for each pixel in the vehicle image.
[0068] In an optional embodiment, a large number of vehicle images are collected, and label data of each vehicle image is obtained, wherein the label data is an image of the same size as the vehicle image, and the pixel value in the label data represents a preset label of the type of vehicle component at each pixel point, and the preset label is an integer from 1 to N, wherein N represents the number of all types of vehicle components including the background class, and the background class is regarded as a special type of vehicle component. The background class is the category of pixel points in the vehicle image that do not belong to vehicle components, and the vehicle component types correspond one-to-one to the preset labels, and all vehicle images and the label data of all vehicle images are stored as a labeled data set.
[0069] In this optional embodiment, a large number of new vehicle images are collected without obtaining label data of the new vehicle images. All new vehicle images are directly stored as unlabeled data sets, and the labeled data sets and the unlabeled data sets are used as training sets for subsequent training of the teacher network and the student network.
[0070] In this way, labeled data sets and unlabeled data sets are collected as training sets, providing a data basis for the subsequent training of teacher networks and student networks.
[0071] S11, building an initial teacher network, and training the initial teacher network based on the training set to obtain a second teacher network.
[0072] In an optional embodiment, an initial teacher network is constructed, wherein the input of the initial teacher network is a vehicle image, and the expected output is a component segmentation result of the vehicle image, wherein the component segmentation result is an image of the same size as the vehicle image, and the component segmentation result includes a category vector for each pixel in the vehicle image, wherein the category vector includes a probability value that the pixel belongs to each type of vehicle component, and the sum of all probability values in the category vector of the same pixel is 1; the type of vehicle component corresponding to the maximum probability value in the category vector is selected as the type of vehicle component corresponding to the pixel.
[0073] In this optional embodiment, the initial teacher network is an encoder and decoder structure, the encoder uses a convolution layer to downsample the input vehicle image to obtain a teacher feature map, and sends the teacher feature map to the decoder to use a deconvolution layer for upsampling to obtain the component segmentation result of the vehicle image. It should be noted that the vehicle component segmentation accuracy of the initial teacher network directly affects the segmentation accuracy of the subsequent student network, so there is no need to consider the parameter amount of the initial teacher network to ensure that the vehicle component segmentation accuracy of the initial teacher network is the top priority condition. Therefore, the initial teacher network can use existing high-precision image segmentation networks such as DeepLapV3 and UNet, and this application does not impose any restrictions.
[0074] In this optional embodiment, in order to ensure that the output of the initial teacher network is a component segmentation result of the vehicle image, it is necessary to train the initial teacher network based on the training set to obtain a second teacher network. The training of the initial teacher network based on the training set to obtain a second teacher network includes:
[0075] Training the initial teacher network based on the labeled data set and the cross entropy loss function to obtain a first teacher network;
[0076] Based on the first teacher network, a component segmentation result of each vehicle image in the unlabeled data set is obtained, wherein the component segmentation result includes a category vector of each pixel in the vehicle image, and the category vector includes a probability value of the pixel belonging to each type of vehicle component;
[0077] Calculate the credibility index of each component segmentation result according to the credibility calculation model;
[0078] Screening the vehicle images in the unlabeled data set based on the credibility index and a preset credibility threshold to obtain abnormal images;
[0079] The label data of all abnormal images are obtained, and the first teacher network is trained based on the abnormal images and the label data of the abnormal images to obtain a second teacher network.
[0080] In this optional embodiment, the initial teacher network is first trained based on a labeled data set and a cross entropy loss function to obtain a first teacher network. During the training process, vehicle images in the labeled data set are continuously input into the initial teacher network to obtain output results. The value of the cross entropy loss function is calculated based on the output results and the label data of the vehicle images, and the parameters in the initial teacher network are updated using the gradient descent method. When the value of the cross entropy loss function no longer changes, the training of the initial teacher network is stopped to obtain the first teacher network.
[0081] In this optional embodiment, all vehicle images in the unlabeled data set are sequentially input into the first teacher network to obtain the component segmentation result of each vehicle image, wherein the component segmentation result includes a category vector of each pixel in the vehicle image, and the category vector includes a probability value of the pixel belonging to each type of vehicle component; further, the credibility index of each component segmentation result is calculated according to the credibility calculation model. Taking the component segmentation result k as an example, the credibility calculation model satisfies the relationship:
[0082]
[0083] Where N is the number of all vehicle parts, W×H is the width and height of the part segmentation result k, is the probability value of the category vector of the pixel point (i, j) in the component segmentation result k belonging to the vehicle component type c, α k is the credibility index of the component segmentation result k, and its value range is (0,1]. The larger the value, the greater the credibility of the component segmentation result k and the higher the accuracy of vehicle component segmentation.
[0084] In the above formula, is the information entropy of the category vector of the pixel point (i, j). The larger the value of the information entropy is, the smaller the difference between different probability values in the category vector is, and the less accurate the segmentation result of the pixel point (i, j) is. When all the probability values in the category vector are equal, the information entropy reaches its maximum value, which is log N.
[0085] For example, assuming that the component segmentation result 1 contains only one pixel, the number of types of all vehicle components is 5, and the category vector corresponding to the pixel is (0.2, 0.1, 0, 0.1, 0.6), the credibility index of the component segmentation result is:
[0086]
[0087] In this optional embodiment, screening the vehicle images in the unlabeled data set based on the credibility index and the preset credibility threshold to obtain abnormal images includes:
[0088] comparing the credibility indicator with a preset credibility threshold;
[0089] If the credibility index is greater than the preset credibility threshold, it means that the accuracy of the component segmentation result corresponding to the credibility index meets the requirement, and the vehicle image corresponding to the component segmentation result is marked as a normal image;
[0090] If the credibility index is not greater than the preset credibility threshold, it means that the accuracy of the component segmentation result corresponding to the credibility index does not meet the requirement, and the vehicle image corresponding to the component segmentation result is marked as an abnormal image. The preset credibility threshold is 0.6.
[0091] In this optional embodiment, the first teacher network cannot learn the distinguishing features of different vehicle parts well, resulting in the accuracy of the component segmentation results of the abnormal image not meeting the requirements, so the label data of all abnormal images are obtained, and the first teacher network is trained based on all abnormal images and the label data of all abnormal images to constrain the first teacher network to learn the distinguishing features of different vehicle parts, wherein the label data of the abnormal images is obtained by manual annotation. During the training process, the abnormal images are continuously input into the first teacher network to obtain the output results, the value of the cross entropy loss function is calculated based on the output results and the label data of the abnormal images, and the parameters in the first teacher network are updated using the gradient descent method. When the value of the cross entropy loss function no longer changes, the training of the first teacher network is stopped to obtain the second teacher network, and the second teacher network can extract the distinguishing features of different vehicle parts in the vehicle image to obtain accurate vehicle part segmentation results.
[0092] In this way, the initial teacher network is trained twice based on the labeled data set and the unlabeled data set in the training set to obtain the second teacher network. The second teacher network can extract the distinctive features of different vehicle parts in the vehicle image, obtain accurate vehicle parts segmentation results, and ensure the vehicle parts segmentation accuracy of the subsequent student network.
[0093] S12, build the initial student network.
[0094] In an optional embodiment, an initial student network is constructed, wherein the input and output of the initial student network are the same as those of the teacher network, and the initial student network is composed of a lightweight encoder and a lightweight decoder, wherein the lightweight encoder uses a convolution layer to downsample the input vehicle image to obtain a student feature map, and the lightweight decoder uses a deconvolution layer to upsample the student feature map to obtain a component segmentation result as large as the input vehicle image, wherein the lightweight encoder can adopt an existing encoder structure with a smaller parameter amount, such as MobileNet and ShuffleNet, and the lightweight decoder can be implemented by reducing the number of deconvolution layers. The specific structures of the lightweight encoder and lightweight decoder are not limited in this application.
[0095] In this optional embodiment, due to the lightweight characteristics of the initial student network, the initial student network can be embedded in a mobile terminal device to improve the speed of vehicle component segmentation in the mobile scenario. The mobile terminal device can be a portable smart device with a camera function, such as a smart phone, a smart watch, or a tablet computer, and this application does not impose any restrictions.
[0096] In this way, the initial student network is completed. The parameters of the initial student network are relatively small and can be well embedded in the mobile terminal device, thereby improving the speed of vehicle component segmentation in the mobile terminal scenario.
[0097] S13, extracting dark knowledge of each vehicle image in the training set based on the second teacher network, and training the initial student network based on the dark knowledge to obtain a second student network, wherein the dark knowledge reflects feature similarity and position correlation between different types of vehicle components.
[0098] In an optional embodiment, after the initial student network is built, a second teacher network with higher vehicle component segmentation accuracy is used to extract dark knowledge of each vehicle image in the training set. The dark knowledge can reflect the feature similarity and position correlation between different types of vehicle components in the vehicle image. The built initial student network is trained based on the training set and the dark knowledge to constrain the output result of the initial student network to be an accurate component segmentation result. The training process diagram of the initial student network is shown in FIG. Figure 2 shown.
[0099] In this optional embodiment, the dark knowledge of each vehicle image in the training set is extracted based on the second teacher network, and the initial student network is trained based on the dark knowledge to obtain a second student network, wherein the dark knowledge reflects the feature similarity and position correlation between different types of vehicle parts, including:
[0100] Inputting a target vehicle image into the second teacher network and the initial student network at the same time, the target vehicle image is any one of all vehicle images in the training set, using the output result of the second teacher network as the teacher component segmentation result, and using the output result of the initial student network as the student component segmentation result, and both the teacher component segmentation result and the student component segmentation result include a category vector for each pixel in the target vehicle image;
[0101] Constructing a position correlation matrix of the target vehicle image based on the teacher component segmentation result;
[0102] Using the category vector of the teacher component segmentation result and the position correlation matrix as dark knowledge of the target vehicle image, and constructing a first preset loss function based on the dark knowledge;
[0103] Determine whether the target vehicle image has label data to obtain a judgment parameter, if the target vehicle image has label data, construct a cross entropy loss function based on the label data and the student component segmentation result, and the value of the judgment parameter is 1; if the target vehicle image does not have label data, the value of the judgment parameter is 0;
[0104] Constructing a preset loss function based on the judgment parameter and the first preset loss function;
[0105] The initial student network is trained based on the preset loss function and all vehicle images in the training set to obtain a second student network.
[0106] In this optional embodiment, the teacher component segmentation result is an accurate component segmentation result, so the difference in different probability values in the category vector of the teacher component segmentation result can reflect the feature similarity between different types of vehicle components. Exemplarily, assuming that the number of types of vehicle components is 5, if the category vector of a pixel point is (0.1, 0.3, 0, 0, 0.6), it means that the probability that the pixel point belongs to vehicle component type 1 is 0.1, the probability of vehicle component type 2 is 0.3, and the probability of belonging to vehicle component type 5 is 0.6, indicating that the feature similarity between vehicle component type 5 and vehicle component type 2 is greater than the feature similarity between vehicle component type 5 and vehicle component type 1.
[0107] In this optional embodiment, the position correlation matrix of the target vehicle image constructed based on the teacher component segmentation result includes: selecting the vehicle component type corresponding to the maximum probability value in the category vector of the teacher component segmentation result as the vehicle component type of each pixel point; setting the pixel values of the pixel points of the same vehicle component type to 1, and the pixel points in other areas to 0 to obtain the regional image of the vehicle component, traversing all vehicle component types in the teacher component segmentation result to obtain the regional image of each vehicle component. For example, the pixel values of all pixel points of the vehicle component type c in the teacher component segmentation result are set to 1, and the pixel values of the pixel points in other areas are set to 0. When the image is 0, the regional image of the vehicle component c in the target vehicle image can be obtained; the pixel values in the regional image of the vehicle component are arranged in a fixed order along the row direction to obtain the position vector of each vehicle component in the target vehicle image; based on the position vector, the position correlation of different vehicle components is calculated to construct the position correlation matrix of the teacher component segmentation result, and the position correlation matrix of the teacher component segmentation result is used as the position correlation matrix X of the target vehicle image, the position correlation matrix X is an N×N square matrix, and the value of the mth row and nth column represents the position correlation between the vehicle component m and the vehicle component n in the target vehicle image, and the position correlation satisfies the relationship:
[0108] x mn =h m (h n ) T
[0109] Among them, h m is the position vector of the vehicle component m in the target vehicle image, (h n ) T is the transpose of the position vector of vehicle component n in the target vehicle image, x mnis the position correlation between vehicle component m and vehicle component n.
[0110] In this optional embodiment, the category vector of the teacher component segmentation result and the position correlation matrix are used as the dark knowledge of the target vehicle image, and a first preset loss function is constructed based on the dark knowledge. The first preset loss function is used to constrain the feature similarity and position correlation between different types of vehicle components in the student component segmentation result to be consistent with the dark knowledge. The first preset loss function satisfies the relationship:
[0111]
[0112] Among them, X s is the position correlation matrix of the student component segmentation result, X is the position correlation matrix of the target vehicle image, ‖X s -X‖2 means X s -L2 distance of X; P s (i, j) is the category vector of the pixel point (i, j) in the student component segmentation result, P(i, j) is the category vector of the pixel point (i, j) in the teacher component segmentation result, KL(P s (i,j),P(i,j)) is used to calculate P s (i,j), KL divergence between P(i,j); W, H are the width and height of the target vehicle image.
[0113] It should be noted that the position correlation matrix X of the student component segmentation result is obtained based on the student component segmentation result. s The method is the same as the method of constructing the position correlation matrix X of the target vehicle image based on the teacher component segmentation result.
[0114] In this optional embodiment, it is determined whether the target vehicle image has label data to obtain a judgment parameter. If the target vehicle image has label data, a cross entropy loss function is constructed based on the label data and the student component segmentation result, and the value of the judgment parameter is 1; if the target vehicle image does not have label data, the value of the judgment parameter is 0; a preset loss function is constructed based on the judgment parameter and the first preset loss function, and the preset loss function satisfies the relationship:
[0115]
[0116] Among them, Loss1 is the first preset loss function, Loss2 is the cross entropy loss function constructed based on the label data and the student component segmentation result, β is the judgment parameter, and Loss is the preset loss function.
[0117] In this optional embodiment, the initial student network and the second teacher network are continuously selected from the training set to calculate the value of a preset loss function, and the parameters in the initial student network are continuously updated using the gradient descent method. When the value of the preset loss function no longer changes, the training of the initial student network is stopped to obtain the second student network. The second student network can extract the distinguishing features of different vehicle parts in the vehicle image and obtain accurate vehicle part segmentation results.
[0118] In this way, the dark knowledge of each vehicle image can be extracted based on the teacher network with higher vehicle parts segmentation accuracy to supervise the training process of the initial student network, and a second student network with a small number of parameters and capable of accurately segmenting vehicle parts can be obtained.
[0119] S14, obtaining a component segmentation result of the real-time vehicle image based on the second student network.
[0120] In an optional embodiment, after a second student network with a small number of parameters is embedded into a mobile device, a user holds the mobile device to capture real-time vehicle images, and the second student network segments the real-time vehicle images to obtain component segmentation results.
[0121] It can be seen from the above technical solution that the present application trains the initial teacher network twice based on labeled data and unlabeled data to obtain a teacher network, and uses the teacher network to extract the dark knowledge of each vehicle image in the labeled data and the unlabeled data to supervise the training of the lightweight initial student network, thereby obtaining a second student network with a small number of parameters and high segmentation accuracy. Embedding the second student network into a mobile device can improve the speed and accuracy of vehicle parts segmentation in mobile scenarios.
[0122] See also Figure 4 , Figure 4 It is a functional module diagram of a preferred embodiment of the vehicle component segmentation device based on knowledge distillation of the present application. The vehicle component segmentation device 11 based on knowledge distillation includes a collection unit 110, a training unit 111, a construction unit 112, an extraction unit 113, and a segmentation unit 114. The module / unit referred to in this application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and can complete fixed functions, which are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.
[0123] In an optional embodiment, the acquisition unit 110 is used to acquire vehicle images with labeled data as a labeled data set, and acquire vehicle images without labeled data as an unlabeled data set, and use the labeled data set and the unlabeled data set as training sets, wherein the labeled data includes the type of vehicle parts for each pixel in the vehicle image.
[0124] In an optional embodiment, a large number of vehicle images are collected, and label data of each vehicle image is obtained, wherein the label data is an image of the same size as the vehicle image, and the pixel value in the label data represents a preset label of the type of vehicle component at each pixel point, and the preset label is an integer from 1 to N, wherein N represents the number of all types of vehicle components including the background class, and the background class is regarded as a special type of vehicle component. The background class is the category of pixel points in the vehicle image that do not belong to vehicle components, and the vehicle component types correspond one-to-one to the preset labels, and all vehicle images and the label data of all vehicle images are stored as a labeled data set.
[0125] In this optional embodiment, a large number of new vehicle images are collected without obtaining label data of the new vehicle images. All new vehicle images are directly stored as unlabeled data sets, and the labeled data sets and the unlabeled data sets are used as training sets for subsequent training of the teacher network and the student network.
[0126] In an optional embodiment, the training unit 111 is used to build an initial teacher network, and train the initial teacher network based on the training set to obtain a second teacher network.
[0127] In an optional embodiment, an initial teacher network is constructed, wherein the input of the initial teacher network is a vehicle image, and the expected output is a component segmentation result of the vehicle image, wherein the component segmentation result is an image of the same size as the vehicle image, and the component segmentation result includes a category vector for each pixel in the vehicle image, wherein the category vector includes a probability value that the pixel belongs to each type of vehicle component, and the sum of all probability values in the category vector of the same pixel is 1; the type of vehicle component corresponding to the maximum probability value in the category vector is selected as the type of vehicle component corresponding to the pixel.
[0128] In this optional embodiment, the initial teacher network is an encoder and decoder structure, the encoder uses a convolution layer to downsample the input vehicle image to obtain a teacher feature map, and sends the teacher feature map to the decoder to use a deconvolution layer for upsampling to obtain the component segmentation result of the vehicle image. It should be noted that the vehicle component segmentation accuracy of the initial teacher network directly affects the segmentation accuracy of the subsequent student network, so there is no need to consider the parameter amount of the initial teacher network to ensure that the vehicle component segmentation accuracy of the initial teacher network is the top priority condition. Therefore, the initial teacher network can use existing high-precision image segmentation networks such as DeepLapV3 and UNet, and this application does not impose any restrictions.
[0129] In this optional embodiment, in order to ensure that the output of the initial teacher network is a component segmentation result of the vehicle image, it is necessary to train the initial teacher network based on the training set to obtain a second teacher network. The training of the initial teacher network based on the training set to obtain a second teacher network includes:
[0130] Training the initial teacher network based on the labeled data set and the cross entropy loss function to obtain a first teacher network;
[0131] Based on the first teacher network, a component segmentation result of each vehicle image in the unlabeled data set is obtained, wherein the component segmentation result includes a category vector of each pixel in the vehicle image, and the category vector includes a probability value of the pixel belonging to each type of vehicle component;
[0132] Calculate the credibility index of each component segmentation result according to the credibility calculation model;
[0133] Screening the vehicle images in the unlabeled data set based on the credibility index and a preset credibility threshold to obtain abnormal images;
[0134] The label data of all abnormal images are obtained, and the first teacher network is trained based on the abnormal images and the label data of the abnormal images to obtain a second teacher network.
[0135] In this optional embodiment, the initial teacher network is first trained based on a labeled data set and a cross entropy loss function to obtain a first teacher network. During the training process, vehicle images in the labeled data set are continuously input into the initial teacher network to obtain output results. The value of the cross entropy loss function is calculated based on the output results and the label data of the vehicle images, and the parameters in the initial teacher network are updated using the gradient descent method. When the value of the cross entropy loss function no longer changes, the training of the initial teacher network is stopped to obtain the first teacher network.
[0136] In this optional embodiment, all vehicle images in the unlabeled data set are sequentially input into the first teacher network to obtain the component segmentation result of each vehicle image, wherein the component segmentation result includes a category vector of each pixel in the vehicle image, and the category vector includes a probability value of the pixel belonging to each type of vehicle component; further, the credibility index of each component segmentation result is calculated according to the credibility calculation model. Taking the component segmentation result k as an example, the credibility calculation model satisfies the relationship:
[0137]
[0138] Where N is the number of all vehicle parts, W×H is the width and height of the part segmentation result k, is the probability value of the category vector of the pixel point (i, j) in the component segmentation result k belonging to the vehicle component type c, α k is the credibility index of the component segmentation result k, and its value range is (0,1]. The larger the value, the greater the credibility of the component segmentation result k and the higher the accuracy of vehicle component segmentation.
[0139] In the above formula, is the information entropy of the category vector of the pixel point (i, j). The larger the value of the information entropy is, the smaller the difference between different probability values in the category vector is, and the less accurate the segmentation result of the pixel point (i, j) is. When all the probability values in the category vector are equal, the information entropy reaches its maximum value, which is log N.
[0140] For example, assuming that the component segmentation result 1 contains only one pixel, the number of types of all vehicle components is 5, and the category vector corresponding to the pixel is (0.2, 0.1, 0, 0.1, 0.6), the credibility index of the component segmentation result is:
[0141]
[0142] In this optional embodiment, screening the vehicle images in the unlabeled data set based on the credibility index and the preset credibility threshold to obtain abnormal images includes:
[0143] comparing the credibility indicator with a preset credibility threshold;
[0144] If the credibility index is greater than the preset credibility threshold, it means that the accuracy of the component segmentation result corresponding to the credibility index meets the requirement, and the vehicle image corresponding to the component segmentation result is marked as a normal image;
[0145] If the credibility index is not greater than the preset credibility threshold, it means that the accuracy of the component segmentation result corresponding to the credibility index does not meet the requirement, and the vehicle image corresponding to the component segmentation result is marked as an abnormal image. The preset credibility threshold is 0.6.
[0146] In this optional embodiment, the first teacher network cannot learn the distinguishing features of different vehicle parts well, resulting in the accuracy of the component segmentation results of the abnormal image not meeting the requirements, so the label data of all abnormal images are obtained, and the first teacher network is trained based on all abnormal images and the label data of all abnormal images to constrain the first teacher network to learn the distinguishing features of different vehicle parts, wherein the label data of the abnormal images is obtained by manual annotation. During the training process, the abnormal images are continuously input into the first teacher network to obtain the output results, the value of the cross entropy loss function is calculated based on the output results and the label data of the abnormal images, and the parameters in the first teacher network are updated using the gradient descent method. When the value of the cross entropy loss function no longer changes, the training of the first teacher network is stopped to obtain the second teacher network, and the second teacher network can extract the distinguishing features of different vehicle parts in the vehicle image to obtain accurate vehicle part segmentation results.
[0147] In an optional embodiment, the building unit 112 is used to build an initial student network.
[0148] In an optional embodiment, an initial student network is constructed, wherein the input and output of the initial student network are the same as those of the teacher network, and the initial student network is composed of a lightweight encoder and a lightweight decoder, wherein the lightweight encoder uses a convolution layer to downsample the input vehicle image to obtain a student feature map, and the lightweight decoder uses a deconvolution layer to upsample the student feature map to obtain a component segmentation result as large as the input vehicle image, wherein the lightweight encoder can adopt an existing encoder structure with a smaller parameter amount, such as MobileNet and ShuffleNet, and the lightweight decoder can be implemented by reducing the number of deconvolution layers. The specific structures of the lightweight encoder and lightweight decoder are not limited in this application.
[0149] In this optional embodiment, due to the lightweight characteristics of the initial student network, the initial student network can be embedded in a mobile terminal device to improve the speed of vehicle component segmentation in the mobile scenario. The mobile terminal device can be a portable smart device with a camera function, such as a smart phone, a smart watch, or a tablet computer, and this application does not impose any restrictions.
[0150] In an optional embodiment, the extraction unit 113 is used to extract dark knowledge of each vehicle image in the training set based on the second teacher network, and train the initial student network based on the dark knowledge to obtain a second student network, wherein the dark knowledge reflects feature similarity and position correlation between different types of vehicle parts.
[0151] In an optional embodiment, after the initial student network is built, a second teacher network with higher vehicle component segmentation accuracy is used to extract dark knowledge of each vehicle image in the training set. The dark knowledge can reflect the feature similarity and position correlation between different types of vehicle components in the vehicle image. The built initial student network is trained based on the training set and the dark knowledge to constrain the output result of the initial student network to be an accurate component segmentation result. The training process diagram of the initial student network is shown in FIG. Figure 2 shown.
[0152] In this optional embodiment, the dark knowledge of each vehicle image in the training set is extracted based on the second teacher network, and the initial student network is trained based on the dark knowledge to obtain a second student network, wherein the dark knowledge reflects the feature similarity and position correlation between different types of vehicle parts, including:
[0153] Inputting a target vehicle image into the second teacher network and the initial student network at the same time, the target vehicle image is any one of all vehicle images in the training set, using the output result of the second teacher network as the teacher component segmentation result, and using the output result of the initial student network as the student component segmentation result, and both the teacher component segmentation result and the student component segmentation result include a category vector for each pixel in the target vehicle image;
[0154] Constructing a position correlation matrix of the target vehicle image based on the teacher component segmentation result;
[0155] Using the category vector of the teacher component segmentation result and the position correlation matrix as dark knowledge of the target vehicle image, and constructing a first preset loss function based on the dark knowledge;
[0156] Determine whether the target vehicle image has label data to obtain a judgment parameter, if the target vehicle image has label data, construct a cross entropy loss function based on the label data and the student component segmentation result, and the value of the judgment parameter is 1; if the target vehicle image does not have label data, the value of the judgment parameter is 0;
[0157] Constructing a preset loss function based on the judgment parameter and the first preset loss function;
[0158] The initial student network is trained based on the preset loss function and all vehicle images in the training set to obtain a second student network.
[0159] In this optional embodiment, the teacher component segmentation result is an accurate component segmentation result, so the difference in different probability values in the category vector of the teacher component segmentation result can reflect the feature similarity between different types of vehicle components. Exemplarily, assuming that the number of types of vehicle components is 5, if the category vector of a pixel point is (0.1, 0.3, 0, 0, 0.6), it means that the probability that the pixel point belongs to vehicle component type 1 is 0.1, the probability of vehicle component type 2 is 0.3, and the probability of belonging to vehicle component type 5 is 0.6, indicating that the feature similarity between vehicle component type 5 and vehicle component type 2 is greater than the feature similarity between vehicle component type 5 and vehicle component type 1.
[0160] In this optional embodiment, the position correlation matrix of the target vehicle image constructed based on the teacher component segmentation result includes: selecting the vehicle component type corresponding to the maximum probability value in the category vector of the teacher component segmentation result as the vehicle component type of each pixel point; setting the pixel values of the pixel points of the same vehicle component type to 1, and the pixel points in other areas to 0 to obtain the regional image of the vehicle component, traversing all vehicle component types in the teacher component segmentation result to obtain the regional image of each vehicle component. For example, the pixel values of all pixel points of the vehicle component type c in the teacher component segmentation result are set to 1, and the pixel values of the pixel points in other areas are set to 0. When the image is 0, the regional image of the vehicle component c in the target vehicle image can be obtained; the pixel values in the regional image of the vehicle component are arranged in a fixed order along the row direction to obtain the position vector of each vehicle component in the target vehicle image; based on the position vector, the position correlation of different vehicle components is calculated to construct the position correlation matrix of the teacher component segmentation result, and the position correlation matrix of the teacher component segmentation result is used as the position correlation matrix X of the target vehicle image, the position correlation matrix X is an N×N square matrix, and the value of the mth row and nth column represents the position correlation between the vehicle component m and the vehicle component n in the target vehicle image, and the position correlation satisfies the relationship:
[0161] x mn =h m (h n ) T
[0162] Among them, h m is the position vector of the vehicle component m in the target vehicle image, (h n ) T is the transpose of the position vector of vehicle component n in the target vehicle image, x mn is the position correlation between vehicle component m and vehicle component n.
[0163] In this optional embodiment, the category vector of the teacher component segmentation result and the position correlation matrix are used as the dark knowledge of the target vehicle image, and a first preset loss function is constructed based on the dark knowledge. The first preset loss function is used to constrain the feature similarity and position correlation between different types of vehicle components in the student component segmentation result to be consistent with the dark knowledge. The first preset loss function satisfies the relationship:
[0164]
[0165] Among them, X s is the position correlation matrix of the student component segmentation result, X is the position correlation matrix of the target vehicle image, ‖X s -X‖2 means X s -L2 distance of X; P s (i, j) is the category vector of the pixel point (i, j) in the student component segmentation result, P(i, j) is the category vector of the pixel point (i, j) in the teacher component segmentation result, KL(P s (i,j),P(i,j)) is used to calculate P s (i,j), KL divergence between P(i,j); W, H are the width and height of the target vehicle image.
[0166] It should be noted that the position correlation matrix X of the student component segmentation result is obtained based on the student component segmentation result. s The method is the same as the method of constructing the position correlation matrix X of the target vehicle image based on the teacher component segmentation result.
[0167] In this optional embodiment, it is determined whether the target vehicle image has label data to obtain a judgment parameter. If the target vehicle image has label data, a cross entropy loss function is constructed based on the label data and the student component segmentation result, and the value of the judgment parameter is 1; if the target vehicle image does not have label data, the value of the judgment parameter is 0; a preset loss function is constructed based on the judgment parameter and the first preset loss function, and the preset loss function satisfies the relationship:
[0168]
[0169] Among them, Loss1 is the first preset loss function, Loss2 is the cross entropy loss function constructed based on the label data and the student component segmentation result, β is the judgment parameter, and Loss is the preset loss function.
[0170] In this optional embodiment, the initial student network and the second teacher network are continuously selected from the training set to calculate the value of a preset loss function, and the parameters in the initial student network are continuously updated using the gradient descent method. When the value of the preset loss function no longer changes, the training of the initial student network is stopped to obtain the second student network. The second student network can extract the distinguishing features of different vehicle parts in the vehicle image and obtain accurate vehicle part segmentation results.
[0171] In an optional embodiment, the segmentation unit 114 is configured to obtain a component segmentation result of the real-time vehicle image based on the second student network.
[0172] In an optional embodiment, after a second student network with a small number of parameters is embedded into a mobile device, a user holds the mobile device to capture real-time vehicle images, and the second student network segments the real-time vehicle images to obtain component segmentation results.
[0173] It can be seen from the above technical solution that the present application trains the initial teacher network twice based on labeled data and unlabeled data to obtain a teacher network, and uses the teacher network to extract the dark knowledge of each vehicle image in the labeled data and the unlabeled data to supervise the training of the lightweight initial student network, thereby obtaining a second student network with a small number of parameters and high segmentation accuracy. Embedding the second student network into a mobile device can improve the speed and accuracy of vehicle parts segmentation in mobile scenarios.
[0174] See also Figure 4 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 1 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 is used to execute the computer-readable instructions stored in the memory to implement the vehicle component segmentation method based on knowledge distillation described in any of the above embodiments.
[0175] In an optional embodiment, the electronic device 1 further includes a bus, a computer program stored in the memory 12 and executable on the processor 13 , such as a vehicle component segmentation program based on knowledge distillation.
[0176] Figure 4 Only the electronic device 1 having the memory 12 and the processor 13 is shown, and those skilled in the art can understand that Figure 4 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0177] Combination Figure 1The memory 12 in the electronic device 1 stores a plurality of computer-readable instructions to implement a vehicle component segmentation method based on knowledge distillation, and the processor 13 can execute the plurality of instructions to implement:
[0178] Collecting vehicle images with labeled data as a labeled data set, collecting vehicle images without labeled data as an unlabeled data set, and using the labeled data set and the unlabeled data set as training sets, wherein the labeled data includes the type of vehicle parts for each pixel in the vehicle image;
[0179] Building an initial teacher network, and training the initial teacher network based on the training set to obtain a second teacher network;
[0180] Building an initial student network;
[0181] Extracting dark knowledge of each vehicle image in the training set based on the second teacher network, and training the initial student network based on the dark knowledge to obtain a second student network, wherein the dark knowledge reflects feature similarity and position correlation between different types of vehicle parts;
[0182] A component segmentation result of a real-time vehicle image is obtained based on the second student network.
[0183] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.
[0184] Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 may be a bus structure or a star structure. The electronic device 1 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the electronic device 1 may also include input and output devices, network access devices, etc.
[0185] It should be noted that the electronic device 1 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.
[0186] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the vehicle component segmentation program based on knowledge distillation, etc., but also can be used to temporarily store data that has been output or is to be output.
[0187] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect the various components of the entire electronic device 1, and executes or executes programs or modules stored in the memory 12 (for example, executing a vehicle component segmentation program based on knowledge distillation, etc.), and calls the data stored in the memory 12 to execute various functions of the electronic device 1 and process data.
[0188] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned vehicle parts segmentation method embodiments based on knowledge distillation, for example Figure 1 Steps shown.
[0189] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a collection unit 110, a training unit 111, a construction unit 112, an extraction unit 113, and a segmentation unit 114.
[0190] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute the part of the vehicle component segmentation method based on knowledge distillation described in each embodiment of the present application.
[0191] If the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware devices to complete through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.
[0192] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory, etc.
[0193] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.
[0194] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0195] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Figure 4 Only one arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13, etc.
[0196] An embodiment of the present application also provides a computer-readable storage medium (not shown), in which computer-readable storage medium is stored computer-readable instructions, and the computer-readable instructions are executed by a processor in an electronic device to implement the vehicle component segmentation method based on knowledge distillation described in any of the above embodiments.
[0197] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
[0198] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0199] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0200] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.
[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.
Claims
1. A vehicle parts segmentation method based on knowledge distillation, characterized in that: The method comprises: Collecting vehicle images with labeled data as a labeled data set, collecting vehicle images without labeled data as an unlabeled data set, and using the labeled data set and the unlabeled data set as training sets, wherein the labeled data includes the type of vehicle parts for each pixel in the vehicle image; Building an initial teacher network, and training the initial teacher network based on the training set to obtain a second teacher network; Building an initial student network; Based on the second teacher network, dark knowledge of each vehicle image in the training set is extracted, and the initial student network is trained based on the dark knowledge to obtain a second student network, wherein the dark knowledge reflects the feature similarity and position correlation between different types of vehicle components, including: inputting the target vehicle image into the second teacher network and the initial student network at the same time, wherein the target vehicle image is any one of all vehicle images in the training set, using the output result of the second teacher network as the teacher component segmentation result, and using the output result of the initial student network as the student component segmentation result, wherein both the teacher component segmentation result and the student component segmentation result include the category vector of each pixel point in the target vehicle image; constructing the target vehicle image based on the teacher component segmentation result. a position correlation matrix of a labeled vehicle image; using the category vector of the teacher component segmentation result and the position correlation matrix as the dark knowledge of the target vehicle image, and constructing a first preset loss function based on the dark knowledge; judging whether the target vehicle image has label data to obtain a judgment parameter, if the target vehicle image has label data, constructing a cross entropy loss function based on the label data and the student component segmentation result, and the value of the judgment parameter is 1; if the target vehicle image does not have label data, the value of the judgment parameter is 0; constructing a preset loss function based on the judgment parameter and the first preset loss function; training the initial student network based on the preset loss function and all vehicle images in the training set to obtain a second student network; A component segmentation result of a real-time vehicle image is obtained based on the second student network.
2. The vehicle parts segmentation method based on knowledge distillation according to claim 1, characterized in that: The training of the initial teacher network based on the training set to obtain a second teacher network comprises: Training the initial teacher network based on the labeled data set and the cross entropy loss function to obtain a first teacher network; Based on the first teacher network, a component segmentation result of each vehicle image in the unlabeled data set is obtained, wherein the component segmentation result includes a category vector of each pixel in the vehicle image, and the category vector includes a probability value of the pixel belonging to each type of vehicle component; Calculate the credibility index of each component segmentation result according to the credibility calculation model; Screening the vehicle images in the unlabeled data set based on the credibility index and a preset credibility threshold to obtain abnormal images; The label data of all abnormal images are obtained, and the first teacher network is trained based on the abnormal images and the label data of the abnormal images to obtain a second teacher network.
3. The vehicle parts segmentation method based on knowledge distillation according to claim 2, characterized in that: The credibility calculation model satisfies the relationship: Where N is the number of types of all vehicle parts, is the width and height of the component segmentation result k, is the probability value of the category vector of the pixel point (i, j) in the component segmentation result k belonging to the vehicle component type c, is the credibility index of the component segmentation result k, and its value range is (0,1]. The larger the value, the greater the credibility of the component segmentation result k and the higher the accuracy of vehicle component segmentation.
4. The vehicle parts segmentation method based on knowledge distillation according to claim 2, characterized in that: The screening of the vehicle images in the unlabeled data set based on the credibility index and the preset credibility threshold to obtain abnormal images includes: comparing the credibility indicator with a preset credibility threshold; If the credibility index is greater than the preset credibility threshold, it means that the accuracy of the component segmentation result corresponding to the credibility index meets the requirement, and the vehicle image corresponding to the component segmentation result is marked as a normal image; If the credibility index is not greater than the preset credibility threshold, it means that the accuracy of the component segmentation result corresponding to the credibility index does not meet the requirement, and the vehicle image corresponding to the component segmentation result is marked as an abnormal image.
5. The vehicle parts segmentation method based on knowledge distillation according to claim 1, characterized in that: The constructing of the position correlation matrix of the target vehicle image based on the teacher component segmentation result comprises: Selecting the vehicle component type corresponding to the maximum probability value in the category vector of the teacher component segmentation result as the vehicle component type of each pixel; The pixel values of the pixels of the same vehicle component type are set to 1, and the pixel values of other regions are set to 0 to obtain the regional image of the vehicle component, and all the vehicle component types in the teacher component segmentation result are traversed to obtain the regional image of each vehicle component; Arranging pixel values in the regional image of the vehicle parts in a fixed order along the row direction to obtain a position vector of each vehicle part in the target vehicle image; Based on the position vector, the position correlation of different vehicle components is calculated to construct a position correlation matrix of the target vehicle image, and the position correlation satisfies the relationship: in, is the position vector of the vehicle component m in the target vehicle image, is the transpose of the position vector of vehicle component n in the target vehicle image, is the position correlation between vehicle component m and vehicle component n.
6. The vehicle parts segmentation method based on knowledge distillation according to claim 1, characterized in that: In constructing the preset loss function based on the judgment parameter and the first preset loss function, the preset loss function satisfies the relationship: in, is the first preset loss function, is a cross entropy loss function constructed based on the label data and the student component segmentation results, Judgment parameter, the value is 0 or 1, is the preset loss function; The first preset loss function satisfies the relationship: in, is the position correlation matrix of the student component segmentation results, is the position correlation matrix of the target vehicle image, express L2 distance; is the category vector of the pixel point (i, j) in the student component segmentation result, is the category vector of the pixel point (i, j) in the teacher component segmentation result, Used for calculation The KL divergence between is the width and height of the target vehicle image.
7. A vehicle parts segmentation device based on knowledge distillation, characterized in that: The device comprises: A collection unit, used to collect vehicle images with labeled data as a labeled data set, collect vehicle images without labeled data as an unlabeled data set, and use the labeled data set and the unlabeled data set as training sets, wherein the labeled data includes the type of vehicle parts for each pixel in the vehicle image; A training unit, configured to build an initial teacher network, and train the initial teacher network based on the training set to obtain a second teacher network; Building unit, used to build the initial student network; An extraction unit is used to extract dark knowledge of each vehicle image in the training set based on the second teacher network, and train the initial student network based on the dark knowledge to obtain a second student network, wherein the dark knowledge reflects the feature similarity and position correlation between different types of vehicle parts, including: inputting a target vehicle image into the second teacher network and the initial student network at the same time, wherein the target vehicle image is any one of all vehicle images in the training set, using the output result of the second teacher network as a teacher component segmentation result, and using the output result of the initial student network as a student component segmentation result, wherein both the teacher component segmentation result and the student component segmentation result include a category vector for each pixel in the target vehicle image; constructing a target vehicle image based on the teacher component segmentation result. The position correlation matrix of the target vehicle image; using the category vector of the teacher component segmentation result and the position correlation matrix as the dark knowledge of the target vehicle image, and constructing a first preset loss function based on the dark knowledge; judging whether the target vehicle image has label data to obtain a judgment parameter, if the target vehicle image has label data, constructing a cross entropy loss function based on the label data and the student component segmentation result, and the value of the judgment parameter is 1; if the target vehicle image does not have label data, the value of the judgment parameter is 0; constructing a preset loss function based on the judgment parameter and the first preset loss function; training the initial student network based on the preset loss function and all vehicle images in the training set to obtain a second student network; A segmentation unit is used to obtain a component segmentation result of a real-time vehicle image based on the second student network.
8. An electronic device, characterized in that: The electronic device comprises: a memory storing computer-readable instructions; and A processor executes computer-readable instructions stored in the memory to implement the vehicle parts segmentation method based on knowledge distillation as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the vehicle component segmentation method based on knowledge distillation as described in any one of claims 1 to 6 is implemented.
