Image data processing method and device

Abstract

The present application provides an image data processing method, including: obtaining an image set; classifying the image set to obtain multiple sub-image sets corresponding to the image set; from each sub-image set, determining the target image corresponding to each sub-image set; inputting the target image into a deep residual network, performing layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, obtaining the layer-by-layer feature extraction results and cross-layer feature extraction results of the target image corresponding to each sub-image set; fusing the layer-by-layer feature extraction results and the cross-layer feature extraction results to obtain the first fusion feature of the target image corresponding to each sub-image set; sampling the first fusion feature to obtain the sampled fusion feature of the target image corresponding to each sub-image set; based on the sampled fusion feature, determining the classification result of the target image corresponding to each sub-image set. The scheme of the embodiment of the present application can improve the efficiency and accuracy of image classification and reduce the cost of image classification.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an image data processing method and device. Background Art

[0002] Related technologies usually use manual screening to process images, and the collected images or videos are judged through manual screening. However, due to the influence of adverse factors such as poor lighting conditions, occlusion caused by obstacles such as pedestrians and vehicles, and blurred and missing lane boundaries, it is challenging to use manual screening of the collected images or videos, and the screening results have large errors.

[0003] It can be seen that the related technology has a high processing cost for image processing, and low processing efficiency and processing accuracy. Summary of the invention

[0004] In order to overcome the defects of the above-mentioned prior art, the present application provides an image data processing method and device that can improve image annotation efficiency and processing accuracy and reduce classification costs.

[0005] In order to solve the above technical problems, this application provides the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, there is provided an image data processing method, the method comprising:

[0007] Get the image set;

[0008] Classifying the image set to obtain multiple categories of sub-image sets corresponding to the image set;

[0009] From each type of the sub-image set, respectively determine the target image corresponding to each type of the sub-image set;

[0010] Inputting the target images into the deep residual network respectively, performing layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and obtaining layer-by-layer feature extraction results and cross-layer feature extraction results of the target images corresponding to each type of the sub-image set;

[0011] Fusing the layer-by-layer feature extraction result and the cross-layer feature extraction result to obtain a first fused feature of the target image corresponding to each type of the sub-image set;

[0012] The first fusion features are sampled and processed respectively to obtain sampled fusion features of the target image corresponding to each type of the sub-image set;

[0013] Based on the sampled fusion features, a classification result of the target image corresponding to each type of the sub-image set is determined.

[0014] In an exemplary embodiment, the cross-layer feature traversal extraction includes odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction;

[0015] The target images are respectively input into the deep residual network to perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and the layer-by-layer feature extraction results and cross-layer feature extraction results of the target images corresponding to each type of the sub-image set are obtained, including:

[0016] The target images are respectively input into the deep residual network to perform layer-by-layer feature traversal extraction, odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction, so as to obtain layer-by-layer feature extraction results, odd-numbered layer feature extraction results and even-numbered layer feature extraction results of the target image corresponding to each type of the sub-image set.

[0017] In an exemplary embodiment, the fusing the layer-by-layer feature extraction result and the cross-layer feature extraction result to obtain a first fused feature of the target image corresponding to each type of the sub-image set includes:

[0018] The layer-by-layer feature extraction results are respectively fused with the odd-numbered layer feature extraction results and the even-numbered layer feature extraction results to obtain odd-numbered layer fusion features and even-numbered layer fusion features;

[0019] The odd-numbered layer fusion features and the even-numbered layer fusion features are used as the first fusion features of the target image corresponding to each type of the sub-image set.

[0020] In an exemplary embodiment, the sampling process is performed on the first fusion features to obtain the sampled fusion features of the target image corresponding to each type of the sub-image set, including:

[0021] Performing upsampling and downsampling processing on the odd-numbered layer fusion features respectively to obtain odd-numbered layer upsampling features and odd-numbered layer downsampling features;

[0022] Performing up-sampling and down-sampling processing on the even-numbered layer fusion features respectively to obtain even-numbered layer up-sampling features and even-numbered layer down-sampling features;

[0023] The odd-numbered layer up-sampling features, the odd-numbered layer down-sampling features, the even-numbered layer up-sampling features and the even-numbered layer down-sampling features are used as sampling fusion features of the target image corresponding to each type of the sub-image set.

[0024] In an exemplary embodiment, determining the classification result of the target image corresponding to each type of the sub-image set based on the sampling fusion feature includes:

[0025] The odd-numbered layer up-sampling features, the odd-numbered layer down-sampling features, the even-numbered layer up-sampling features, and the even-numbered layer down-sampling features are fused to obtain a second fused feature of the target image corresponding to each type of the sub-image set;

[0026] A classification result of the target image is determined based on the second fusion feature.

[0027] In an exemplary embodiment, the method further comprises:

[0028] Acquire a sample image, wherein the sample image is annotated with sample features;

[0029] Inputting the sample image into a preset deep residual network to perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, to obtain a preset layer-by-layer feature extraction result and a preset cross-layer feature extraction result;

[0030] Fusing the preset layer-by-layer feature extraction result and the preset cross-layer feature extraction result to obtain a first preset fusion feature;

[0031] Performing sampling processing on the first preset fusion feature to obtain a sampled preset fusion feature;

[0032] Fusing the sampled preset fusion features to obtain a second preset fusion feature;

[0033] The preset deep residual network is trained according to the loss information between the second preset fusion feature and the sample feature to obtain the deep residual network.

[0034] In an exemplary embodiment, classifying the image set to obtain multiple types of sub-image sets corresponding to the image set includes:

[0035] The image set is classified based on a K-means clustering algorithm to obtain multiple categories of sub-image sets corresponding to the image set.

[0036] In an exemplary embodiment, the image set is classified based on the K-means clustering algorithm to obtain multiple sub-image sets corresponding to the image set, including:

[0037] Acquire a preset number of images from the image set, and use the preset number of images as a preset number of current cluster centers; the preset number is set according to the complexity of the scene contained in the image set;

[0038] Calculating the distance between each image in the image set and a preset number of the current cluster centers;

[0039] According to the distances between each image and a preset number of the current cluster centers, a target cluster center corresponding to each image is obtained from the preset number of the current cluster centers, and the distance between the target cluster center and each image is less than a preset distance threshold;

[0040] Assigning each image to a target cluster center corresponding to each image to obtain a preset number of current candidate image sets;

[0041] Calculating the cluster center of each of the current candidate image sets, and re-using the calculated candidate cluster centers as a preset number of current cluster centers;

[0042] Repeat the step of calculating the distance between each image in the image set and a preset number of the current cluster centers, and the step of using the calculated candidate cluster centers as the preset number of current cluster centers, until the images in the current candidate image set meet the preset conditions or the number of repetitions meets the preset number threshold.

[0043] In an exemplary embodiment, acquiring an image set includes:

[0044] Obtain traffic video data;

[0045] Deduplication processing is performed on the traffic video data to obtain deduplication image data;

[0046] The image set is obtained from the deduplicated image data.

[0047] In an exemplary embodiment, determining the target image corresponding to each type of the sub-image set from each type of the sub-image set includes:

[0048] Any image in each category of the sub-image set is used as the target image corresponding to each category of the sub-image set.

[0049] According to a second aspect of an embodiment of the present application, there is provided an image data processing device, the device comprising:

[0050] An image set acquisition module, used for acquiring an image set;

[0051] An image classification module, used to classify the image set to obtain multiple types of sub-image sets corresponding to the image set;

[0052] A target image determination module, used to determine the target image corresponding to each type of the sub-image set from each type of the sub-image set;

[0053] An image feature extraction module is used to input the target images into a deep residual network respectively, perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and obtain layer-by-layer feature extraction results and cross-layer feature extraction results of the target images corresponding to each type of the sub-image set;

[0054] A first feature fusion module, used for fusing the layer-by-layer feature extraction result and the cross-layer feature extraction result to obtain a first fusion feature of the target image corresponding to each type of the sub-image set;

[0055] A fusion feature sampling module, used for sampling the first fusion features respectively to obtain sampled fusion features of the target image corresponding to each type of the sub-image set;

[0056] The classification result determination module is used to determine the classification result of the target image corresponding to each type of the sub-image set based on the sampling fusion features.

[0057] In an exemplary embodiment, the apparatus further comprises:

[0058] A video data acquisition module, used to acquire traffic video data;

[0059] The deduplication module is used to perform deduplication processing on the traffic video data to obtain deduplicated image data.

[0060] By adopting the above technical solution, the present application has the following beneficial effects:

[0061] The present application provides an image data processing method and device, which respectively inputs the target image corresponding to each type of sub-image set into a deep residual network, performs layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, obtains the layer-by-layer feature extraction result and the cross-layer feature extraction result of the target image corresponding to each type of sub-image set, then fuses the layer-by-layer feature extraction result and the cross-layer feature extraction result to obtain the first fusion feature of the target image corresponding to each type of sub-image set, samples the first fusion feature to obtain the sampled fusion feature of the target image corresponding to each type of sub-image set, and determines the classification result of the target image corresponding to each type of sub-image set based on the sampled fusion feature. By dividing the image set into multiple sub-image sets and determining the target image corresponding to each type of sub-image set from each type of sub-image set, the image set can be deduplicated. By setting a deep residual network, the efficiency and accuracy of image classification can be improved and the cost of image classification can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 A schematic diagram of a flow chart of an image data processing method provided in an embodiment of the present application;

[0064] Figure 2 A schematic diagram of a process for extracting a second fusion feature provided in an embodiment of the present application;

[0065] Figure 3 A training flow chart of a deep residual network provided in an embodiment of the present application;

[0066] Figure 4 A structural block diagram of an image data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] The term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present application. In the description of the embodiments of the present application, it should be understood that the orientation or positional relationship indicated by the terms "upper", "lower", "top", "bottom", etc. is based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may include one or more of the features explicitly or implicitly. Moreover, the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here.

[0069] See also Figure 1 , which is a flow chart of an image data processing method provided in an embodiment of the present application, the image data processing method comprising:

[0070] Step S101: obtaining an image set;

[0071] Step S102: classifying the image set to obtain multiple categories of sub-image sets corresponding to the image set;

[0072] Step S103: from each type of the sub-image set, respectively determine the target image corresponding to each type of the sub-image set;

[0073] Step S104: inputting the target images into the deep residual network respectively, performing layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and obtaining layer-by-layer feature extraction results and cross-layer feature extraction results of the target images corresponding to each type of the sub-image set;

[0074] Step S105: fusing the layer-by-layer feature extraction result and the cross-layer feature extraction result to obtain a first fused feature of the target image corresponding to each type of the sub-image set;

[0075] Step S106: performing sampling processing on the first fusion features respectively to obtain sampling fusion features of the target image corresponding to each type of the sub-image set;

[0076] Step S107: Based on the sampled fusion features, a classification result of the target image corresponding to each type of the sub-image set is determined.

[0077] In a specific embodiment, an image set is obtained through step S101; the image set is classified through step S102 to obtain multiple categories of sub-image sets; through step S103, the target image corresponding to each category of sub-image set is determined, and optionally, the target image is an image randomly selected from the corresponding sub-image set; through steps S102 and S103, the image can be deduplicated to effectively improve the quality of the image data set; in step S104, the target images are respectively input into the deep residual network to perform several layers of feature traversal extraction, wherein the several layers of feature traversal extraction include layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, that is, the target images are respectively subjected to layer-by-layer feature traversal extraction and cross-layer feature traversal extraction to obtain layer-by-layer feature extraction results and cross-layer feature extraction results of the target images corresponding to each category of sub-image set. Since step S104 uses the deep residual network to perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, more image features can be extracted to improve classification accuracy, and the deep residual network feature extraction process is simple and can improve image quality. The efficiency and accuracy of classification are improved, the cost of image classification is reduced, and the traditional deep learning algorithm is avoided, which causes some features of the image features to disappear during the layer-by-layer extraction process when only layer-by-layer feature traversal extraction is performed, thereby generating classification errors and reducing classification accuracy; the layer-by-layer feature extraction results and the cross-layer feature extraction results are fused through step S105 to obtain the first fused features of the target image corresponding to each type of sub-image set. Compared with the traditional deep algorithm, only layer-by-layer feature traversal extraction may cause some features to disappear, and more image features can be retained. Therefore, the first fused features obtained are also more accurate than those under the traditional deep algorithm; the first fused features are sampled and processed respectively through step S106 to obtain the sampled fused features of the target image corresponding to each type of sub-image set, which is conducive to reducing errors; in step S107, based on the sampled fused features, the classification results of the target images corresponding to each type of sub-image set are determined. Since the sampled fused features are the results of the first fused features after reducing the errors, the classification results determined by the sampled fused features are more accurate.

[0078] This embodiment uses a deep residual network to extract image features through layer-by-layer feature traversal extraction and cross-layer feature traversal extraction. More original detail features can be retained during the extraction process to improve classification accuracy. In addition, the deep residual network feature extraction process is simple, which can improve the efficiency and accuracy of image classification and reduce the cost of image classification. It avoids the situation where only layer-by-layer feature traversal extraction is performed under the traditional deep learning algorithm, causing some features of the image to disappear during the layer-by-layer extraction process, thereby generating classification errors and reducing the classification accuracy.

[0079] See also Figure 2, which is a schematic diagram of a process of extracting a second fusion feature provided in an embodiment of the present application, wherein the cross-layer feature traversal extraction in the embodiment of the present application includes odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction;

[0080] In an optional embodiment, in the above step S104, the target images are respectively input into the deep residual network, and layer-by-layer feature traversal extraction and cross-layer feature traversal extraction are performed to obtain the layer-by-layer feature extraction results and cross-layer feature extraction results of the target image corresponding to each type of the sub-image set, which may include:

[0081] The target images are respectively input into the deep residual network to perform layer-by-layer feature traversal extraction, odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction, so as to obtain layer-by-layer feature extraction results, odd-numbered layer feature extraction results and even-numbered layer feature extraction results of the target image corresponding to each type of the sub-image set.

[0082] For example, the target images are respectively input into the deep residual network, and the first layer features, the second layer features, ... are traversed and extracted layer by layer in sequence, and several results obtained by the layer-by-layer feature traversal extraction are fused to obtain the layer-by-layer feature extraction results of the target image corresponding to each type of the sub-image set, the first layer features, the third layer features, ... are traversed and extracted in sequence, and several results obtained by the odd-numbered layer feature traversal extraction are fused to obtain the odd-numbered layer feature extraction results of the target image corresponding to each type of the sub-image set, and the second layer features, the fourth layer features, ... are traversed and extracted in sequence, and several results obtained by the even-numbered layer feature traversal extraction are fused to obtain the even-numbered layer feature extraction results of the target image corresponding to each type of the sub-image set; by setting the cross-layer feature traversal extraction to odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction, it can be ensured that each layer of features can be extracted twice, thereby further improving the classification accuracy.

[0083] In an optional embodiment, in the above step S105, the above fusion of the layer-by-layer feature extraction result and the cross-layer feature extraction result to obtain the first fusion feature of the target image corresponding to each type of the sub-image set may include:

[0084] The layer-by-layer feature extraction results are respectively fused with the odd-numbered layer feature extraction results and the even-numbered layer feature extraction results to obtain odd-numbered layer fusion features and even-numbered layer fusion features;

[0085] The odd-numbered layer fusion features and the even-numbered layer fusion features are used as the first fusion features of the target image corresponding to each type of the sub-image set.

[0086] For example, the layer-by-layer feature extraction results are fused with the odd-layer feature extraction results to obtain the odd-layer fusion features, and the layer-by-layer feature extraction results are fused with the even-layer feature extraction results to obtain the even-layer fusion features; the layer-by-layer feature extraction results are fused with the odd-layer feature extraction results and the even-layer feature extraction results respectively, so that the layer-by-layer feature extraction results and the cross-layer feature extraction results can be fully integrated to further improve the classification accuracy.

[0087] In an optional embodiment, in the above step S106, the above sampling process is performed on the first fusion features to obtain the sampled fusion features of the target image corresponding to each type of the sub-image set, which may include:

[0088] Performing upsampling and downsampling processing on the odd-numbered layer fusion features respectively to obtain odd-numbered layer upsampling features and odd-numbered layer downsampling features;

[0089] Performing up-sampling and down-sampling processing on the even-numbered layer fusion features respectively to obtain even-numbered layer up-sampling features and even-numbered layer down-sampling features;

[0090] The odd-numbered layer up-sampling features, the odd-numbered layer down-sampling features, the even-numbered layer up-sampling features and the even-numbered layer down-sampling features are used as sampling fusion features of the target image corresponding to each type of the sub-image set.

[0091] For example, the odd-numbered layer fusion features are upsampled to obtain the odd-numbered layer upsampled features, the odd-numbered layer fusion features are downsampled to obtain the odd-numbered layer downsampled features, the even-numbered layer fusion features are upsampled to obtain the even-numbered layer upsampled features, and the even-numbered layer fusion features are downsampled to obtain the even-numbered layer downsampled features.

[0092] In an optional embodiment, in the above step S107, the above determining the classification result of the target image corresponding to each type of the sub-image set based on the sampling fusion feature may include:

[0093] The odd-numbered layer up-sampling features, the odd-numbered layer down-sampling features, the even-numbered layer up-sampling features, and the even-numbered layer down-sampling features are fused to obtain a second fused feature of the target image corresponding to each type of the sub-image set;

[0094] A classification result of the target image is determined based on the second fusion feature.

[0095] For example, the odd-numbered layer fusion features and the even-numbered layer fusion features are upsampled and downsampled respectively to obtain the odd-numbered layer upsampled features, the odd-numbered layer downsampled features, the even-numbered layer upsampled features and the even-numbered layer downsampled features, and then the odd-numbered layer upsampled features, the odd-numbered layer downsampled features, the even-numbered layer upsampled features and the even-numbered layer downsampled features are fused to obtain the second fusion features of the target image corresponding to each type of sub-image set. The second fusion features obtained by fusing the first fusion features after the sampling process are more conducive to reducing errors than the second fusion features obtained by directly fusing the first fusion features.

[0096] See also Figure 3 , which shows a training flow chart of a deep residual network provided in an embodiment of the present application, wherein the training of the deep residual network includes:

[0097] Step S201: Acquire a sample image, wherein the sample image is annotated with sample features;

[0098] Step S202: inputting the sample image into a preset deep residual network to perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and obtaining a preset layer-by-layer feature extraction result and a preset cross-layer feature extraction result;

[0099] Step S203: Fusing the preset layer-by-layer feature extraction result and the preset cross-layer feature extraction result to obtain a first preset fusion feature;

[0100] Step S204: performing sampling processing on the first preset fusion feature to obtain a sampled preset fusion feature;

[0101] Step S205: fusing the sampled preset fusion features to obtain a second preset fusion feature;

[0102] Step S206: determining loss information between the second preset fusion feature and the sample feature;

[0103] Step S207: determine whether the loss information meets the preset conditions, if yes, execute step S208, if no, repeat steps S201 to S207;

[0104] Step S208: Obtain the deep residual network.

[0105] In a specific embodiment, a sample image is obtained through step S201, and the sample image is annotated with sample features; through step S202, the sample image is input into a preset deep residual network to perform several layers of feature traversal extraction, wherein the several layers of feature traversal extraction include layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and optionally, the cross-layer feature traversal extraction includes odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction, that is, the sample image is subjected to layer-by-layer feature traversal extraction, odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction, respectively, to obtain a preset layer-by-layer feature extraction result and a preset cross-layer feature traversal extraction result. feature extraction results, wherein the preset cross-layer feature extraction results include preset odd-numbered layer feature extraction results and preset even-numbered layer feature extraction results; in step S203, the preset layer-by-layer feature extraction results and the preset cross-layer feature extraction results are fused, and optionally, the preset cross-layer feature extraction results include preset odd-numbered layer feature extraction results and preset even-numbered layer feature extraction results, that is, the preset layer-by-layer feature extraction results, the preset odd-numbered layer feature extraction results and the preset even-numbered layer feature extraction results are fused to obtain preset odd-numbered layer fusion features and preset even-numbered layer fusion features, and the preset odd-numbered layer fusion features and the preset even-numbered layer fusion features are fused. The even-numbered layer fusion feature is set as the first preset fusion feature; in step S204, the first preset fusion feature is sampled, and optionally, the sampling process includes upsampling and downsampling to obtain a sampled preset fusion feature, wherein the sampled preset fusion feature includes a preset odd-numbered layer upsampling feature, a preset odd-numbered layer downsampling feature, a preset even-numbered layer upsampling feature, and a preset even-numbered layer downsampling feature; through step S205, the sampled preset fusion feature is fused, that is, the preset odd-numbered layer upsampling feature, the preset odd-numbered layer downsampling feature, the preset even-numbered layer upsampling feature, and the preset even-numbered layer downsampling feature are fused. The down-sampled features of several layers are fused to obtain a second preset fusion feature; in step S206, the loss information can be determined by calculating the similarity between the second preset fusion feature and the sample feature, wherein the similarity between the second preset fusion feature and the sample feature can be determined by the Euclidean distance between the second preset fusion feature and the sample feature; through step S207, it is determined whether the loss information meets the preset condition, if so, step S208 is executed, if not, steps S201 to S207 are repeated until the loss information meets the preset condition; through step S208, a deep residual network is obtained.

[0106] In an optional embodiment, in the above step S102, the above classification of the image set to obtain multiple types of sub-image sets corresponding to the image set may include:

[0107] The image set is classified based on a K-means clustering algorithm to obtain multiple categories of sub-image sets corresponding to the image set.

[0108] Classifying the image set using the K-means clustering algorithm is helpful to improve the accuracy of classification.

[0109] In an optional embodiment, in the above step S102, classifying the image set based on the K-means clustering algorithm to obtain multiple types of sub-image sets corresponding to the image set may include:

[0110] Acquire a preset number of images from the image set, and use the preset number of images as a preset number of current cluster centers; the preset number is set according to the complexity of the scene contained in the image set;

[0111] Calculating the distance between each image in the image set and a preset number of the current cluster centers;

[0112] According to the distances between each image and a preset number of the current cluster centers, a target cluster center corresponding to each image is obtained from the preset number of the current cluster centers, and the distance between the target cluster center and each image is less than a preset distance threshold;

[0113] Assigning each image to a target cluster center corresponding to each image to obtain a preset number of current candidate image sets;

[0114] Calculating the cluster center of each of the current candidate image sets, and re-using the calculated candidate cluster centers as a preset number of current cluster centers;

[0115] Repeat the step of calculating the distance between each image in the image set and a preset number of the current cluster centers, and the step of using the calculated candidate cluster centers as the preset number of current cluster centers, until the images in the current candidate image set meet the preset conditions or the number of repetitions meets the preset number threshold.

[0116] For example, there are 20 images in an image set. 5 images are obtained from the image set and set as the current cluster centers. Then, the distances between the remaining 15 images in the image set and the 5 current cluster centers are calculated. When the distance between an image and a current cluster center is less than a preset distance threshold, the current cluster center is the target cluster center of the image. According to this rule, the 15 images are respectively assigned to the 5 current cluster centers to form 5 current candidate image sets. The cluster center of each current candidate image set is calculated, and the 5 calculated candidate cluster centers are used as the new current cluster centers. The previous steps are repeated until the images in the current candidate image set meet the preset conditions or the number of repetitions meets the preset number threshold.

[0117] In an optional embodiment, in the above step S101, the above acquiring the image set may include:

[0118] Obtain traffic video data;

[0119] Deduplication processing is performed on the traffic video data to obtain deduplication image data;

[0120] The image set is obtained from the deduplicated image data.

[0121] For example, the traffic video data may be sliced ​​and framed to obtain image data after slicing and frame extraction, deduplication processing may be performed on the image data after slicing and frame extraction to obtain deduplicated image data, and then an image set may be obtained from the deduplicated image data.

[0122] In an optional embodiment, in the above step S103, respectively determining the target image corresponding to each type of the sub-image set from each type of the sub-image set may include:

[0123] Any image in each category of the sub-image set is used as the target image corresponding to each category of the sub-image set.

[0124] For example, an image is randomly selected as the target image of the sub-image set.

[0125] It can be seen from the above technical solutions of the embodiments of the present application that the embodiments of the present application utilize a deep residual network to perform image feature extraction through layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, which can retain more original detail features during the extraction process and improve classification accuracy. In addition, the feature extraction process of the deep residual network is simple, which can improve the efficiency and accuracy of image classification and reduce the cost of image classification. It avoids the situation where only layer-by-layer feature traversal extraction is performed under the traditional deep learning algorithm, causing some features of the image to disappear during the layer-by-layer extraction process, thereby generating classification errors and reducing the classification accuracy.

[0126] Corresponding to the image data processing method provided in the above embodiment, the embodiment of the present application also provides an image data processing device. Since the image data processing device provided in the embodiment of the present application corresponds to the image data processing method provided in the above embodiment, the implementation method of the aforementioned image data processing method is also applicable to the image data processing device provided in this embodiment, and will not be described in detail in this embodiment.

[0127] See also Figure 4 , which is a structural block diagram of an image data processing device proposed in an embodiment of the present application, the image data processing device includes:

[0128] An image set acquisition module, used for acquiring an image set;

[0129] An image classification module, used to classify the image set to obtain multiple types of sub-image sets corresponding to the image set;

[0130] A target image determination module, used to determine the target image corresponding to each type of the sub-image set from each type of the sub-image set;

[0131] An image feature extraction module is used to input the target images into a deep residual network respectively, perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and obtain layer-by-layer feature extraction results and cross-layer feature extraction results of the target images corresponding to each type of the sub-image set;

[0132] A first feature fusion module, used for fusing the layer-by-layer feature extraction result and the cross-layer feature extraction result to obtain a first fusion feature of the target image corresponding to each type of the sub-image set;

[0133] A fusion feature sampling module, used for sampling the first fusion features respectively to obtain sampled fusion features of the target image corresponding to each type of the sub-image set;

[0134] The classification result determination module is used to determine the classification result of the target image corresponding to each type of the sub-image set based on the sampling fusion features.

[0135] In a specific embodiment, an image set is acquired through an image set acquisition module; the image set is classified through an image classification module to obtain multiple categories of sub-image sets; a target image corresponding to each category of sub-image sets is determined through a target image determination module, and optionally, the target image is an image randomly selected from the corresponding sub-image set; through the image classification module and the target image determination module, the image can be deduplicated to effectively improve the quality of the image data set; in the image feature extraction module, the target images are respectively input into a deep residual network to perform several layers of feature traversal extraction, wherein the several layers of feature traversal extraction include layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, that is, the target images are respectively subjected to layer-by-layer feature traversal extraction and cross-layer feature traversal extraction to obtain layer-by-layer feature extraction results and cross-layer feature extraction results of the target images corresponding to each category of sub-image sets. Since the image feature extraction module adopts a deep residual network to perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, more image features can be extracted to improve classification accuracy, and the deep residual network feature extraction process is simple and can improve The efficiency and accuracy of image classification are improved, the cost of image classification is reduced, and the situation that only layer-by-layer feature traversal extraction is performed under the traditional deep learning algorithm, causing some features of the image features to disappear during the layer-by-layer extraction process, thereby generating classification errors and reducing the classification accuracy; the layer-by-layer feature extraction results and the cross-layer feature extraction results are fused through the first feature fusion module to obtain the first fusion feature of the target image corresponding to each type of sub-image set. Compared with the traditional deep algorithm, only layer-by-layer feature traversal extraction may cause some features to disappear, and more image features can be retained. Therefore, the first fusion feature obtained is also more accurate than that under the traditional deep algorithm; the first fusion feature is sampled and processed by the fusion feature sampling module respectively to obtain the sampled fusion feature of the target image corresponding to each type of sub-image set, which is conducive to reducing errors; in the classification result determination module, based on the sampled fusion feature, the classification result of the target image corresponding to each type of sub-image set is determined. Since the sampled fusion feature is the result after the first fusion feature reduces the error, the classification result determined by the sampled fusion feature is more accurate.

[0136] This embodiment uses a deep residual network to extract image features through layer-by-layer feature traversal extraction and cross-layer feature traversal extraction. More original detail features can be retained during the extraction process to improve classification accuracy. In addition, the deep residual network feature extraction process is simple, which can improve the efficiency and accuracy of image classification and reduce the cost of image classification. It avoids the situation where only layer-by-layer feature traversal extraction is performed under the traditional deep learning algorithm, causing some features of the image to disappear during the layer-by-layer extraction process, thereby generating classification errors and reducing the classification accuracy.

[0137] Specifically, this embodiment may also include a training module for a deep residual network, which specifically includes:

[0138] A sample image acquisition module, used to acquire a sample image, wherein the sample image is annotated with sample features;

[0139] A preset feature result extraction module, used for inputting the sample image into a preset deep residual network to perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and obtaining a preset layer-by-layer feature extraction result and a preset cross-layer feature extraction result;

[0140] A first preset feature fusion module, used to fuse the preset layer-by-layer feature extraction result and the preset cross-layer feature extraction result to obtain a first preset fusion feature;

[0141] A sampling processing module, used for performing sampling processing on the first preset fusion feature to obtain a sampled preset fusion feature;

[0142] A second preset feature fusion module, used to fuse the sampled preset fusion features to obtain a second preset fusion feature;

[0143] A loss information determination module, used to determine the loss information between the second preset fusion feature and the sample feature;

[0144] A judgment module, used to judge whether the loss information meets the preset conditions;

[0145] The deep residual network acquisition module is used to obtain the deep residual network.

[0146] In a specific embodiment, a sample image is obtained by a sample image acquisition module, and the sample image is annotated with sample features; the sample image is input into a preset deep residual network through a preset feature result extraction module to perform several layers of feature traversal extraction, wherein the several layers of feature traversal extraction include layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and optionally, the cross-layer feature traversal extraction includes odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction, that is, the sample image is subjected to layer-by-layer feature traversal extraction, odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction, respectively, to obtain a preset layer-by-layer feature traversal extraction. The preset cross-layer feature extraction result and the preset cross-layer feature extraction result are respectively obtained by: extracting the preset layer-by-layer feature and the preset cross-layer feature extraction result, wherein the preset cross-layer feature extraction result includes the preset odd-numbered layer feature extraction result and the preset even-numbered layer feature extraction result; in the first preset feature fusion module, the preset layer-by-layer feature extraction result and the preset cross-layer feature extraction result are fused, and optionally, the preset cross-layer feature extraction result includes the preset odd-numbered layer feature extraction result and the preset even-numbered layer feature extraction result, that is, the preset layer-by-layer feature extraction result, the preset odd-numbered layer feature extraction result and the preset even-numbered layer feature extraction result are fused to obtain the preset odd-numbered layer fusion feature and the preset even-numbered layer fusion feature. The preset odd-layer fusion feature and the preset even-layer fusion feature are used as the first preset fusion feature; in the sampling processing module, the first preset fusion feature is sampled and processed, and optionally, the sampling processing includes upsampling processing and downsampling processing to obtain a sampled preset fusion feature, wherein the sampled preset fusion feature includes a preset odd-layer upsampling feature, a preset odd-layer downsampling feature, a preset even-layer upsampling feature and a preset even-layer downsampling feature; the sampled preset fusion feature is fused by a second preset feature fusion module, that is, the preset odd-layer upsampling feature, the preset odd-layer downsampling feature, the preset even-layer upsampling feature and the preset even-layer downsampling feature are fused to obtain a second preset fusion feature; in the loss information determination module, the loss information can be determined by calculating the similarity between the second preset fusion feature and the sample feature, wherein the similarity between the second preset fusion feature and the sample feature can be determined by the Euclidean distance between the second preset fusion feature and the sample feature; the judgment module is used to judge whether the loss information meets the preset condition; until the loss information meets the preset condition, the deep residual network is obtained by the deep residual network acquisition module.

[0147] Specifically, the embodiment of the present application may further include a module for acquiring an image set, and the module specifically includes:

[0148] A video data acquisition module, used to acquire traffic video data;

[0149] The deduplication module is used to perform deduplication processing on the traffic video data to obtain deduplicated image data.

[0150] In a specific embodiment, traffic video data is first acquired through a video data acquisition module, and then the traffic video data is sliced ​​and framed through a deduplication module to obtain image data after slicing and frame extraction, deduplication processing is performed on the image data after slicing and frame extraction to obtain deduplicated image data, and then an image set is acquired from the deduplication module through an image set acquisition module.

[0151] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0152] The image data processing device of the embodiment of the present application utilizes a deep residual network to perform image feature extraction through layer-by-layer feature traversal extraction and cross-layer feature traversal extraction. In the process of extraction, more original detail features can be retained to improve classification accuracy. In addition, the feature extraction process of the deep residual network is simple, which can improve the efficiency and accuracy of image classification and reduce the cost of image classification. It avoids the situation in which only layer-by-layer feature traversal extraction is performed under the traditional deep learning algorithm, causing some features of the image to disappear during the layer-by-layer extraction process, thereby generating classification errors and reducing the classification accuracy.

[0153] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of this specification are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-image data processing and parallel processing are also possible or may be advantageous.

[0154] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0155] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for processing image data, characterized in that: The method comprises: Get the image set; Classifying the image set to obtain multiple categories of sub-image sets corresponding to the image set; From each type of the sub-image set, respectively determine the target image corresponding to each type of the sub-image set; The target images are respectively input into the deep residual network to perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and the layer-by-layer feature extraction results and cross-layer feature extraction results of the target images corresponding to each type of the sub-image set are obtained; the cross-layer feature traversal extraction includes odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction; the cross-layer feature extraction results include odd-numbered layer feature extraction results and even-numbered layer feature extraction results; Fusing the layer-by-layer feature extraction result and the cross-layer feature extraction result to obtain a first fused feature of the target image corresponding to each type of the sub-image set; The first fusion feature includes an odd-numbered layer fusion feature and an even-numbered layer fusion feature; The first fusion features are sampled and processed respectively to obtain sampled fusion features of the target image corresponding to each type of the sub-image set; The sampling fusion features include odd-numbered layer upsampling features, odd-numbered layer downsampling features, even-numbered layer upsampling features and even-numbered layer downsampling features; Based on the sampled fusion features, a classification result of the target image corresponding to each type of the sub-image set is determined.

2. The image data processing method according to claim 1, characterized in that: The cross-layer feature traversal extraction includes odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction; The target images are respectively input into the deep residual network to perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and the layer-by-layer feature extraction results and cross-layer feature extraction results of the target images corresponding to each type of the sub-image set are obtained, including: The target images are respectively input into the deep residual network to perform layer-by-layer feature traversal extraction, odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction, so as to obtain layer-by-layer feature extraction results, odd-numbered layer feature extraction results and even-numbered layer feature extraction results of the target image corresponding to each type of the sub-image set.

3. The image data processing method according to claim 2, characterized in that: The step of fusing the layer-by-layer feature extraction result and the cross-layer feature extraction result to obtain a first fused feature of the target image corresponding to each type of the sub-image set includes: The layer-by-layer feature extraction results are respectively fused with the odd-numbered layer feature extraction results and the even-numbered layer feature extraction results to obtain odd-numbered layer fusion features and even-numbered layer fusion features; The odd-numbered layer fusion features and the even-numbered layer fusion features are used as the first fusion features of the target image corresponding to each type of the sub-image set.

4. The image data processing method according to claim 3, characterized in that: The sampling process is performed on the first fusion features respectively to obtain the sampled fusion features of the target image corresponding to each type of the sub-image set, including: Performing upsampling and downsampling processing on the odd-numbered layer fusion features respectively to obtain odd-numbered layer upsampling features and odd-numbered layer downsampling features; Performing up-sampling and down-sampling processing on the even-numbered layer fusion features respectively to obtain even-numbered layer up-sampling features and even-numbered layer down-sampling features; The odd-numbered layer up-sampling features, the odd-numbered layer down-sampling features, the even-numbered layer up-sampling features and the even-numbered layer down-sampling features are used as sampling fusion features of the target image corresponding to each type of the sub-image set.

5. The image data processing method according to claim 4, characterized in that: The step of determining the classification result of the target image corresponding to each type of the sub-image set based on the sampling fusion feature includes: The odd-numbered layer up-sampling features, the odd-numbered layer down-sampling features, the even-numbered layer up-sampling features, and the even-numbered layer down-sampling features are fused to obtain a second fused feature of the target image corresponding to each type of the sub-image set; A classification result of the target image is determined based on the second fusion feature.

6. The image data processing method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire a sample image, wherein the sample image is annotated with sample features; Inputting the sample image into a preset deep residual network to perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, to obtain a preset layer-by-layer feature extraction result and a preset cross-layer feature extraction result; Fusing the preset layer-by-layer feature extraction result and the preset cross-layer feature extraction result to obtain a first preset fusion feature; Performing sampling processing on the first preset fusion feature to obtain a sampled preset fusion feature; Fusing the sampled preset fusion features to obtain a second preset fusion feature; The preset deep residual network is trained according to the loss information between the second preset fusion feature and the sample feature to obtain the deep residual network.

7. The image data processing method according to claim 1, characterized in that: The step of classifying the image set to obtain multiple types of sub-image sets corresponding to the image set includes: The image set is classified based on a K-means clustering algorithm to obtain multiple categories of sub-image sets corresponding to the image set.

8. The image data processing method according to claim 7, characterized in that: The image set is classified based on the K-means clustering algorithm to obtain multiple sub-image sets corresponding to the image set, including: Acquire a preset number of images from the image set, and use the preset number of images as a preset number of current cluster centers; the preset number is set according to the complexity of the scene contained in the image set; Calculating the distance between each image in the image set and a preset number of the current cluster centers; According to the distances between each image and a preset number of the current cluster centers, a target cluster center corresponding to each image is obtained from the preset number of the current cluster centers, and the distance between the target cluster center and each image is less than a preset distance threshold; Assigning each image to a target cluster center corresponding to each image to obtain a preset number of current candidate image sets; Calculating the cluster center of each of the current candidate image sets, and re-using the calculated candidate cluster centers as a preset number of current cluster centers; Repeat the step of calculating the distance between each image in the image set and a preset number of the current cluster centers, and the step of using the calculated candidate cluster centers as the preset number of current cluster centers, until the images in the current candidate image set meet the preset conditions or the number of repetitions meets the preset number threshold.

9. The image data processing method according to claim 1, characterized in that: The acquiring of the image set comprises: Obtain traffic video data; Deduplication processing is performed on the traffic video data to obtain deduplication image data; The image set is obtained from the deduplicated image data.

10. The image data processing method according to claim 1, characterized in that: The step of determining the target image corresponding to each type of the sub-image set from each type of the sub-image set comprises: Any image in each category of the sub-image set is used as the target image corresponding to each category of the sub-image set.

11. An image data processing device, characterized in that: The device comprises: An image set acquisition module, used for acquiring an image set; An image classification module, used to classify the image set to obtain multiple types of sub-image sets corresponding to the image set; A target image determination module, used to determine the target image corresponding to each type of the sub-image set from each type of the sub-image set; An image feature extraction module is used to input the target images into a deep residual network respectively, perform layer-by-layer feature traversal extraction and cross-layer feature traversal extraction, and obtain layer-by-layer feature extraction results and cross-layer feature extraction results of the target images corresponding to each type of sub-image set; the cross-layer feature traversal extraction includes odd-numbered layer feature traversal extraction and even-numbered layer feature traversal extraction; the cross-layer feature extraction results include odd-numbered layer feature extraction results and even-numbered layer feature extraction results; A first feature fusion module, used for fusing the layer-by-layer feature extraction result and the cross-layer feature extraction result to obtain a first fusion feature of the target image corresponding to each type of the sub-image set; the first fusion feature includes an odd-numbered layer fusion feature and an even-numbered layer fusion feature; A fusion feature sampling module is used to sample the first fusion features respectively to obtain sampling fusion features of the target image corresponding to each type of the sub-image set; the sampling fusion features include odd-numbered layer upsampling features, odd-numbered layer downsampling features, even-numbered layer upsampling features and even-numbered layer downsampling features; The classification result determination module is used to determine the classification result of the target image corresponding to each type of the sub-image set based on the sampling fusion features.

12. The image data processing device according to claim 11, characterized in that: The device also includes: A video data acquisition module, used to acquire traffic video data; The deduplication module is used to perform deduplication processing on the traffic video data to obtain deduplicated image data.

Images