Data processing method and device, product and terminal

Abstract

The invention provides a data processing method and device, a product and a terminal, and belongs to the technical field of image processing. The method comprises the steps that according to feature vectors of all to-be-processed images in a to-be-processed image set, the to-be-processed images in the to-be-processed image set are filtered, a candidate image set is generated, and the similarity of any two images in the candidate image set is smaller than a similarity threshold value; images in the candidate image set are input into a license plate and face hybrid screening model, a to-be-labeled image set is output, and to-be-labeled images in the to-be-labeled image set comprise face images and / or license plate images. The invention aims to improve the efficiency of the screening process of the to-be-labeled image.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of image data processing, and specifically, to a data processing method, device, product, and terminal. Background Art

[0002] With the rapid development of information technology, the risk of information leakage is increasing. Data leakage may lead to serious consequences. Measures need to be taken to protect consumers' personal data. License plate information and facial information are sensitive data. For example, in the actual demonstration of the traffic monitoring video system, if the desensitization of license plate information or facial information is ignored, the user's personal information will be leaked.

[0003] With the development of deep learning algorithms, technologies such as deep learning can be used to detect and identify sensitive information, and even to erase or mask sensitive data.

[0004] However, in the development process of deep learning models, a large amount of labeled data is required as training samples to train the model. In the actual process of labeling training samples, the process of screening out image data to be labeled that can be further manually labeled from massive data is very cumbersome, resulting in low efficiency in the screening process of image data to be labeled. Summary of the Invention

[0005] The embodiments of the present application provide a data processing method, device, product, and terminal, which are intended to improve the efficiency of the screening process of images to be labeled.

[0006] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising: Filtering the images to be processed in the image set to be processed according to the feature vectors of each image to be processed in the image set to generate a candidate image set, wherein the similarity between any two images in the candidate image set is less than a similarity threshold; The images in the candidate image set are input into a license plate and face mixed screening model, and a set of images to be labeled is output, where the images to be labeled in the set of images to be labeled include face images and / or license plate images.

[0007] Optionally, the method further includes: Obtain original road sampling data; The original road sampling data is preprocessed to generate a to-be-processed image set including a plurality of to-be-processed images.

[0008] Optionally, the original road sampling data includes an original road sampling video, and preprocessing the original road sampling data to generate a to-be-processed image set including a plurality of to-be-processed images includes: Extracting multiple target videos from the original road video, wherein the target videos include at least one vehicle or one person; Generating an initial image sequence set according to the multiple target videos; The initial image sequence set is preprocessed to generate a to-be-processed image set including a plurality of to-be-processed images.

[0009] Optionally, preprocessing the initial image sequence set to generate a to-be-processed image set comprising a plurality of to-be-processed images includes: In the initial image sequence set, each image is classified according to its name, and multiple images in the same classification are obtained by the same image acquisition device; Eliminating damaged images from the initial image sequence set; Changing the formats of all images in the initial image sequence set to a target image format; The resolutions of all images in the initial image sequence set are modified to a target resolution.

[0010] Optionally, before filtering the images to be processed in the image set to be processed based on the feature vector of each image to be processed in the image set to be processed and generating the candidate image set, the method further includes: Input each image to be processed in the set of images to be processed into the MobileNetV2-Reid neural network, and obtain the feature vector corresponding to each image to be processed output by the feature extraction layer of the MobileNetV2-Reid neural network.

[0011] Optionally, filtering the images to be processed in the set of images to be processed according to the feature vector of each image to be processed in the set of images to be processed to generate a set of candidate images includes: Using the first image in the set of images to be processed as a reference image, and moving the reference image into the set of candidate images; Calculating the similarity between the image to be processed and the reference image based on the feature vectors of the image to be processed and the feature vectors of the reference image in the order of arrangement of the image to be processed in the image set to be processed; When the similarity between any image to be processed and the reference image is greater than a similarity threshold, the image to be processed is removed from the set of images to be processed; When the similarity between any image to be processed and the reference image is less than or equal to a similarity threshold, the image is updated as the current reference image; Until the set of images to be processed is empty, the candidate image set is obtained.

[0012] Optionally, calculating the similarity between the image to be processed and the reference image according to the feature vector of the image to be processed and the feature vector of the reference image includes: The cosine similarity between the image to be processed and the reference image is calculated according to the feature vector of the image to be processed and the feature vector of the reference image, and the cosine similarity is used as the similarity.

[0013] Optionally, the method further includes: The YOLOv5 detection model is iteratively trained for a target number of times based on the face detection benchmark dataset and the road license plate dataset to obtain a fine-tuned YOLOv5 detection model as a license plate face hybrid screening model.

[0014] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising: an image filtering module, configured to filter the images to be processed in the image set to be processed according to the feature vectors of the images to be processed, and generate a candidate image set, wherein the similarity between any two images in the candidate image set is less than a similarity threshold; The data output module is used to input the images in the candidate image set into the license plate and face mixed screening model, and output a set of images to be labeled, wherein the images to be labeled in the set of images to be labeled include face images and / or license plate images.

[0015] In a third aspect, an embodiment of the present application provides an electronic device comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the data processing method described in the first aspect of the embodiment is executed.

[0016] In a fourth aspect, an embodiment of the present application provides a non-volatile readable storage medium, wherein the non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the data processing method described in the first aspect of the embodiment is executed.

[0017] In a fifth aspect, an embodiment of the present application provides a terminal, which includes a data processing device described in the second aspect of the embodiment, or is used to execute the data processing method described in the first aspect of the embodiment.

[0018] Optionally, the terminal is a vehicle or a server.

[0019] Beneficial effects: This method filters similar images in a set of images to be processed based on the feature vectors of each image to be processed to generate a candidate image set. If the similarity between any two images in the candidate image set is less than a similarity threshold, similar images can be eliminated, that is, the image processing amount can be reduced and the efficiency of generating images to be labeled can be improved. The images in the candidate image set are then input into a license plate and face mixed screening model. The license plate and face mixed screening model is used to quickly select images to be labeled that include face images and / or license plate images from the candidate image set, thereby improving the efficiency of the screening process of the images to be labeled. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 This is a flowchart of the steps of the data processing method proposed in one embodiment of the present application; Figure 2 This is a schematic diagram of the structure of the MobileNetV2-Reid neural network proposed in one embodiment of the present application; Figure 3 is an execution flow chart of a similar image filtering process provided by an embodiment of the present application; Figure 4 This is a functional module diagram of a data processing device proposed in one embodiment of the present application; Figure 5 is a schematic diagram of an electronic device proposed in one embodiment of the present application; Figure 6 Schematic diagram of a non-volatile readable storage medium provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] With the rapid development of information technology, the risk of information leakage is increasing. Data leakage may lead to serious consequences. Enterprises need to take measures to protect consumers' personal data. For example, license plate information and facial information are sensitive information. In the actual demonstration of the traffic monitoring video system, if sensitive information such as license plate information and facial information is not desensitized, information leakage will occur.

[0024] With the development of deep learning technology, deep learning models can now be used to detect and identify sensitive information and erase sensitive information. However, the development of deep learning models requires a large amount of manually annotated data as training samples to train a robust model.

[0025] However, the current process of screening massive amounts of data to obtain training samples for training deep learning models is very cumbersome. For example, first, a large number of images are obtained, and then images containing vehicles and faces are selected from the massive image set. The license plate information and face information are then annotated in the images as training samples after annotation. This results in a very low efficiency in the screening process for obtaining images to be annotated.

[0026] In order to improve the efficiency of the screening process of images to be annotated, this embodiment provides a data de-redundancy method. The images to be annotated obtained by this method can be further manually annotated to obtain training samples for training deep learning desensitization models, thereby helping to protect users' privacy information.

[0027] Reference Figure 1 , shows a flowchart of the steps of a data processing method in an embodiment of the present application, and the method may specifically include the following steps: S101: filtering the images to be processed in the image set to be processed according to the feature vectors of each image to be processed in the image set to generate a candidate image set, wherein the similarity between any two images in the candidate image set is less than a similarity threshold.

[0028] Specifically, the set of images to be processed includes multiple images to be processed, and the images to be processed include vehicle or person images.

[0029] In a feasible embodiment, the method also includes obtaining original road data, preprocessing the original road data, and generating a set of images to be processed including multiple images to be processed. The original road data is data collected on the road, which can be either video or image.

[0030] For example, a fixed image or video acquisition device can be set up to collect road data as original road data, or original road video can be obtained through a vehicle. In the actual implementation process, original road videos of different road sections can also be obtained through different methods, such as using road monitoring and other methods to collect original road videos.

[0031] When the original road acquisition data is an original road acquisition video, image de-framing is performed according to the original road acquisition video to obtain an initial image sequence set of the original road acquisition video, and the images in the initial image sequence set are arranged in the order of image frames.

[0032] Since the original road video collected may include video content such as vehicles, people or empty roads, but the images to be annotated must include license plate information and facial information, the video segments in the original road video that do not contain vehicles or people are invalid video segments. If the invalid video segments are decomposed into image frames for processing, a large amount of processing resources will be wasted, reducing the efficiency of generating images to be annotated.

[0033] In a feasible implementation, taking the original road sampling data as an original road sampling video as an example, the process of generating a set of images to be processed including a plurality of images to be processed may include the following steps: A1: Get the original road video.

[0034] For example, multiple vehicles may be selected as designated target vehicles for obtaining the original road video. The designated multiple vehicles may be distributed in different geographical areas, so as to facilitate obtaining richer license plate information or person information.

[0035] A2: Extract multiple target videos from the original road video, where the target videos include at least one vehicle or one person.

[0036] For example, the original road video is analyzed based on the existing target detection algorithm to identify and extract the target video including any vehicle or any person in the original road video, thereby filtering out invalid video segments.

[0037] During actual implementation, a suitable target detection model may be selected according to the needs of the actual application to analyze and identify vehicles and people in the original road shooting video, and this embodiment does not impose any restrictions.

[0038] A3: Generate an initial image sequence set based on the multiple target videos.

[0039] After extracting multiple target videos containing vehicles or people from the original road video, each target video is deframed to obtain the image sequence of the target video, and then the image sequences of multiple target videos are spliced to obtain the initial image sequence set.

[0040] A4: Preprocessing the initial image sequence set to generate a to-be-processed image set including a plurality of to-be-processed images.

[0041] In order to further improve the efficiency of data processing, the initial image sequence set can be preprocessed. For example, the preprocessing process includes: removing damaged images, unifying the format and resolution, etc. In the actual implementation process, other preprocessing processes can also be selected according to the needs of the actual application.

[0042] For example, the process of preprocessing the initial image sequence set to generate a to-be-processed image set including a plurality of to-be-processed images may include the following steps: B1: In the initial image sequence set, each image is classified, and multiple images in the same classification are acquired by the same image acquisition device.

[0043] Specifically, different image acquisition devices obtain different video or image data. If the original road video is obtained through the target vehicle, the on-board camera on the vehicle is used as the image acquisition device. Since the vehicle is usually equipped with multiple cameras, the video data collected by each camera is also different, such as different resolutions. In order to batch process the images corresponding to the video data collected by the same camera, the images are classified according to the on-board camera.

[0044] For example, a vehicle may include 11 on-board cameras, such as cameraFrontNarrow (front narrow angle), cameraFrontWide (front wide angle), cameraRear (rear wide angle), cameraSideLeftFront (left front view), cameraSideLeftRear (left rear view), cameraSideRightFront (right front view), cameraSideRightRear (right rear view), cameraSurroundFront (front surround view), cameraSurroundLeft (left surround view), cameraSurroundRear (rear surround view) / cameraSurroundRight (right surround view).

[0045] A unique ID can be set for each camera, so the name of the video captured by any vehicle-mounted camera carries the ID of the camera, and a naming rule can be preset for deframing multiple images from the video. For example, the naming rule can be set to: camera ID_timestamp_frame number.extension, so that each image has a different name.

[0046] After acquiring original road-collected videos through multiple cameras and obtaining an initial image sequence set, the images in the initial image sequence set can be reordered according to the image names, and the multiple images acquired through the same camera can be classified.

[0047] B2: Eliminate damaged images from the initial image sequence set.

[0048] Detect whether there are damaged images, including images that cannot be displayed or opened. If there are damaged images, discard the images.

[0049] B3: changing the formats of all images in the initial image sequence set to the target image format.

[0050] To facilitate subsequent processing, the formats of all images can be unified. The target image format can be set according to the needs of actual applications. In this embodiment, the target image format is defined as the JPG format. For example, the image in the JPEG format is converted to the JPG format. Although JPG and JPEG can be used interchangeably in most cases, unifying the image format helps to ensure the format consistency of the obtained image set to be processed and avoid compatibility issues caused by different image formats.

[0051] B4: Modify the resolutions of all images in the initial image sequence set to the target resolution.

[0052] The target resolution can be selected as the smallest resolution in the initial image sequence set, and can also be set according to the needs of the actual application during the actual implementation process.

[0053] For example, among the 11 cameras of a vehicle, the front wide-angle and front narrow-angle cameras generally have higher image resolutions than other cameras, such as 3840×2160. The resolutions of other cameras are generally 1920×1080, so the target resolution can be set to 1920×1080.

[0054] By classifying the images according to the cameras in step B1, the resolution of multiple images corresponding to the same camera can be modified in batches, such as adjusting the resolution of multiple images corresponding to the vehicle's front wide-angle and front narrow-angle cameras to 1920×1080 in batches.

[0055] By reducing the resolution of the image, the file size of each image can be reduced, thereby reducing the storage space required for the entire set of images to be processed, which can effectively reduce costs and is especially important for scenarios with large-scale images; and reducing the resolution means that each image contains fewer pixels, which makes the subsequent processing of this method faster, and when the desensitization model is trained based on the images to be annotated, the training speed of the desensitization module can also be accelerated.

[0056] When the original road shooting data is the original road shooting image, the process of preprocessing to obtain the set of images to be processed is similar to the preprocessing process of the original road shooting video. The target detection algorithm can be used to filter out images that do not contain any vehicles or any people, and then the damaged images can be eliminated and the image format and resolution can be modified to obtain the set of images to be processed.

[0057] After obtaining the set of images to be processed, the method further includes extracting feature vectors of each image to be processed.

[0058] In a feasible implementation, when extracting the feature vectors of each image to be processed, each image to be processed in the set of images to be processed can be input into the MobileNetV2-Reid neural network to obtain the feature vectors corresponding to each image to be processed output by the feature extraction layer of the MobileNetV2-Reid neural network.

[0059] Reference Figure 2 , shows a structural schematic diagram of the MobileNetV2-Reid neural network provided in an embodiment of the present application. The MobileNetV2-Reid neural network describes the features of the image based on a deep learning convolutional neural network. The specific processing process includes: after the input image data is processed by the backbone network, a three-dimensional feature map of C (number of channels) × H (height) × W (width) is obtained, and then the feature map is average pooled (Average Pooling) through the feature extraction layer to obtain the feature vector (one-dimensional vector) of the image data. Finally, after the feature vector passes through the fully connected layer and the deep learning classifier, the classification result of the image data can be obtained; the MobileNetV2-Reid neural network usually uses the cross entropy loss function commonly used in classification tasks for network training.

[0060] When training the MobileNetV2-Reid neural network, the training dataset used can be any image retrieval dataset, and the training labels use the target category labels in the dataset. In this embodiment, the pedestrian re-identification (Person ReID) dataset Market-1501 is used, the uniform input size of the images used for training is 384×128, and random flipping and random erasing are used as image enhancement methods. Then, the stochastic gradient descent algorithm with a momentum parameter of 0.9 is used for model optimization, and the batch size is set to 32. There are 25 training cycles in total. In the first 5 cycles, the warm-up strategy is used to gradually change the learning rate from 0.001 to 0.1, and in the remaining 20 cycles, the cosine learning rate decay strategy is used to gradually change the learning rate from 0.1 to 0.

[0061] During the training of the MobileNetV2-Reid neural network, for the current batch of images participating in the training, for a picture , i ∈{1,2,64}, l is the pedestrian ID corresponding to the picture, picture The corresponding feature description vector is , in the eigenvector Then add a fully connected layer, the output of which is , where I is the number of pedestrian identity categories in the training dataset, then the pedestrian identity (ID) prediction objective function satisfies:

[0062] in, , set up For the real pedestrian identity of this image data, , and when When satisfied .

[0063] In this embodiment, after each image to be processed in the set of images to be processed is input into the trained MobileNetV2-Reid neural network, the feature vector corresponding to each image to be processed output by the feature extraction layer of the MobileNetV2-Reid neural network is obtained.

[0064] In the actual implementation process, the frame rate of the original road video is relatively high. For example, taking the original road video collected by the target vehicle as an example, most of them are continuous frame images with a frame rate greater than 10FPS, and the content similarity between consecutive frames is very high. If all images are used as images to be labeled, it will not only increase the cost of manual labeling, but also use images with similar content to train the desensitization model, which will not help improve the model's capabilities.

[0065] Therefore, this embodiment performs de-redundancy on similar images of the images to be processed. Specifically, the images to be processed in the image set to be processed are filtered according to the feature vectors of each image to be processed, so that the similarity between any two images in the generated candidate image set is less than the similarity threshold, that is, the contents of the images in the candidate image set are relatively different.

[0066] In a feasible implementation, the process of filtering the images to be processed in the image set to be processed according to the feature vectors of the images to be processed to generate the candidate image set may include the following steps: C1: taking the first image in the set of images to be processed as a reference image, and moving the reference image into the set of candidate images.

[0067] C2: Calculating the similarity between the image to be processed and the reference image based on the feature vectors of the image to be processed and the feature vectors of the reference image in accordance with the arrangement order of the images to be processed in the set of images to be processed.

[0068] C2: when the similarity between any image to be processed and the reference image is greater than a similarity threshold, the image to be processed is removed from the set of images to be processed; C3: When the similarity between any image to be processed and the reference image is less than or equal to a similarity threshold, the image to be processed is updated as the current reference image; C4: until the set of images to be processed is empty, the candidate image set is obtained.

[0069] Reference Figure 3 , shows an execution flow chart of the similar image filtering process provided by an embodiment of the present application. The process of filtering the images to be processed in the image set to be processed based on the feature vectors of each image to be processed and generating a candidate image set includes: S1: The first image in the image set to be processed is used as the reference image; S2: Move the reference image into the candidate image set; S3: Determine whether the next image is included; If the next image is included, execute step S4; If the next image is not included, execute step S8: S4: Calculate similarity based on the feature vector of the image and the feature vector of the reference image; S5: Determine whether the similarity is greater than a similarity threshold; If the similarity is greater than the similarity threshold, execute step S6; If the similarity is less than or equal to the similarity threshold, execute step S7: S6: Eliminate the image to be processed from the set of images to be processed; Continue to step S3; S7: updating the image to be processed as the current reference image; Continue to step S2; S8: Output a set of candidate images.

[0070] For example, the images to be processed in the set of images to be processed may be numbered, and the image numbers of the set of images to be processed are: image 1, image 2, . ...

[0071] First, the first image in the image set to be processed, that is, image 1, is used as the reference image, and image 1 is first moved into the candidate image set.

[0072] Then, according to the arrangement order of the images to be processed, the similarity 1 between image 2 and image 1 is calculated based on the feature vector of image 2 and the feature vector of image 1. Assuming that the similarity 1 is greater than the similarity threshold, it indicates that the content of image 2 and image 1 is similar, and image 2 is removed from the image set to be processed.

[0073] Assume that the similarity between image 3 and image 1 is 2, and the similarity between image 4 and image 1 is 3, and both similarity 2 and similarity 3 are greater than the similarity threshold, then image 3 and image 4 are removed from the set of images to be processed.

[0074] When the calculated similarity 4 between image 5 and image 1 is less than or equal to the similarity threshold, it is indicated that the contents of image 5 and image 1 are not similar, and image 5 is updated as the current reference image and moved into the candidate image set.

[0075] Continue to calculate the similarity between image 6 and image 5, and so on, until the set of images to be processed is empty.

[0076] Therefore, dissimilar images in the image set to be processed can be extracted as candidate images, reducing the redundancy of similar images.

[0077] In this embodiment, when calculating the similarity between images, the cosine similarity between the image to be processed and the reference image is calculated based on the feature vector of the image to be processed and the feature vector of the reference image, and the cosine similarity is used as the similarity.

[0078] Specifically, the similarity calculation formula is: Eu(feat1,feat2)=feat1·feat2 / |feat1||feat2| Among them, feat1 is the feature vector of one image; feat2 is the feature vector of another image; Eu(feat1, feat2) is the similarity between the two images.

[0079] In actual implementation, other methods, such as Euclidean distance, may be used to calculate the similarity between the image to be processed and the reference image based on the feature vector of the image to be processed and the feature vector of the reference image.

[0080] S102: Input the images in the candidate image set into a license plate and face mixed screening model, and output a set of images to be labeled, where the images to be labeled in the set include face images and / or license plate images.

[0081] The license plate and face mixed screening model is a model obtained by using deep reinforcement learning methods, based on a public training set and trained with optimization of detection accuracy as the indicator. The training set of the license plate and face mixed screening model includes at least images of license plate and face data, fisheye distortion training images, and images of different scenes, such as daytime, nighttime, sunny days, rainy days, and basements.

[0082] In a feasible implementation, a YOLOv5 (You Only Look Once V5; a target detection algorithm detection model) detection model can be fine-tuned to obtain a trained license plate and face hybrid screening model.

[0083] Specifically, the original YOLOv5s detection model was trained on the road license plate dataset of the application area. Then, based on the original YOLOv5s detection model, the YOLOv5 detection model was iteratively trained for a target number of times using the face detection benchmark dataset and the road license plate dataset to obtain a fine-tuned YOLOv5 detection model as a license plate and face hybrid screening model.

[0084] In this embodiment, the target number of iterative training for the YOLOv5 detection model is 50 epochs.

[0085] After the images in the candidate image set are input into the license plate and face mixed screening model, the license plate and face mixed screening model will identify images containing license plate images and / or face images, and use all the output images as the set of images to be labeled. The obtained set of images to be labeled can be used when training a desensitizing model for detecting and identifying license plate information and face information.

[0086] In order to solve the problem of slow efficiency in acquiring training data containing license plate and facial information during the training of deep learning models for desensitizing license plate and facial information, this method can obtain an initial image sequence set based on massive original road-collected videos. Then, after preprocessing including removing damaged images, unifying the format and resolution, the feature vector of the image is quickly extracted through the MobileNetV2-ReID neural network, and the similarity between the two images is calculated based on the feature vector. Images with similar content are de-redundant based on the similarity, which not only improves the efficiency of data processing, but also enhances the accuracy and robustness of recognition, providing high-quality training data for subsequent desensitization model training applications. Finally, the license plate and face mixed screening model can quickly select images containing license plate images and / or facial images from the de-redundant images of similar images as images to be labeled, and eliminate invalid images, thereby greatly improving the efficiency of the acquisition or screening process of images to be labeled for training the desensitization model.

[0087] In actual application, using one RTX A4500 GPU, this method can process 100,000 images in one hour (approximately 3 hours of video data collected by a single camera at 10FPS), significantly improving the processing efficiency of the images to be labeled.

[0088] Reference Figure 4 , shows a functional module diagram of a data processing device provided in an embodiment of the present application, the device comprising: An image filtering module 100 is configured to filter the images to be processed in the image set to be processed based on the feature vectors of the images to be processed, and generate a candidate image set, wherein the similarity between any two images in the candidate image set is less than a similarity threshold; The data output module 200 is used to input the images in the candidate image set into the license plate and face mixed screening model, and output a set of images to be labeled, where the images to be labeled in the set include face images and / or license plate images.

[0089] Optionally, the device further comprises: Acquisition module, used to obtain original road sampling data; The preprocessing module is used to preprocess the original road sampling data to generate a set of images to be processed including multiple images to be processed.

[0090] Optionally, the preprocessing module includes: An extraction unit is used to extract multiple target videos from the original road video, wherein the target videos include at least one vehicle or one person; An initial image sequence set generating unit, configured to generate an initial image sequence set based on the plurality of target videos; The preprocessing unit is used to preprocess the initial image sequence set to generate a to-be-processed image set including a plurality of to-be-processed images.

[0091] Optionally, the pre-processing unit includes: a classification unit, configured to classify each image in the initial image sequence set according to its name, wherein multiple images in the same classification are acquired by the same image acquisition device; a culling unit, configured to cull damaged images from the initial image sequence set; a format modification unit, configured to change the formats of all images in the initial image sequence set to a target image format; The resolution modification unit is configured to modify the resolutions of all images in the initial image sequence set to a target resolution.

[0092] Optionally, the device further comprises: The feature extraction module is used to input each image to be processed in the set of images to be processed into the MobileNetV2-Reid neural network, and obtain the feature vector corresponding to each image to be processed output by the feature extraction layer of the MobileNetV2-Reid neural network.

[0093] Optionally, the image filtering module includes: a reference image determining unit, configured to use the first image in the set of images to be processed as a reference image, and move the reference image into the set of candidate images; a similarity calculation unit, configured to calculate the similarity between the image to be processed and the reference image according to the arrangement order of the images to be processed in the set of images to be processed and the feature vectors of the reference image; an image elimination unit, configured to eliminate any image to be processed from the set of images to be processed when the similarity between the image to be processed and the reference image is greater than a similarity threshold; A reference image updating unit, configured to update any image to be processed to the current reference image when the similarity between the image to be processed and the reference image is less than or equal to a similarity threshold; The candidate image set determining unit is configured to obtain the candidate image set until the to-be-processed image set is empty.

[0094] Optionally, the similarity calculation unit includes: The similarity calculation subunit is configured to calculate the cosine similarity between the image to be processed and the reference image based on the feature vector of the image to be processed and the feature vector of the reference image, and use the cosine similarity as the similarity.

[0095] Optionally, the device further comprises: The license plate and face mixed screening model determination module is used to iteratively train the YOLOv5 detection model for a target number of times based on the face detection benchmark dataset and the road license plate dataset, and obtain a fine-tuned YOLOv5 detection model as the license plate and face mixed screening model.

[0096] Reference Figure 5 , shows a schematic diagram of an electronic device provided in an embodiment of the present application, comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein the processor executes the data processing method described in the embodiment when executing the computer program.

[0097] Reference Figure 6, shows a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application, wherein the non-volatile readable storage medium stores a computer program, wherein the computer program executes the data processing method described in the embodiment when executed by a processor.

[0098] An embodiment of the present application also provides a terminal, which includes a data processing device described in the embodiment. For example, the terminal is a vehicle or a server. Specifically, after the vehicle collects the original road video, the processing process of the image to be labeled can be completed on the vehicle side, and then the image to be labeled can be uploaded to the upper-level server side; or after the vehicle collects the original road video, the original road video can be uploaded to the server, and the server executes this method to obtain the image to be labeled.

[0099] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0100] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, apparatuses, or computer program products. Therefore, the embodiments of the present application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. Furthermore, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1The function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0104] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0105] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0106] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A data processing method, characterized in that: The method comprises: Filtering the images to be processed in the image set to be processed according to the feature vectors of each image to be processed in the image set to generate a candidate image set, wherein the similarity between any two images in the candidate image set is less than a similarity threshold; The images in the candidate image set are input into a license plate and face mixed screening model, and a set of images to be labeled is output, where the images to be labeled in the set of images to be labeled include face images and / or license plate images.

2. The method according to claim 1, characterized in that The method further comprises: Obtain original road sampling data; The original road sampling data is preprocessed to generate a to-be-processed image set including a plurality of to-be-processed images.

3. The method according to claim 2, characterized in that The original road sampling data includes an original road sampling video, and the original road sampling data is preprocessed to generate a to-be-processed image set including a plurality of to-be-processed images, including: Extracting multiple target videos from the original road video, wherein the target videos include at least one vehicle or one person; Generating an initial image sequence set according to the multiple target videos; The initial image sequence set is preprocessed to generate a to-be-processed image set including a plurality of to-be-processed images.

4. The method according to claim 3, characterized in that Preprocessing the initial image sequence set to generate a to-be-processed image set comprising a plurality of to-be-processed images includes: In the initial image sequence set, each image is classified according to its name, and multiple images in the same classification are obtained by the same image acquisition device; Eliminating damaged images from the initial image sequence set; Changing the formats of all images in the initial image sequence set to a target image format; The resolutions of all images in the initial image sequence set are modified to a target resolution.

5. The method according to any one of claims 1 to 4, characterized in that Before generating a candidate image set, filtering the images to be processed in the image set to be processed based on the feature vectors of each image to be processed in the image set to be processed, the method further includes: Input each image to be processed in the set of images to be processed into the MobileNetV2-Reid neural network, and obtain the feature vector corresponding to each image to be processed output by the feature extraction layer of the MobileNetV2-Reid neural network.

6. The method according to any one of claims 1 to 4, characterized in that Filtering the images to be processed in the image set to be processed according to the feature vectors of each image to be processed in the image set to be processed to generate a candidate image set, including: Using the first image in the set of images to be processed as a reference image, and moving the reference image into the set of candidate images; Calculating the similarity between the image to be processed and the reference image based on the feature vectors of the image to be processed and the feature vectors of the reference image in the order of arrangement of the image to be processed in the image set to be processed; When the similarity between any image to be processed and the reference image is greater than a similarity threshold, the image to be processed is removed from the set of images to be processed; When the similarity between any image to be processed and the reference image is less than or equal to a similarity threshold, the image is updated as the current reference image; Until the set of images to be processed is empty, the candidate image set is obtained.

7. The method according to claim 6, characterized in that Calculating the similarity between the image to be processed and the reference image according to the feature vector of the image to be processed and the feature vector of the reference image, including: The cosine similarity between the image to be processed and the reference image is calculated according to the feature vector of the image to be processed and the feature vector of the reference image, and the cosine similarity is used as the similarity.

8. The method according to any one of claims 1, 2, 3, 4 and 7, characterized in that The method further comprises: The YOLOv5 detection model is iteratively trained for a target number of times based on the face detection benchmark dataset and the road license plate dataset to obtain a fine-tuned YOLOv5 detection model as a license plate face hybrid screening model.

9. A data processing device, characterized in that: The device comprises: an image filtering module, configured to filter the images to be processed in the image set to be processed according to the feature vectors of the images to be processed, and generate a candidate image set, wherein the similarity between any two images in the candidate image set is less than a similarity threshold; The data output module is used to input the images in the candidate image set into the license plate and face mixed screening model, and output a set of images to be labeled, wherein the images to be labeled in the set of images to be labeled include face images and / or license plate images.

10. An electronic device, characterized in that: include: At least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein the processor executes the data processing method according to any one of claims 1 to 8 when executing the computer program.

11. A non-volatile readable storage medium, characterized in that: The non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 8 is executed.

12. A terminal, characterized in that: The terminal includes a data processing device according to claim 9, or is used to execute the data processing method according to any one of claims 1-8.

13. The terminal according to claim 12, characterized in that The terminal is a vehicle or a server.

Images