Sample image acquisition method, model training method and device
By detecting and blurring the image in the deep learning data amplification method, combined with label smoothing processing, the problem of ignoring image chirality in data amplification is solved, and the performance of the perception algorithm is significantly improved.
Patent Information
- Application Number
- CN202311354928.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-18
- Publication Date
- 2025-04-18
AI Technical Summary
The existing deep learning data amplification method ignores the chirality of the image when processing images, resulting in the amplified data that violates the distribution of the actual data and affects the performance of the perception algorithm.
By detecting the target object on the image to be processed, blurring or adding labeling information is performed after confirming the presence of chiral target object, the label is smoothed in combination with the image amplification method to reduce the influence of chirality.
It effectively weakens the chiral effect in the training data, makes the generated amplified data more in line with the actual distribution, thereby improving the performance of the perception algorithm and training smaller models when hardware resources are limited.
Smart Images
Figure CN119851323A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a sample image acquisition method, a model training method and a device. Background Art
[0002] Image-based deep learning requires a large amount of training data. In order to improve the perception effect as much as possible and reduce the dependence on data, the perception algorithm usually adopts various algorithmic means, among which the most commonly used method is data augmentation. There are many different methods for data augmentation. For example, the commonly used methods are: image flipping, image rotation, image scaling, etc. However, these methods all ignore an important issue, which is the chirality of the image. The more commonly used deep learning data augmentation methods currently destroy the chirality of the image to varying degrees, and even some scenes that do not exist in the real world appear after the amplification process. These data obviously violate the distribution of real data. Summary of the invention
[0003] In order to overcome the problems existing in the related art, the present disclosure provides a sample image acquisition method, a model training method and a device.
[0004] According to a first aspect of an embodiment of the present disclosure, a sample image acquisition method is provided, comprising:
[0005] Acquire a sample to be processed, where the sample to be processed includes a first image and an image label;
[0006] Detecting a first target object on the first image to obtain a detection result;
[0007] When it is confirmed based on the detection result that the first target object exists, blurring the first target object in the first image to obtain a processed first image, or adding annotation information to the first image; wherein the annotation information is used to indicate whether there is a chiral target object in the image;
[0008] Performing image amplification processing on the processed first image by a first image amplification method to obtain a sample image;
[0009] Based on the annotation information and the destructiveness of the first image augmentation method to chirality, the image label is smoothed to obtain a sample label.
[0010] In some implementations, the detecting a first target object on the first image to obtain a detection result includes:
[0011] Performing face detection on the first image to obtain a face detection result;
[0012] When it is confirmed based on the detection result that the first target object exists, blurring the first target object in the first image to obtain a processed first image, or adding annotation information to the first image, includes:
[0013] When it is confirmed that there are facial features based on the face detection result, blurring is performed on the facial features to obtain a processed first image.
[0014] In some implementations, the detecting a first target object on the first image to obtain a detection result includes:
[0015] Performing obstacle detection on the sample image to be processed to obtain an obstacle detection result;
[0016] When it is confirmed based on the detection result that the first target object exists, blurring the first target object in the first image to obtain a processed first image, or adding annotation information to the first image, includes:
[0017] When it is confirmed that there is an obstacle based on the obstacle detection result, annotation information is added to the first image.
[0018] In some implementations, the detecting a first target object on the first image to obtain a detection result includes:
[0019] Performing optical character recognition (OCR) detection on the sample image to be processed to obtain a character detection result;
[0020] When it is confirmed based on the detection result that the first target object exists, blurring the first target object in the first image to obtain a processed first image, or adding annotation information to the first image, includes:
[0021] Based on the character detection result, when it is confirmed that there are characters, identifying the characters in the character detection result to obtain at least one character set;
[0022] Classifying each character set in the at least one character set to obtain classification results of each character set; the classification includes autonomous driving related types and non-autonomous driving types;
[0023] When the character set belongs to a non-autonomous driving type, the character set in the first image is blurred to obtain a processed first image; and when the character set belongs to an autonomous driving related type, annotation information is added to the first image.
[0024] In some implementations, classifying each character set in the at least one character set includes:
[0025] Matching each character set in the at least one character set with a character library;
[0026] If the character set successfully matches the character library, then the character set belongs to a type related to autonomous driving;
[0027] If the character set fails to match the character library, the character set belongs to a non-autonomous driving type.
[0028] In some implementations, the step of smoothing the image label based on the annotation information and the destructiveness of the first image augmentation method to chirality includes:
[0029] When the annotation information indicates that a target object with chirality exists in the image, determining whether the first image augmentation method is destructive to the chirality;
[0030] When the first image augmentation method is destructive to chirality, the image label is smoothed.
[0031] According to a second aspect of an embodiment of the present disclosure, a method for training a perceptual deep learning model is provided, the method comprising:
[0032] Acquire a plurality of sample data, wherein the sample data includes sample images and sample labels, and the sample images and sample labels are obtained by the method according to the first aspect;
[0033] The perceptual deep learning model is trained by using the plurality of sample data to obtain a trained perceptual deep learning model. According to a third aspect of an embodiment of the present disclosure, a sample image acquisition device is provided, comprising:
[0034] A data acquisition module, used to acquire a sample to be processed, wherein the sample to be processed includes a first image and an image label;
[0035] An object detection module, configured to detect a first target object in the first image and obtain a detection result;
[0036] a data processing module, configured to, when confirming the presence of the first target object based on the detection result, perform blurring processing on the first target object in the first image to obtain a processed first image, or add annotation information to the first image; wherein the annotation information is used to indicate whether there is a chiral target object in the image;
[0037] A data augmentation module, configured to perform image augmentation processing on the processed first image by a first image augmentation method to obtain a sample image;
[0038] The label processing module is used to perform smoothing processing on the image label based on the annotation information and the destructiveness of the first image augmentation method to chirality to obtain a sample label.
[0039] In some implementations, the object detection module includes a face detection unit configured to:
[0040] Performing face detection on the first image to obtain a face detection result;
[0041] The data processing module includes a facial feature processing unit, which is used to:
[0042] When it is confirmed that there are facial features based on the face detection result, blurring is performed on the facial features to obtain a processed first image.
[0043] In some implementations, the object detection module includes an obstacle detection unit configured to:
[0044] Performing obstacle detection on the sample image to be processed to obtain an obstacle detection result;
[0045] The data processing module includes an obstacle processing unit for:
[0046] When it is confirmed that there is an obstacle based on the obstacle detection result, annotation information is added to the first image.
[0047] In some implementations, the object detection module includes a character detection unit configured to:
[0048] Performing optical character recognition (OCR) detection on the sample image to be processed to obtain a character detection result;
[0049] The data processing module includes a character processing unit for:
[0050] Based on the character detection result, when it is confirmed that there are characters, identifying the characters in the character detection result to obtain at least one character set;
[0051] Classifying each character set in the at least one character set to obtain classification results of each character set; the classification includes autonomous driving related types and non-autonomous driving types;
[0052] When the character set belongs to a non-autonomous driving type, the character set in the first image is blurred to obtain a processed first image; and when the character set belongs to an autonomous driving related type, annotation information is added to the first image.
[0053] In some implementations, the character processing unit, when classifying each character set in the at least one character set, is configured to:
[0054] Matching each character set in the at least one character set with a character library;
[0055] If the character set successfully matches the character library, then the character set belongs to a type related to autonomous driving;
[0056] If the character set fails to match the character library, the character set belongs to a non-autonomous driving type.
[0057] In some implementations, the tag processing module is specifically configured to:
[0058] When the annotation information indicates that a target object with chirality exists in the image, determining whether the first image augmentation method is destructive to the chirality;
[0059] When the first image augmentation method is destructive to chirality, the image label is smoothed.
[0060] According to a fourth aspect of an embodiment of the present disclosure, a training device for a perceptual deep learning model is provided, comprising:
[0061] A sample acquisition module, used to acquire a plurality of sample data, wherein the sample data includes a sample image and a sample label, wherein the sample image and the sample label are obtained by the method according to the first aspect;
[0062] The model training module is used to train the perceptual deep learning model through the multiple sample data to obtain the trained perceptual deep learning model.
[0063] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:
[0064] at least one processor; and
[0065] a memory communicatively connected to the at least one processor; wherein,
[0066] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any one of the first aspects, or to enable the at least one processor to execute the method described in the second aspect.
[0067] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, comprising:
[0068] The storage medium stores a computer program, and when the computer program is executed by the processor, the method described in any one of the first aspects is implemented, or when the computer program is executed by the processor, the method described in any one of the second aspects is implemented.
[0069] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
[0070] Obvious chiral features in the training data are removed through fuzzification, and the confidence is reduced through label smoothing, thereby reducing the impact of chirality in the training data, so that the generated amplified data is more consistent with the actual distribution, which can significantly improve the performance of the perception algorithm. At the same time, when hardware resources are limited, smaller models can be trained. In addition, targets with obvious chirality in the training data usually contain some sensitive information, and fuzzification can better achieve data desensitization.
[0071] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0073] Figure 1 The figure is a flow chart of a method for acquiring a sample image according to an exemplary embodiment.
[0074] Figure 2 is a flowchart of a method for acquiring a sample image according to another exemplary embodiment.
[0075] Figure 3 is a flowchart of a method for acquiring a sample image according to yet another exemplary embodiment.
[0076] Figure 4 is a flowchart of a method for acquiring a sample image according to yet another exemplary embodiment.
[0077] Figure 5 is a flowchart of a method for acquiring a sample image according to yet another exemplary embodiment.
[0078] Figure 6 is a flowchart of a method for training a perceptual deep learning model according to yet another exemplary embodiment.
[0079] Figure 7 is a block diagram of a sample image acquisition device according to an exemplary embodiment.
[0080] Figure 8 It is a block diagram of a training device for a perceptual deep learning model according to yet another exemplary embodiment.
[0081] Fig. 9 It is a block diagram of a device according to an exemplary embodiment. DETAILED DESCRIPTION
[0082] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.
[0083] With the development of deep learning, deep learning technology has penetrated into various algorithm modules in the field of autonomous driving. Deep learning algorithms play an important role in autonomous driving perception tasks. Almost all perception algorithms use deep learning algorithms. Currently, visual perception is an important perception module of vehicles, responsible for multiple perception tasks, such as obstacle detection, lane line detection, pedestrian recognition, etc.
[0084] Image-based deep learning requires a large amount of training data. In order to maximize the perception effect and reduce the dependence on data, the perception algorithm module usually adopts various algorithmic means, among which the most commonly used method is data augmentation. There are many different data augmentation methods at present, and with the development of deep learning, new augmentation methods are proposed every year. For example, the commonly used image augmentation methods include image flipping, image rotation, image scaling, etc. However, these image augmentation methods all ignore an important issue, which is the chirality of the image.
[0085] Chirality usually refers to the fact that an object cannot overlap with its mirror image, and mainly refers to the asymmetry of an object. For example, the left and right hands of a person are a more obvious example. Similar phenomena also exist in images. For example, the text in an image is no longer repeated after being flipped. It is more difficult for people to read the text in the real world when it becomes a mirror image. Traffic signs on the road have clear directions. In the image of a pedestrian pushing a bicycle, the pedestrian is on the left side of the bicycle.
[0086] Chirality is common in the real world, and is more obvious in autonomous driving scenarios, which are also more complex. Currently, the more commonly used data augmentation methods for deep learning have destroyed the chirality of images to varying degrees. Some images even show some scenes that do not exist in the real world after image augmentation. These image data clearly violate the distribution of real data. After mass production of vehicles on the road, such scenes will not be encountered. This operation obviously violates the assumption of independent and identically distributed data in deep learning.
[0087] In addition, autonomous driving algorithms often face hardware-constrained scenarios. For example, the generally recognized autonomous driving levels L2 / L3 / L4 require corresponding hardware resources in addition to hardware sensors. In this context, the detection model needs to occupy less resources, which puts higher requirements on the detection model. Under such hardware conditions, deep learning also needs to cover data distribution that does not exist, making the algorithm task face greater challenges.
[0088] In order to solve the above problems, the present disclosure provides a sample image acquisition method, a model training method and a device, which remove or reduce the influence of the chirality of the training data when performing data augmentation, and can significantly improve the performance of the perception algorithm obtained by training the processed training data. This method has obvious advantages in scenarios such as limited hardware resources and small models.
[0089] Figure 1 is a flow chart of a sample image acquisition method according to an exemplary embodiment. Figure 1 As shown, the sample image acquisition method is used in electronic devices such as terminals and may include the following steps.
[0090] In step S101, a sample to be processed is obtained, where the sample to be processed includes a first image and an image label.
[0091] As an implementation manner, a first image is acquired through a data acquisition system, and an image label is added to the first image manually or in other possible implementation manners.
[0092] In step S102, a first target object is detected on the first image to obtain a detection result.
[0093] It can be understood that there may be multiple types of chiral targets in the image, such as faces, characters, and obstacles with chirality, etc. The types of chiral targets that may exist in the image can be listed in advance so as to perform corresponding detection for each type of target. For example, after determining to detect targets of the face type in the image, the first image is subjected to corresponding face detection to obtain a face detection result, so as to confirm whether there is a target of the face type in the first image according to the face detection result.
[0094] There may be multiple types of chiral targets in one image at the same time, for example, a face, a character, and a chiral obstacle may exist in the first image at the same time. Therefore, they can be detected one by one so that corresponding processing can be performed later according to the detection results. There is no restriction on the detection order of multiple targets, for example, face detection is performed first, obstacle detection is performed later, and character detection is performed last.
[0095] In step S103, based on the detection result, when it is confirmed that the first target object exists, the first target object in the first image is blurred to obtain a processed first image, or annotation information is added to the first image; wherein the annotation information is used to indicate whether there is a chiral target object in the image.
[0096] It should be noted that for targets with obvious chirality in the image, blurring or adding annotation information to the first image can be selected according to the type of the target. Blurring is to weaken or remove features with obvious chirality, such as facial features, so that in the subsequent data augmentation processing, the chiral influence of the chiral features can be reduced and a large amount of training data that does not conform to reality will not be generated. Adding annotation information is also used for downstream tasks, such as data augmentation, so that the original image labels can be processed accordingly when data augmentation is performed.
[0097] It should also be noted that removing features with obvious chirality by processing cannot completely remove the chirality information in the image. For example, the blurring operation may introduce new chirality information, but the high-frequency texture features with obvious characteristics have been removed by blurring, and deep learning is sensitive to high-frequency information. In other words, blurring a target object cannot completely remove the chirality of the target object, but it can reduce the influence of the chirality of the target object.
[0098] As an example, steps S101-S103 can be implemented through a data acquisition system, that is, chirality judgment and corresponding processing after target object recognition are added to the data acquisition system, thereby proposing a new chirality-based data acquisition and processing solution to better organize data.
[0099] In step S104, the processed first image is subjected to image amplification processing by a first image amplification method to obtain a sample image.
[0100] As mentioned above, since image-based deep learning requires a large amount of training data, the perception algorithm module usually uses various algorithmic means to improve the perception effect as much as possible and reduce the dependence on data. The most commonly used method is data augmentation. This embodiment adopts an image augmentation method to obtain a large number of sample images for training the deep learning model.
[0101] In step S105, based on the annotation information and the destructiveness of the first image augmentation method to chirality, the image label is smoothed to obtain a sample label.
[0102] In this embodiment, when the annotation information indicates that there is a chiral target in the image, it is determined whether the first image augmentation method is destructive to the chirality; when the first image augmentation method is destructive to the chirality, the image label is smoothed.
[0103] There are many methods for image augmentation, such as color conversion, image cropping, image flipping, etc. When there is a chiral target in an image, if the image is subjected to color conversion or image cropping, the chiral information of the target in the image will not be changed; if the image is flipped, the chiral information of the target in the image will be changed.
[0104] In order to obtain a large number of sample images, multiple image augmentation methods may be used simultaneously for processing. Depending on whether the image augmentation method destroys the chirality of the sample image, it is decided whether to perform label smoothing.
[0105] When the selected image augmentation method changes the chirality of the object in the image, the original image label of the sample image is smoothed. As an example, in a certain detection scenario, the perceptual deep learning model to be trained is for classification, and the image label of the sample image is the category label. For example, the image label is [0,0,0], which means that the corresponding image belongs to type 1, and the image label is [0,1,0], which means that the corresponding image belongs to type 2. Since the image obtained by the image augmentation method destroys the chirality of the image and does not appear in the real world, the original category label [0,1,0] is smoothed, that is, the confidence is reduced, and the smoothed category label is [0.1,0.6,0.2].
[0106] In addition, the objects with obvious chirality in the image usually contain some sensitive information, such as faces, license plates, etc. After blurring the objects, the data desensitization effect is also achieved. More stringent data processing can be achieved at the data collection end to meet data confidentiality requirements.
[0107] The sample image acquisition method of the disclosed embodiment removes obvious chiral features in the training data through fuzzification processing, reduces the confidence through label smoothing processing, and thus weakens the influence of chirality in the training data, so that the generated amplified data is more consistent with the actual distribution, thereby significantly improving the performance of the perception algorithm and achieving better generalization performance in the mass production stage. At the same time, when hardware resources are limited, smaller models can be trained. In addition, targets with obvious chirality in the training data usually contain some sensitive information, and fuzzification processing can better achieve data desensitization and meet data confidentiality requirements.
[0108] There may be one or more chiral targets in the image, and the processing methods adopted for different types of targets may be different. The following describes steps S102-S103 in detail according to the different targets.
[0109] Figure 2is a flow chart of a sample image acquisition method according to another exemplary embodiment. Figure 2 As shown, the sample image acquisition method may include the following steps.
[0110] In step S201, face detection is performed on the first image to obtain a face detection result.
[0111] Since the human face has obvious chirality in the image, such as the face is not completely symmetrical, the offset direction of the hair, etc. Therefore, it is necessary to process the human face target in the image.
[0112] As an example, after obtaining an image collected by a vehicle-side sensor, the image is input into a face detection model to determine whether there are facial features in the image.
[0113] In step S202, based on the face detection result, when it is confirmed that there are facial features, blurring is performed on the facial features to obtain a processed first image.
[0114] When the face detection result shows that there are face features in the image, face blurring processing is introduced to blur the face features in the image to remove the face features with obvious hand. The implementation method of face blurring processing can be any possible implementation method in the relevant technology, and is not limited here.
[0115] The sample image acquisition method of the disclosed embodiment removes features with obvious chirality in the facial features in the image through fuzzy processing, thereby reducing the influence of chirality in the training data, so that the generated augmented data is more consistent with the actual distribution, thereby significantly improving the performance of the perception algorithm. At the same time, facial information is usually confidential information, and fuzzy processing can better achieve data desensitization.
[0116] Figure 3 is a flow chart of a sample image acquisition method according to another exemplary embodiment. Figure 3 As shown, the sample image acquisition method may include the following steps.
[0117] In step S301, obstacle detection is performed on the sample image to be processed to obtain an obstacle detection result.
[0118] It should also be noted that obstacles are divided into two types, namely static obstacles and dynamic obstacles. For example, ground locks, traffic lights, signboards, etc. are all static obstacles with chirality, and for example, sidecars, people pushing bicycles, etc. are dynamic obstacles with chirality.
[0119] For the detection of static obstacles and dynamic obstacles, any possible method in the relevant technology, such as a target detection method, may be used, and no limitation is made here.
[0120] In step S302, based on the obstacle detection result, when it is confirmed that there is an obstacle, annotation information is added to the first image.
[0121] When an obstacle is detected in the image, annotation information is added to annotate the image as having chirality, thereby smoothing the image label according to the annotation information.
[0122] It should also be noted that static obstacles play an important role in autonomous driving positioning, mapping, and regulation and control. When static obstacles are detected, their information needs to be retained.
[0123] The sample image acquisition method of the disclosed embodiment adds chirality annotation information through obstacle detection, thereby deciding whether to smooth the image label to reduce the confidence level, thereby weakening the influence of chirality in the training data, so that the generated amplified data is more consistent with the actual distribution, thereby significantly improving the performance of the perception algorithm.
[0124] Figure 4 is a flow chart of a sample image acquisition method according to another exemplary embodiment. Figure 4 As shown, the sample image acquisition method may include the following steps.
[0125] In step S401, an OCR (Optical Character Recognition) detection is performed on the sample image to be processed to obtain a character detection result.
[0126] Through OCR detection, it is detected whether there are characters in the sample image to be processed, so as to perform corresponding processing on the characters.
[0127] In step S402, based on the character detection result, when it is confirmed that characters exist, the characters in the character detection result are recognized to obtain at least one character set.
[0128] After determining that there are characters in the sample image to be processed, the characters need to be recognized in OCR so that the characters can be classified according to the character content later.
[0129] Each character set contains at least one character.
[0130] In step S403, each character set in at least one character set is classified to obtain a classification result of each character set; the classification includes an autonomous driving related type and a non-autonomous driving type.
[0131] Different characters need to be processed differently, so the recognized characters need to be classified first. The classification types include autonomous driving related types and non-autonomous driving types. Non-autonomous driving types include advertisements, addresses, license plates, etc. Autonomous driving related types include signboards, etc.
[0132] In one implementation, a method for classifying each character set in at least one character set includes: matching each character set in the at least one character set with a character library; if the character set matches the character library successfully, the character set belongs to an autonomous driving related type; if the character set matches the character library unsuccessfully, the character set belongs to a non-autonomous driving type.
[0133] In step S404, when the character set belongs to a non-autonomous driving type, the character set in the first image is blurred to obtain a processed first image; and when the character set belongs to an autonomous driving related type, annotation information is added to the first image.
[0134] Different types of characters are processed differently. For characters that are not related to autonomous driving, the chirality of the text area is removed through fuzzy processing to avoid destroying the chirality of the image after data amplification, which may lead to some scenes that do not exist in the real world. If it is determined that there are characters related to autonomous driving in the image, annotation information is added to indicate whether there are chiral targets in the image. At the same time, character sets related to autonomous driving also need to be retained. As an example, character set information related to autonomous driving can be retained in the data system for use by downstream tasks.
[0135] The sample image acquisition method of the disclosed embodiment removes the chiral features in the non-autonomous driving type characters in the image through fuzzy processing, and adds annotation information when there are characters related to autonomous driving, so as to decide whether to smooth the image labels and reduce the confidence, thereby weakening the influence of chirality in the training data, so that the generated amplified data is more in line with the actual distribution, thereby significantly improving the performance of the perception algorithm.
[0136] As an example, Figure 5 is a flow chart of a sample image acquisition method according to another exemplary embodiment. Figure 5 As shown, the sample image acquisition method may include the following steps.
[0137] In step S501, a sample to be processed is obtained, where the sample to be processed includes a first image and an image label.
[0138] In step S502, face detection is performed on the first image to obtain a face detection result; and based on the face detection result, when it is confirmed that there are face features, blurring is performed on the face features to obtain a processed first image.
[0139] In step S503, obstacle detection is performed on the sample image to be processed to obtain an obstacle detection result; and based on the obstacle detection result, when it is confirmed that an obstacle exists, annotation information is added to the first image.
[0140] In step S504, optical character recognition (OCR) detection is performed on the sample image to be processed to obtain a character detection result; based on the character detection result, when it is confirmed that characters exist, the characters in the character detection result are recognized to obtain at least one character set; each character set in the at least one character set is classified to obtain a classification result of each character set; the classification includes autonomous driving related types and non-autonomous driving types; when the character set belongs to the non-autonomous driving type, the character set in the first image is blurred to obtain a processed first image; and when the character set belongs to the autonomous driving related type, annotation information is added to the first image.
[0141] It should be noted that, in the embodiments of the present disclosure, step S501, step S502 and step S503 do not distinguish the execution order. Optionally, step S501, step S502 and step S503 may be executed in sequence, or step S501, step S503 and step S502 may be executed in sequence, or step S503, step S501, step S502 may be executed in sequence, and no specific limitation is made here.
[0142] In step S505, the processed first image is subjected to image enlargement processing by using the first image enlargement method to obtain a sample image.
[0143] In step S506, based on the annotation information and the destructiveness of the first image augmentation method to chirality, the image label is smoothed to obtain a sample label.
[0144] In the embodiment of the present disclosure, steps S501-S506 may be implemented in any manner in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.
[0145] The sample image acquisition method of the disclosed embodiment removes obvious chiral features in the training data through fuzzification processing, reduces the confidence through label smoothing processing, and thus weakens the influence of chirality in the training data, so that the generated amplified data is more consistent with the actual distribution, thereby significantly improving the performance of the perception algorithm and achieving better generalization performance in the mass production stage. At the same time, when hardware resources are limited, smaller models can be trained. In addition, targets with obvious chirality in the training data usually contain some sensitive information, and fuzzification processing can better achieve data desensitization and meet data confidentiality requirements.
[0146] Figure 6 is a flowchart of a method for training a perceptual deep learning model according to an exemplary embodiment. Figure 6 As shown, the training method of the perceptual deep learning model is used in electronic devices such as terminals or servers, and may include the following steps.
[0147] In step S601, a plurality of sample data are acquired, the sample data including sample images and sample labels, and the sample images and sample labels are sample images and sample labels obtained by the sample image acquisition method of any of the above embodiments.
[0148] In step S602, a perceptual deep learning model is trained using multiple sample data to obtain a trained perceptual deep learning model.
[0149] The training method of the perception deep learning model of the embodiment of the present disclosure can significantly improve the performance of the perception algorithm by training the perception deep learning model with training data that removes or weakens the influence of chirality information.
[0150] In summary, the embodiments of the present disclosure reduce the chiral information in the image under the existing conditions, reduce the impact of the chiral information, and propose a systematic solution to the chiral problem existing in the image: from data acquisition, data processing, to model training, multiple angles are used to solve the chiral problem of the image involved in the perception algorithm.
[0151] Corresponding to the above-mentioned sample image acquisition method embodiment, Figure 7 is a block diagram of a sample image acquisition device according to an exemplary embodiment. Figure 7 The device includes a data acquisition module 701, an object detection module 702, a data processing module 703, a data amplification module 704 and a label processing module 705.
[0152] Specifically, the data acquisition module 701 is used to acquire a sample to be processed, where the sample to be processed includes a first image and an image label;
[0153] The object detection module 702 is used to detect a first target object in the first image and obtain a detection result;
[0154] The data processing module 703 is used to perform blurring processing on the first target in the first image to obtain a processed first image, or to add annotation information to the first image when confirming the presence of the first target based on the detection result; wherein the annotation information is used to indicate whether there is a chiral target in the image;
[0155] The data augmentation module 704 is used to perform image augmentation processing on the processed first image by a first image augmentation method to obtain a sample image;
[0156] The label processing module 705 is used to perform smoothing processing on the image label based on the annotation information and the destructiveness of the first image augmentation method to chirality, so as to obtain a sample label.
[0157] In some implementations, the object detection module 702 includes a face detection unit 7021, which is used to:
[0158] Performing face detection on the first image to obtain a face detection result;
[0159] The data processing module 703 includes a face feature processing unit 7031, which is used to:
[0160] Based on the face detection result, when it is confirmed that there are facial features, the facial features are blurred to obtain a processed first image.
[0161] In some implementations, the object detection module 702 includes an obstacle detection unit 7022 configured to:
[0162] Perform obstacle detection on the sample image to be processed to obtain an obstacle detection result;
[0163] The data processing module 703 includes an obstacle processing unit 7032, which is used to:
[0164] Based on the obstacle detection result, when it is confirmed that there is an obstacle, annotation information is added to the first image.
[0165] In some implementations, the object detection module 702 includes a character detection unit 7023, which is used to:
[0166] Perform optical character recognition (OCR) detection on the sample image to be processed to obtain a character detection result;
[0167] The data processing module 703 includes a character processing unit 7033, which is used to:
[0168] Based on the character detection result, when it is confirmed that the characters exist, based on the character detection result, when it is confirmed that the characters exist, recognizing the characters in the character detection result to obtain at least one character set;
[0169] Classifying each character set in at least one character set to obtain classification results for each character set; the classification includes autonomous driving related types and non-autonomous driving types;
[0170] When the character set belongs to a non-autonomous driving type, the character set in the first image is blurred to obtain a processed first image; and when the character set belongs to an autonomous driving related type, annotation information is added to the first image.
[0171] In some implementations, the character processing unit 7033, when classifying each character set in at least one character set, is configured to:
[0172] Matching each character set in at least one character set with a character library;
[0173] If the character set matches the character library successfully, then the character set belongs to the type related to autonomous driving;
[0174] If the character set fails to match the character library, the character set belongs to a non-autonomous driving type.
[0175] In some implementations, the tag processing module 705 is specifically configured to:
[0176] When the annotation information indicates that a target object with chirality exists in the image, determining whether the first image augmentation method is destructive to the chirality;
[0177] When the first image augmentation method is destructive to chirality, the image labels are smoothed.
[0178] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0179] The sample image acquisition device of the disclosed embodiment removes obvious chiral features in the training data through fuzzification processing, reduces the confidence through label smoothing processing, and thus weakens the influence of chirality in the training data, so that the generated amplified data is more consistent with the actual distribution, thereby significantly improving the performance of the perception algorithm and achieving better generalization performance in the mass production stage. At the same time, when hardware resources are limited, smaller models can be trained. In addition, targets with obvious chirality in the training data usually contain some sensitive information, and fuzzification processing can better achieve data desensitization and meet data confidentiality requirements.
[0180] Corresponding to the training method embodiment of the above-mentioned perceptual deep learning model, Figure 8 is a block diagram of a training device for a perceptual deep learning model according to an exemplary embodiment. Figure 8 , the device includes: a sample acquisition module 801 and a model training module 802.
[0181] Specifically, the sample acquisition module 801 is used to acquire a plurality of sample data, where the sample data includes a sample image and a sample label, where the sample image and the sample label are the sample images and the sample labels obtained by the method of the first aspect;
[0182] The model training module 802 is used to train the perceptual deep learning model through multiple sample data to obtain a trained perceptual deep learning model.
[0183] The training device for the perceptual deep learning model of the disclosed embodiment can significantly improve the performance of the perceptual algorithm by training the perceptual deep learning model with training data that removes or weakens the influence of chirality information.
[0184] Fig. 9 1 is a block diagram of a device 1800 for implementing a sample image acquisition method or a perceptual deep learning model training method according to an exemplary embodiment. For example, the device 1800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0185] Reference Fig. 9 , the device 1800 may include one or more of the following components: a processing component 1802 , a memory 1804 , a power component 1806 , a multimedia component 1808 , an audio component 1810 , an input / output (I / O) interface 1812 , a sensor component 1814 , and a communication component 1816 .
[0186] The processing component 1802 generally controls the overall operation of the device 1800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1802 may include one or more processors 1820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 1802 may include one or more modules to facilitate interaction between the processing component 1802 and other components. For example, the processing component 1802 may include a multimedia module to facilitate interaction between the multimedia component 1808 and the processing component 1802.
[0187] The memory 1804 is configured to store various types of data to support operations on the device 1800. Examples of such data include instructions for any application or method operating on the device 1800, contact data, phone book data, messages, pictures, videos, etc. The memory 1804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0188] The power component 1806 provides power to the various components of the device 1800. The power component 1806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 1800.
[0189] The multimedia component 1808 includes a screen that provides an output interface between the device 1800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1808 includes a front camera and / or a rear camera. When the device 1800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0190] The audio component 1810 is configured to output and / or input audio signals. For example, the audio component 1810 includes a microphone (MIC), and when the device 1800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1804 or sent via the communication component 1816. In some embodiments, the audio component 1810 also includes a speaker for outputting audio signals.
[0191] I / O interface 1812 provides an interface between processing component 1802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0192] The sensor assembly 1814 includes one or more sensors for providing various aspects of the status assessment of the device 1800. For example, the sensor assembly 1814 can detect the open / closed state of the device 1800, the relative positioning of components, such as the display and keypad of the device 1800, the sensor assembly 1814 can also detect the position change of the device 1800 or a component of the device 1800, the presence or absence of user contact with the device 1800, the orientation or acceleration / deceleration of the device 1800, and the temperature change of the device 1800. The sensor assembly 1814 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1814 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1814 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0193] The communication component 1816 is configured to facilitate wired or wireless communication between the device 1800 and other devices. The device 1800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0194] In an exemplary embodiment, the device 1800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned methods.
[0195] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1804 including instructions, and the instructions can be executed by a processor 1820 of the device 1800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0196] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.
[0197] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A sample image acquisition method, characterized in that: include: Acquire a sample to be processed, where the sample to be processed includes a first image and an image label; Detecting a first target object on the first image to obtain a detection result; Based on the detection result, when it is confirmed that the first target object exists, blurring the first target object in the first image to obtain a processed first image, or adding annotation information to the first image; wherein the annotation information is used to indicate whether there is a chiral target object in the image; Performing image amplification processing on the processed first image by a first image amplification method to obtain a sample image; Based on the annotation information and the destructiveness of the first image augmentation method to chirality, the image label is smoothed to obtain a sample label.
2. The method according to claim 1, characterized in that The detecting of the first target object on the first image to obtain a detection result includes: Performing face detection on the first image to obtain a face detection result; When it is confirmed based on the detection result that the first target object exists, blurring the first target object in the first image to obtain a processed first image, or adding annotation information to the first image, includes: When it is confirmed that there are facial features based on the face detection result, blurring is performed on the facial features to obtain a processed first image.
3. The method according to claim 1 or 2, characterized in that: The detecting of the first target object on the first image to obtain a detection result includes: Performing obstacle detection on the sample image to be processed to obtain an obstacle detection result; When it is confirmed based on the detection result that the first target object exists, blurring the first target object in the first image to obtain a processed first image, or adding annotation information to the first image, includes: When it is confirmed that there is an obstacle based on the obstacle detection result, annotation information is added to the first image.
4. The method according to any one of claims 1 to 3, characterized in that: The detecting of the first target object on the first image to obtain a detection result includes: Performing optical character recognition (OCR) detection on the sample image to be processed to obtain a character detection result; When it is confirmed based on the detection result that the first target object exists, blurring the first target object in the first image to obtain a processed first image, or adding annotation information to the first image, includes: Based on the character detection result, when it is confirmed that there are characters, identifying the characters in the character detection result to obtain at least one character set; Classifying each character set in the at least one character set to obtain classification results of each character set; the classification includes autonomous driving related types and non-autonomous driving types; When the character set belongs to a non-autonomous driving type, the character set in the first image is blurred to obtain a processed first image; and when the character set belongs to an autonomous driving related type, annotation information is added to the first image.
5. The method according to claim 4, characterized in that The classifying each character set in the at least one character set comprises: Matching each character set in the at least one character set with a character library; If the character set successfully matches the character library, then the character set belongs to a type related to autonomous driving; If the character set fails to match the character library, the character set belongs to a non-autonomous driving type.
6. The method according to claim 1, characterized in that The smoothing process of the image label based on the annotation information and the destructiveness of the hand of the first image augmentation method comprises: When the annotation information indicates that a target object with chirality exists in the image, determining whether the first image augmentation method is destructive to the chirality; When the first image augmentation method is destructive to chirality, the image label is smoothed.
7. A method for training a perceptual deep learning model, characterized in that: The method comprises: Acquire a plurality of sample data, wherein the sample data includes a sample image and a sample label, wherein the sample image and the sample label are obtained by the method according to any one of claims 1 to 6; The perceptual deep learning model is trained using the multiple sample data to obtain a trained perceptual deep learning model.
8. A sample image acquisition device, characterized in that: include: A data acquisition module, used to acquire a sample to be processed, wherein the sample to be processed includes a first image and an image label; An object detection module, configured to detect a first target object in the first image and obtain a detection result; a data processing module, configured to, when confirming the presence of the first target object based on the detection result, perform blurring processing on the first target object in the first image to obtain a processed first image, or add annotation information to the first image; wherein the annotation information is used to indicate whether there is a chiral target object in the image; A data augmentation module, configured to perform image augmentation processing on the processed first image by a first image augmentation method to obtain a sample image; The label processing module is used to perform smoothing processing on the image label based on the annotation information and the destructiveness of the first image augmentation method to chirality to obtain a sample label.
9. A training device for a perceptual deep learning model, characterized in that: include: A sample acquisition module, used to acquire a plurality of sample data, wherein the sample data includes a sample image and a sample label, wherein the sample image and the sample label are obtained by the method according to any one of claims 1 to 6; The model training module is used to train the perceptual deep learning model through the multiple sample data to obtain the trained perceptual deep learning model.
10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6, or to enable the at least one processor to perform the method of claim 7.
11. A computer-readable storage medium, characterized in that: include: The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 6 is implemented, or when the computer program is executed by the processor, the method according to claim 7 is implemented.