Picture data enhancement method and device and electronic equipment
By dynamically adjusting the image data enhancement strategy and selecting appropriate enhancement strategies based on the verification results of model training, the overfitting and insufficient complexity caused by the static nature of the enhancement strategy in the prior art are solved, and the model's adaptability to complex scenarios is improved.
Patent Information
- Application Number
- CN202411940768.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-05-06
AI Technical Summary
The static and unified image enhancement strategy is adopted in the existing technology, and it cannot be dynamically adjusted according to different needs of the model training stage, resulting in possible overfitting in the early stage and insufficient complexity in the middle and late stages, limiting the model's adaptability to complex scenarios.
By obtaining the model verification results of model training at every preset time, including verification loss, verification accuracy and generalization error, the image data enhancement strategy is dynamically adjusted based on these results, and the primary, intermediate and higher-order enhancement strategies are adopted.
This dynamically adjusted enhancement strategy matches the enhanced complexity of each stage with the current learning needs of the model, effectively reducing the risk of overfitting or insufficient complexity, and allowing the model to adapt to a wider range of scenario changes.
Smart Images
Figure CN119941520A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine learning and computer vision technology, and in particular to a method, device and electronic device for image data enhancement. Background Art
[0002] Image data augmentation is a technique for transforming and modifying images during training, aiming to expand the size and diversity of the training dataset by applying a series of transformation operations to generate multiple variant images. Data augmentation can help improve the generalization ability of the model, increase the robustness of the model, and reduce the risk of overfitting.
[0003] Existing image enhancement methods usually adopt static and unified enhancement strategies, which cannot be dynamically adjusted according to the different requirements of the model training stage. Such enhancement strategies are prone to overfitting in the early stage of model training and insufficient complexity in the middle and late stages of training, which limits the model's adaptability to complex scenes (such as night, rainy days, strong light, etc.). Therefore, there is an urgent need for a method that can adjust the enhancement strategy according to the training results of model training. Summary of the invention
[0004] To this end, the present invention provides a method, device and electronic device for image data enhancement to solve the problem that the static and unified enhancement strategy adopted in the prior art cannot be dynamically adjusted according to the different requirements of the model training stage, which easily leads to overfitting in the early stage or insufficient complexity in the middle and late stages.
[0005] In a first aspect, a method for enhancing image data is provided, the method comprising:
[0006] Obtaining the model verification results of the model training once every preset time, wherein the model verification results include: verification loss, verification accuracy and generalization error;
[0007] Determine the next image data enhancement strategy for the preset duration according to the model verification result; the image data enhancement strategy includes a primary enhancement strategy, a mid-order enhancement strategy and a high-order enhancement strategy.
[0008] Furthermore, the step of determining the next image data enhancement strategy of the preset duration according to the model verification result includes:
[0009] Obtaining the current image data enhancement strategy as a first image data enhancement strategy;
[0010] If the first picture data enhancement strategy is a primary enhancement strategy, and the model verification result meets the preset enhancement condition, then determining that the next picture data enhancement strategy of the preset duration is a mid-order enhancement strategy;
[0011] If the first picture data enhancement strategy is a medium-order enhancement strategy, and the model verification result meets the preset enhancement condition, then determining that the next picture data enhancement strategy of the preset duration is a high-order enhancement strategy;
[0012] If the first picture data enhancement strategy is a mid-order enhancement strategy, and the model verification result meets the preset fallback condition, then determining that the next picture data enhancement strategy of the preset duration is a primary enhancement strategy;
[0013] If the first image data enhancement strategy is a high-order enhancement strategy and the model verification result meets the preset fallback condition, then the next image data enhancement strategy of the preset duration is determined to be a mid-order enhancement strategy.
[0014] Furthermore, the model verification result meets the preset enhancement conditions, including:
[0015] If the verification loss continues to decrease within the preset time period, and the verification accuracy continues to improve, then the model verification result meets the preset enhancement condition;
[0016] and / or,
[0017] If the verification loss continues to stagnate within the preset time period, then the model verification result meets the preset enhancement condition;
[0018] and / or,
[0019] If the change fluctuation of the verification loss within the preset time period exceeds the preset fluctuation, the model verification result meets the preset enhancement condition.
[0020] Furthermore, the model verification result meets the preset fallback conditions, including:
[0021] If the generalization error continues to increase within the preset time period, the model verification result meets the preset backoff condition.
[0022] Furthermore, the enhancement methods of the primary enhancement strategy include horizontal flipping, random cropping, rotating at a preset angle, and adding noise of a first preset intensity.
[0023] Furthermore, the enhancement methods of the mid-order enhancement strategy include adjusting brightness, adjusting contrast, adjusting saturation, blurring, and adding noise of a second preset intensity.
[0024] Furthermore, the enhancement methods of the high-order enhancement strategy include occlusion simulation, affine transformation and adding a third preset intensity noise.
[0025] Furthermore, the method further comprises:
[0026] If the first picture data enhancement strategy is a preliminary enhancement strategy, and the model verification result meets the preset fallback condition, the picture data enhancement strategy for the next preset duration remains unchanged;
[0027] If the first image data enhancement strategy is a high-order enhancement strategy, and the model verification result meets the preset enhancement conditions, the next image data enhancement strategy of the preset duration remains unchanged.
[0028] In a second aspect, a device for enhancing image data is provided, the device comprising:
[0029] An acquisition module is used to obtain a model verification result of model training every preset time period, wherein the model verification result includes: verification loss, verification accuracy and generalization error;
[0030] The strategy adjustment module is used to determine the next picture data enhancement strategy of the preset duration according to the model verification result; the picture data enhancement strategy includes a primary enhancement strategy, a mid-order enhancement strategy and a high-order enhancement strategy.
[0031] In a third aspect, an electronic device is provided, including:
[0032] at least one processor; and
[0033] a memory communicatively connected to the at least one processor; wherein,
[0034] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform any of the above-mentioned methods for enhancing image data.
[0035] The present invention adopts the above technical solution and has at least the following beneficial effects:
[0036] Provided are a method, device and electronic device for image data enhancement, which obtain a model verification result of model training every preset time period, and determine an image data enhancement strategy for the next preset time period based on the model verification result; the image data enhancement strategy is adjusted by verifying the loss, verification accuracy and generalization error during the training progress of the model, so that the enhancement complexity of each stage can match the current learning requirements of the model, effectively reducing the risk of overfitting or insufficient complexity of the model, and enabling the model to adapt to a wider range of scene changes.
[0037] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0039] Figure 1 is a flow chart of a method for enhancing picture data shown in an exemplary embodiment of the present invention;
[0040] Figure 2 is a schematic block diagram of a device for enhancing picture data according to an exemplary embodiment of the present invention;
[0041] Figure 3 It is a schematic block diagram of an electronic device shown in an exemplary embodiment of the present invention. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0043] In the perception system of autonomous driving, images collected by the camera are an important data source for vehicle environmental perception. However, existing image enhancement methods usually adopt static and unified enhancement strategies, and do not dynamically adjust according to the different needs of the model training stage. Such enhancement strategies are prone to overfitting in the early stages of model training, especially in the early training stage, complex enhancement methods may have a negative impact on model learning; in the later stages of training, the complexity of existing enhancement methods is insufficient, which limits the model's ability to adapt to complex scenes (such as night, rainy days, strong light, etc.). In addition, there are various environmental factors in autonomous driving scenarios (such as lighting changes, occlusion, bad weather, etc.), and these changes put higher requirements on model training.
[0044] The embodiment of the present application provides a method, device and electronic device for image data enhancement, wherein the model verification result of model training is obtained once every preset time period, and the image data enhancement strategy for the next preset time period is determined according to the model verification result; the image data enhancement strategy is adjusted by verifying the loss, verification accuracy and generalization error during the training progress of the model, which can improve the training stability: by adopting a simple enhancement method in the early stage, the interference of complex enhancement on the initial learning process of the model is avoided, thereby improving the stability of the model training; the generalization ability can be enhanced: as the training progresses, the complexity of the enhancement is gradually increased, which ensures the adaptability of the model to diversified image scenes (such as different lighting, weather and other environmental changes). The generalization performance of the model is significantly improved; it can adapt to complex environments: by introducing simulations of complex and extreme scenes (such as night, fog, rainy days, etc.) in the later stage of training, the model can maintain efficient perception capabilities in complex driving environments, thereby improving the safety and robustness of the autonomous driving system; it can dynamically adjust strategies: the present invention dynamically adjusts the data enhancement strategy according to the training progress of the model, so that the enhancement complexity of each stage matches the current learning needs of the model, thereby ensuring the effectiveness and flexibility of the enhancement strategy; it can reduce the risk of overfitting: by detecting the generalization error and automatically adjusting the enhancement complexity, it can effectively reduce the risk of overfitting of the model and ensure that the model can adapt to a wider range of scene changes.
[0045] The method and device in this application are described below through specific embodiments.
[0046] See also Figure 1 , Figure 1 is a flowchart of a method for enhancing image data according to an exemplary embodiment of the present invention, see Figure 1 , the method comprising:
[0047] Step S11, obtaining the model verification result of the model training once every preset time, the model verification result includes: verification loss, verification accuracy and generalization error;
[0048] Step S12: Determine the image data enhancement strategy for the next preset duration based on the model verification result; the image data enhancement strategy includes a primary enhancement strategy, a mid-order enhancement strategy, and a high-order enhancement strategy.
[0049] It should be noted that the technical solution provided in this embodiment can be added to the existing system in the form of a small program in specific practice, or it can also be in the form of an independent application to provide an interface to the outside world to complete the image data enhancement function; applicable scenarios include but are not limited to: image data enhancement for autonomous driving scenario model training.
[0050] It can be understood that the method provided in this embodiment obtains the model verification result of model training once every preset time period, and determines the image data enhancement strategy for the next preset time period based on the model verification result; this method of adjusting the image data enhancement strategy through verification loss, verification accuracy and generalization error during model training progress can make the enhancement complexity of each stage match the current learning needs of the model, effectively reduce the risk of overfitting or insufficient complexity of the model, and enable the model to adapt to a wider range of scenario changes.
[0051] In specific practice, step S11 "obtaining the model verification result of model training every preset time period" includes: there will be a strategy switching cycle in the process of model training, and the verification loss, verification accuracy and generalization error within the cycle will be obtained in each strategy switching cycle.
[0052] It should be noted that the length of the preset time determines the quality of model training. The shorter the preset time, the more frequently the image data enhancement strategy is switched, and the better the training quality.
[0053] Specifically, validation loss (Validation Loss, (L_v)): the loss value of the model is calculated through the validation set, and the cross entropy loss or mean square error is usually used as the measurement standard; validation accuracy (Validation Accuracy, (A_v)): the classification or detection accuracy of the model on the validation set; generalization error (Generalization Error, (E_g)): calculated by the difference between training loss and validation loss, to evaluate whether the model is overfitting; the formula is as follows: [E_g=|L_t–L_v|] where (L_t) is the training loss and (L_v) is the validation loss.
[0054] In specific practice, step S12 "determine the image data enhancement strategy of the next preset duration based on the model verification result" is specifically: obtain the current image data enhancement strategy as the first image data enhancement strategy; if the first image data enhancement strategy is a preliminary enhancement strategy, and the model verification result meets the preset enhancement conditions, then determine that the image data enhancement strategy of the next preset duration is a medium-order enhancement strategy; if the first image data enhancement strategy is a medium-order enhancement strategy, and the model verification result meets the preset enhancement conditions, then determine that the image data enhancement strategy of the next preset duration is a high-order enhancement strategy; if the first image data enhancement strategy is a medium-order enhancement strategy, and the model verification result meets the preset fallback conditions, then determine that the image data enhancement strategy of the next preset duration is a preliminary enhancement strategy; if the first image data enhancement strategy is a high-order enhancement strategy, and the model verification result meets the preset fallback conditions, then determine that the image data enhancement strategy of the next preset duration is a medium-order enhancement strategy.
[0055] Specifically, the model verification result meets the preset enhancement conditions, including: if the verification loss continues to decrease within a preset time period and the verification accuracy continues to improve, the model verification result meets the preset enhancement conditions; and / or, if the verification loss continues to stagnate within a preset time period, the model verification result meets the preset enhancement conditions; and / or, if the change fluctuation of the verification loss within the preset time period exceeds the preset fluctuation, the model verification result meets the preset enhancement conditions; wherein the preset fluctuation is set according to the image data enhancement strategy, the preset fluctuation of the primary enhancement strategy is smaller than the preset fluctuation of the intermediate enhancement strategy, and the preset fluctuation of the intermediate enhancement strategy is smaller than the preset fluctuation of the high-order enhancement strategy.
[0056] Specifically, the model verification result meets the preset backoff condition, including: if the generalization error continues to increase within a preset time period, then the model verification result meets the preset backoff condition.
[0057] It should be noted that the verification loss (L_v) continues to decrease and the verification accuracy (A_v) continues to improve: If the verification loss continues to decrease and the accuracy increases (the accuracy increases by more than 1% each time), the enhancement strategy can switch to a more complex enhancement method. The generalization error (E_g) increases: When the generalization error increases, it indicates that the model is overfitting and the system should return to the previous enhancement strategy. Verification loss stagnates or fluctuates: When the verification loss stagnates or fluctuates greatly, the model may fall into a local optimal solution. At this time, a stronger enhancement method (such as occlusion simulation, noise enhancement, etc.) can be introduced to help the model break through the local optimal solution.
[0058] In specific practice, the enhancement methods of the primary enhancement strategy include horizontal flipping, random cropping, rotation at a preset angle, and adding noise of a first preset intensity; the preset angle and the first preset intensity are set according to the requirements of model training.
[0059] It should be noted that the initial enhancement strategy is generally used in the initial stage of model training (the first 10-20 epochs), when the model mainly learns the basic features of the image. In order to avoid the interference of complex enhancement on the initial learning process, the initial enhancement strategy generally uses the following enhancement methods: horizontal flipping, random cropping, slight rotation (±15 degrees), and slight noise addition. The purpose of enhancement is to ensure the diversity of data through basic enhancement operations, while ensuring that the basic features of the image remain unchanged, helping the model quickly grasp the basic information in the image.
[0060] In specific practice, the enhancement methods of the mid-order enhancement strategy include adjusting brightness, adjusting contrast, adjusting saturation, blurring, and adding a second preset intensity noise; the second preset intensity is set according to the requirements of model training.
[0061] It should be noted that the mid-order enhancement strategy generally begins to master more complex features as the model training deepens (20-40 epochs). Therefore, the present invention introduces a more complex enhancement strategy at this stage to simulate different environmental changes (such as lighting, weather, etc.) to enhance the performance of the model in diverse image scenes; enhancement methods: brightness, contrast, saturation adjustment, blur processing, increase noise intensity; enhancement purpose: enhance the model's adaptability to different environmental factors, expand the model's learning range for diverse features, especially image features under different lighting and weather conditions.
[0062] In specific practice, the enhancement methods of the high-order enhancement strategy include occlusion simulation, affine transformation and adding a third preset intensity noise; the third preset intensity is set according to the requirements of model training.
[0063] It should be noted that in the later stage of training (after 40 epochs), the high-order enhancement strategy has a certain generalization ability. The robustness of the model can be further improved by introducing more complex and extreme enhancement methods, especially for coping with the harsh environment in autonomous driving; enhancement methods: large brightness changes, occlusion simulation (such as random erasing), high-intensity noise (simulating fog, rainy days, etc.), affine transformation; enhancement purpose: by enhancing image features in extreme scenes, the model can maintain high performance and robustness in complex environments, ensuring that it has a strong ability to cope with complex situations in autonomous driving.
[0064] Specifically, the first preset intensity is smaller than the second preset intensity; and the second preset intensity is smaller than the third preset intensity.
[0065] In specific practice, the method also includes: if the first image data enhancement strategy is a primary enhancement strategy, and the model verification result meets the preset fallback condition, then the image data enhancement strategy for the next preset duration remains unchanged; if the first image data enhancement strategy is a high-order enhancement strategy, and the model verification result meets the preset enhancement condition, then the image data enhancement strategy for the next preset duration remains unchanged.
[0066] See also Figure 2 , Figure 2 is a schematic block diagram of a device for enhancing picture data according to an exemplary embodiment of the present invention, see Figure 2 , the apparatus 100 for enhancing picture data includes:
[0067] The acquisition module 101 is used to obtain the model verification result of the model training once every preset time, and the model verification result includes: verification loss, verification accuracy and generalization error;
[0068] The strategy adjustment module 102 is used to determine the image data enhancement strategy of the next preset duration according to the model verification result; the image data enhancement strategy includes a primary enhancement strategy, a mid-order enhancement strategy and a high-order enhancement strategy.
[0069] It should be noted that the technical solution provided in this embodiment can be applied in specific practice to scenarios including but not limited to: image data enhancement for autonomous driving scenario model training.
[0070] It can be understood that the device provided in this embodiment obtains the model verification result of model training once every preset time period, and determines the image data enhancement strategy for the next preset time period based on the model verification result; this method of adjusting the image data enhancement strategy by verifying the loss, verification accuracy and generalization error during the training progress of the model can make the enhancement complexity of each stage match the current learning needs of the model, effectively reduce the risk of overfitting or insufficient complexity of the model, and enable the model to adapt to a wider range of scene changes.
[0071] See also Figure 3 , Figure 3 is a schematic block diagram of an electronic device according to an exemplary embodiment of the present invention, see Figure 3 , an electronic device 200, comprising:
[0072] at least one processor 202; and
[0073] A memory 201 is communicatively connected to at least one processor 202; wherein,
[0074] The memory 201 stores instructions that can be executed by at least one processor 202. The instructions are executed by the at least one processor 202 so that the at least one processor 202 can perform any of the above-mentioned methods for enhancing picture data.
[0075] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
[0076] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0077] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0078] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A method for image data enhancement, characterized in that: The method comprises: Obtaining the model verification results of the model training once every preset time, wherein the model verification results include: verification loss, verification accuracy and generalization error; Determine the next image data enhancement strategy for the preset duration according to the model verification result; the image data enhancement strategy includes a primary enhancement strategy, a mid-order enhancement strategy and a high-order enhancement strategy.
2. The method according to claim 1, characterized in that The step of determining the next image data enhancement strategy for the preset duration according to the model verification result includes: Obtaining the current image data enhancement strategy as a first image data enhancement strategy; If the first picture data enhancement strategy is a primary enhancement strategy, and the model verification result meets the preset enhancement condition, then determining that the next picture data enhancement strategy of the preset duration is a mid-order enhancement strategy; If the first picture data enhancement strategy is a mid-order enhancement strategy, and the model verification result meets the preset enhancement condition, then determining that the next picture data enhancement strategy of the preset duration is a high-order enhancement strategy; If the first picture data enhancement strategy is a mid-order enhancement strategy, and the model verification result meets the preset fallback condition, then determining that the next picture data enhancement strategy of the preset duration is a primary enhancement strategy; If the first image data enhancement strategy is a high-order enhancement strategy and the model verification result meets the preset fallback condition, then the next image data enhancement strategy of the preset duration is determined to be a mid-order enhancement strategy.
3. The method according to claim 2, characterized in that The model verification results meet the preset enhancement conditions, including: If the verification loss continues to decrease within the preset time period, and the verification accuracy continues to improve, then the model verification result meets the preset enhancement condition; and / or, If the verification loss continues to stagnate within the preset time period, then the model verification result meets the preset enhancement condition; and / or, If the change fluctuation of the verification loss within the preset time period exceeds the preset fluctuation, the model verification result meets the preset enhancement condition.
4. The method according to claim 2, characterized in that: The model verification result meets the preset fallback conditions, including: If the generalization error continues to increase within the preset time period, the model verification result meets the preset backoff condition.
5. The method according to claim 1, characterized in that The enhancement methods of the primary enhancement strategy include horizontal flipping, random cropping, rotating at a preset angle, and adding noise of a first preset intensity.
6. The method according to claim 1, characterized in that The enhancement methods of the mid-order enhancement strategy include adjusting brightness, adjusting contrast, adjusting saturation, blurring, and adding noise of a second preset intensity.
7. The method according to claim 1, characterized in that The enhancement methods of the high-order enhancement strategy include occlusion simulation, affine transformation and adding third preset intensity noise.
8. The method according to claim 2, characterized in that: The method further comprises: If the first picture data enhancement strategy is a preliminary enhancement strategy, and the model verification result meets the preset fallback condition, the picture data enhancement strategy for the next preset duration remains unchanged; If the first image data enhancement strategy is a high-order enhancement strategy, and the model verification result meets the preset enhancement conditions, the next image data enhancement strategy of the preset duration remains unchanged.
9. A device for enhancing image data, characterized in that: The device comprises: An acquisition module is used to obtain a model verification result of model training every preset time period, wherein the model verification result includes: verification loss, verification accuracy and generalization error; The strategy adjustment module is used to determine the next picture data enhancement strategy of the preset duration according to the model verification result; the picture data enhancement strategy includes a primary enhancement strategy, a mid-order enhancement strategy and a high-order enhancement strategy.
10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for enhancing image data according to any one of claims 1 to 8.