Traffic light data augmentation method and device

Abstract

The present invention discloses a data augmentation method and device for traffic lights, relating to the field of intelligent driving technology. A specific embodiment of the method includes: obtaining at least one original image annotated with a truth box, determining a region of interest from the original image based on the truth box, and forming a training sample based on the region of interest; the original image is further annotated with: the environment type of the original image; determining the aspect ratio of the truth box of the training sample, and if the aspect ratio meets a preset longitudinal distribution condition, performing a horizontal flip on the image in the truth box to form a new training sample; for any training sample, obtaining a target image that meets preset environmental conditions similar to the training sample based on the environment type of the training sample, and mixing the target image with the training sample to form a new training sample. This embodiment can avoid semantic errors caused by horizontal flipping and image mixing in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a method and device for augmenting traffic light data. Background Art

[0002] In the field of intelligent driving, it is necessary to train traffic light localization models and traffic light recognition models to detect and identify traffic lights in images, thereby providing assistance for guiding driving. During model training, data augmentation strategies are generally used to enrich training samples and improve the generalization ability of the model. Common data augmentation strategies include horizontal flipping and image blending. However, performing a horizontal flip on a horizontally distributed traffic light image will change the order of the traffic lights, resulting in semantic errors. For example, a traffic light arranged horizontally in a fixed order of red, yellow, and green will become green, yellow, and red after horizontal flipping. Blending two images with significantly different image environments will destroy the color information of the traffic lights. For example, blending two traffic light images during the day and at night will make the traffic lights appear gray. In addition, night traffic light images will produce a halo larger than the traffic light itself. The pre-annotated ground truth box often cannot cover the halo, resulting in incomplete color information of the traffic light. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method and apparatus for traffic light data augmentation, which can avoid semantic errors caused by horizontal flipping and image mixing in existing data augmentation solutions.

[0004] To achieve the above objective, according to one aspect of the present invention, a method for traffic light data augmentation is provided.

[0005] A data augmentation method for a traffic light in an embodiment of the present invention includes: obtaining at least one original image annotated with a truth box, determining a region of interest from the original image based on the truth box, and forming a training sample based on the region of interest; wherein the truth box is used to indicate the location of the traffic light in the original image, and the original image is further annotated with: the environment type of the original image; determining the aspect ratio of the truth box of the training sample, and when the aspect ratio meets a preset longitudinal distribution condition, performing a horizontal flip on the image in the truth box to form a new training sample; for any training sample, obtaining a target image that meets preset environmental conditions similar to the training sample based on the environment type of the training sample, and mixing the target image with the training sample to form a new training sample.

[0006] Optionally, the original image is further annotated with: halo range data of the traffic light; and the method further includes: expanding the range of the true value frame corresponding to the traffic light according to the halo range data of the traffic light; wherein the halo range data is positively correlated with the expansion amplitude of the true value frame.

[0007] Optionally, the target image is an image in an external image library that is marked with a true value box; and mixing the target image with the training sample to form a new training sample includes: cutting out a region of interest from the target image based on the true value box of the target image; and superimposing the region of interest on the background part of the training sample.

[0008] Optionally, the target image is a training sample; and mixing the target image and the training sample to form a new training sample includes: adding pixel values ​​of pixels at the same position of the target image and the training sample according to fixed weights to form a new training sample.

[0009] Optionally, determining a region of interest from the original image based on the truth box and forming a training sample based on the region of interest includes: determining a plurality of regions of interest covering the truth box in the original image; wherein the truth box is at different positions in different regions of interest; and determining each region of interest as a training sample.

[0010] Optionally, the method further includes: when it is determined that the aspect ratio of the true value box of the training sample does not meet the preset vertical distribution condition, not performing horizontal flipping on the image in the true value box; after performing the mixing, using a preset stitching strategy to stitch multiple training samples and perform affine transformation to form a new training sample; transforming the current training sample into hue, saturation, and brightness space, and perturbing the hue, saturation and / or brightness components of the training sample to form a new training sample.

[0011] Optionally, the training samples are used to train a preset traffic light recognition model, and the original image is further annotated with: the color and shape of the traffic light; the longitudinal distribution condition includes: the aspect ratio of the true value box is greater than a preset threshold; the environment types include: sunny daytime, rainy daytime, foggy daytime, sunny night, rainy night, foggy night; the similar environment condition includes: the environment types of the two images to be compared are both daytime or both nighttime.

[0012] To achieve the above objective, according to another aspect of the present invention, a data augmentation device for a traffic light is provided.

[0013] The data augmentation device for a traffic light in an embodiment of the present invention may include: a sample generation unit, configured to obtain at least one original image annotated with a truth box, determine a region of interest from the original image based on the truth box, and form a training sample based on the region of interest; wherein the truth box is used to indicate the location of the traffic light in the original image, and the original image is further annotated with the environment type of the original image; a flipping unit, configured to determine the aspect ratio of the truth box of the training sample, and when the aspect ratio meets a preset longitudinal distribution condition, perform horizontal flipping on the image in the truth box to form a new training sample; and a mixing unit, configured to obtain, for any training sample, a target image that meets preset environmental conditions similar to those of the training sample based on the environment type of the training sample, and mix the target image with the training sample to form a new training sample.

[0014] Optionally, the original image is further annotated with: the halo range data of the traffic light, and the target image is an image or training sample in an external image library that is annotated with a true value frame; and the device further includes: a halo processing unit, used to: expand the range of the true value frame corresponding to the traffic light according to the halo range data of the traffic light; wherein the halo range data is positively correlated with the expansion amplitude of the true value frame; the mixing unit is further used to: cut out a region of interest from the target image based on the true value frame of the target image; superimpose the region of interest on the background part of the training sample; or, add the pixel values ​​of the pixels at the same position of the target image and the training sample according to a fixed weight to form a new training sample.

[0015] To achieve the above objective, according to another aspect of the present invention, an electronic device is provided.

[0016] An electronic device of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the traffic light data augmentation method provided by the present invention.

[0017] To achieve the above objective, according to another aspect of the present invention, a computer-readable storage medium is provided.

[0018] A computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the data augmentation method for a traffic light provided by the present invention.

[0019] According to the technical solution of the present invention, the embodiments of the above invention have the following advantages or beneficial effects:

[0020] First, the region of interest is segmented from the original image to obtain training samples. Then, before executing the horizontal flip data augmentation strategy, the aspect ratio of the ground truth box is first determined to be greater than a preset threshold. If so, it indicates that the current traffic light is vertically distributed, and horizontal flipping is performed to generate new training samples. Otherwise, it indicates that the traffic light may be horizontally distributed, and horizontal flipping is not performed to prevent semantic errors in the training samples. When executing the image mixing data augmentation strategy, it is first determined whether the environment types of the two images are similar. Only when they are similar can they be mixed, thereby avoiding semantic errors caused by mixing images with large environmental differences. When facing images with night environment types, the ground truth box range can be adaptively expanded based on the pre-labeled halo range data so that the final ground truth box can cover the halo, thereby ensuring the integrity of the traffic light color information. Through the above strategies, the quality of training samples after data augmentation can be greatly improved, which helps to improve the performance of traffic light positioning models or traffic light recognition models.

[0021] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0023] Figure 1 is a flow chart of a method for augmenting traffic light data in an embodiment of the present invention;

[0024] Figure 2 1 is a schematic diagram of specific execution steps of the method for augmenting traffic light data in an embodiment of the present invention;

[0025] Figure 3 Schematic diagram of the components of a data augmentation device for a traffic light according to an embodiment of the present invention;

[0026] Figure 4 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0027] Figure 5 2 is a schematic diagram of the structure of an electronic device used to implement the data augmentation method for a traffic light in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] It should be pointed out that, in the absence of conflict, the embodiments of the present invention and the technical features therein may be combined with each other.

[0030] Figure 1 FIG. 4 is a schematic diagram of main steps of a method for augmenting traffic light data according to an embodiment of the present invention.

[0031] like Figure 1 As shown, the data augmentation method for traffic lights according to the embodiment of the present invention can be specifically performed according to the following steps:

[0032] Step S101: obtaining at least one original image marked with a truth box, determining a region of interest from the original image according to the truth box, and forming a training sample according to the region of interest.

[0033] The data augmentation method of the embodiment of the present invention includes but is not limited to being applied to the training process of traffic light-related models. The traffic light-related models may include traffic light recognition models, traffic light positioning models, etc. The former is used to identify the color, shape and other information of traffic lights in the image, and the latter is used to detect the location of traffic lights in the image. The following mainly uses the traffic light recognition model as an example to illustrate the technical solution of the present invention.

[0034] In practical applications, the original image can be pre-annotated with various label information, such as the color and shape information of the ground truth box and the traffic light. The shape of the ground truth box includes but is not limited to: circle, square, diamond, rectangle, etc., which is used to frame the location of each traffic light in the original image; the color of the traffic light can be red, yellow, green, black, etc., among which red, yellow, and green are the colors when the traffic light is on, and black is the color when the traffic light is off; the shape of the traffic light can be circle, arrow, number, text, etc.

[0035] In specific applications, training samples can be generated in the following way. First, multiple regions of interest (ROIs) are determined in the original image based on the range of the truth box. Each region of interest covers the range of the truth box, and the truth boxes are located in different positions in different regions of interest. For example, five regions of interest are determined based on the truth box in the original image. The range of each region of interest contains the truth box, and the truth boxes are located in the upper left, lower left, upper right, lower right and middle of the five regions of interest, respectively. After that, each region of interest is used as an initial training sample.

[0036] When generating training samples, a relatively balanced number of samples can be generated for different traffic light colors and shapes to ensure the model's balanced and generalized capabilities. For example, an equal number of training samples can be generated for each of the 16 combinations of the four colors red, yellow, green, and black, and the four shapes of circle, arrow, number, and text.

[0037] As a preferred solution, corresponding processing for haloes can be performed before or after determining the region of interest. It is understandable that in traffic light images collected at night, a halo larger than the traffic light itself will appear around the traffic light. The color information of the halo is also part of the traffic light information and should be saved and used for training. However, existing annotation methods often define the ground truth box based on the outline of the traffic light itself, which will lose the halo color information and affect the model recognition accuracy.

[0038] Specifically, in an embodiment of the present invention, the original image collected at night is pre-annotated with the halo range data of each traffic light, such as the level of the halo range, which is measured by the ratio of the halo area to the area of ​​the traffic light itself. For example, a ratio greater than 2 corresponds to halo level 3, a ratio greater than 1.5 and less than or equal to 2 corresponds to halo level 2, and a ratio less than or equal to 1.5 corresponds to halo level 1. During processing, the range of the true value frame corresponding to the traffic light is expanded according to the halo range data of the traffic light, wherein the halo range data is positively correlated with the expansion amplitude of the true value frame. For example, when the halo range data of the traffic light is halo level 1, the range of the true value frame is expanded to 1.5 times along the original center; when the halo range data of the traffic light is halo level 2, the range of the true value frame is expanded to 2 times along the original center; when the halo range data of the traffic light is halo level 3, the range of the true value frame is expanded to 4 times along the original center.

[0039] Step S102: Determine the aspect ratio of the true value frame of the training sample, and when the aspect ratio meets the preset vertical distribution condition, perform horizontal flipping on the image in the true value frame to form a new training sample.

[0040] In the field of data augmentation, horizontal flipping is generally used rather than vertical flipping to construct new training samples. This is because vertical flipping is prone to semantic errors. For example, an image of a bird flying above the water will appear underwater after being flipped vertically. However, horizontally flipping an image of horizontally arranged traffic lights will change the order of the lights, similarly causing semantic errors. For example, traffic lights arranged horizontally in a fixed order of red, yellow, and green (from left to right) will become green, yellow, and red after being flipped horizontally.

[0041] Taking the above problems into consideration, in this step, before performing horizontal flipping, the aspect ratio of the true value box of the training sample (i.e., the ratio of the vertical width to the horizontal length) is first calculated to determine whether the aspect ratio meets the preset vertical distribution condition: if so, it means that the traffic light is vertically distributed and horizontal flipping can be performed, then the image in the true value box is horizontally flipped to form a new training sample; if it does not meet the vertical distribution condition, it means that the traffic light may be horizontally distributed, and horizontal flipping is not performed. It can be understood that the above vertical distribution conditions can be flexibly set according to the actual environment, as long as the vertical distribution of the traffic light can be limited. For example, it can be set to: the aspect ratio of the true value box is greater than the threshold (the threshold is generally greater than 1), the true value box is in a preset numerical range, etc.

[0042] Step S103: For any training sample, a target image that meets preset environment conditions similar to those of the training sample is obtained according to the environment type of the training sample, and the target image and the training sample are mixed to form a new training sample.

[0043] In a specific scenario, when using image blending methods to perform data augmentation on traffic lights, blending two images with significantly different image environments can distort the color information of the traffic lights. For example, blending two images of traffic lights during the day and at night will cause the traffic lights to appear gray, resulting in semantic errors. The present invention addresses this issue through the following approach. First, the original images are pre-labeled with their environment types. For example, these environment types may include: sunny day, rainy day, foggy day, sunny night, rainy night, and foggy night. Before performing image blending, two images with similar environment types are selected for blending. That is, for any current training sample, a target image that meets the environmental similarity criteria as the training sample is selected for blending. It is understood that the above environmental similarity criteria can be flexibly set according to actual needs, as long as the environmental information between the different images is sufficiently similar so that no semantic errors will occur after blending. For example, the environmental similarity criteria can be set as follows: the environmental types of the two images to be compared are both daytime or both nighttime. Alternatively, the environmental similarity criteria can be set as follows: the environmental types of the two images to be compared are either sunny daytime / rainy daytime or sunny nighttime / rainy nighttime. Other scenarios are not considered environmentally similar.

[0044] In real-world scenarios, there are multiple methods for generating new training samples through image blending. Two are described below. In the first method, the target image is an image from an external image library, labeled with a ground-truth box. This external image library refers to an image library outside the training set of the training sample. To enhance the model's recognition accuracy for niche scenes, this external library can be used to store specific, niche traffic light images, such as those of traffic lights in backlit conditions or other rare types of traffic lights. During the image blending process, a region of interest (ROI) is first segmented from the target image based on its ground-truth box. Similarly, the ROI's range encompasses the ground-truth box. This ROI is then superimposed at a random position on the background of the training sample to form a new training sample. The background refers to the original ground-truth box of the training sample, meaning the portion of the training sample excluding the original ground-truth box image. As can be seen, the new training sample incorporates the traffic lights from both the original training sample and the target image, helping to optimize model training. Furthermore, after generating a new training sample using any method, the original training sample remains in the training set or batch.

[0045] In the second method, the target image is also a training sample, that is, the method uses two different training samples for mixing. Specifically, the pixel values ​​of the pixels at the same position of the target image and the training sample are added according to fixed weights to form a new training sample. In this way, the traffic lights and backgrounds in the two training samples are fused, which increases the number of traffic lights while improving the background complexity, which is conducive to improving the generalization ability of the model. It should be noted that no matter which image mixing method is used, it is necessary to first determine that the environment types of the images to be mixed are similar. The above two image mixing methods can be used one by one or in combination, and the present invention does not limit this. In addition, other known image mixing algorithms such as Cutmix, Mixup, etc. can also be used or combined.

[0046] As a preferred solution, after performing the above image blending, a preset stitching strategy can be used to stitch multiple training samples together and perform affine transformations (which may include translation, rotation, scaling, etc.) to form new training samples. The above stitching strategy can be implemented based on the known Mosaic algorithm, which can stitch four training samples together.

[0047] In addition, in an embodiment of the present invention, the current training sample can be transformed into a hue, saturation, and lightness space (ie, HSV space), and the hue, saturation, and / or lightness components of the training sample can be perturbed to form a new training sample.

[0048] Following these steps, a certain number of qualified samples with correct semantics can be added, which helps improve model training results. During actual training, you can first use the augmented training set for training. After training is complete, you can use the original samples without data augmentation for final training. This helps to regress the model parameters to the true distribution, thereby improving the model's reasoning performance in natural scenarios.

[0049] Figure 2 This is a schematic diagram of specific execution steps of the traffic light data augmentation method according to an embodiment of the present invention. Figure 2 In step S201, the region of interest is segmented in the original image according to the true value frame of the original image. In step S202, the range of the true value frame is expanded according to the halo range data pre-marked in the original image. In step S203, the aspect ratio of the true value frame is calculated. If the aspect ratio is greater than the threshold, step S204 is executed; otherwise, step S205 is executed. In step S204, the image in the true value frame is horizontally flipped to form a new training sample. In step S205, after detecting that the image environment types are similar, image mixing is performed to generate a new training sample. In step S206, image stitching and affine transformation are performed to form a new training sample. In step S207, the color components are jittered in the HSV space to form a new training sample. After the above steps, the data augmentation of the traffic light positioning model or the traffic light recognition model is completed.

[0050] Exemplarily, when using the training samples obtained through the above data augmentation process to train the traffic light positioning model, each training sample image is used as training input data, and the true value box in the training sample is used as a label. After the training is completed, after the real image is input into the traffic light positioning model, it is possible to detect whether there is a traffic light in the image and the traffic light coordinate data when the traffic light exists. When using the training samples obtained through the above data augmentation process to train the traffic light recognition model, each training sample image is used as training input data, and the shape and color of the traffic light in the pre-labeled training sample is used as a label. After the training is completed, after the traffic light image indicated by the traffic light coordinate data output by the traffic light positioning model is input into the traffic light recognition model, the color and shape of the traffic light in the image can be detected, thereby serving businesses such as autonomous driving.

[0051] In the technical solution of the embodiment of the present invention, the region of interest is first cut out from the original image to obtain a training sample. Thereafter, before executing the data augmentation strategy of horizontal flipping, it is first determined whether the aspect ratio of the true value box is greater than a preset threshold. If so, it indicates that the current traffic light is vertically distributed, and horizontal flipping is performed to generate a new training sample. Otherwise, it indicates that the traffic light may be horizontally distributed, and horizontal flipping is not performed to prevent semantic errors in the training sample. When executing the data augmentation strategy of image mixing, it is first determined whether the environment types of the two images are similar. Only when they are similar can they be mixed, thereby avoiding semantic errors caused by mixing images with large environmental differences. When facing an image with a night environment type, the true value box range can be adaptively expanded according to the pre-labeled halo range data so that the final true value box can cover the halo, thereby ensuring the integrity of the color information of the traffic light. Through the above strategy, the quality of the training samples after data augmentation can be greatly improved, which helps to improve the performance of the traffic light positioning model or the traffic light recognition model.

[0052] It should be noted that for the aforementioned method embodiments, for ease of description, they are expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited to the order of the actions described, and certain steps can actually be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required to implement the present invention.

[0053] In order to better implement the above solutions of the embodiments of the present invention, relevant devices for implementing the above solutions are also provided below.

[0054] See also Figure 3 As shown, the data augmentation device 300 for a traffic light provided by an embodiment of the present invention may include: a sample generating unit 301 , a flipping unit 302 and a mixing unit 303 .

[0055] Among them, the sample generation unit 301 can be used to: obtain at least one original image marked with a true value box, determine the region of interest from the original image based on the true value box, and form a training sample based on the region of interest; wherein, the true value box is used to indicate the location of the traffic light in the original image, and the original image is further marked with: the environment type of the original image; the flipping unit 302 can be used to: determine the aspect ratio of the true value box of the training sample, and when the aspect ratio meets the preset longitudinal distribution condition, perform horizontal flipping on the image in the true value box to form a new training sample; the mixing unit 303 can be used to: for any training sample, obtain a target image that meets the preset environment conditions similar to the training sample based on the environment type of the training sample, and mix the target image with the training sample to form a new training sample.

[0056] In an embodiment of the present invention, the original image is further annotated with: the halo range data of the traffic light, and the target image is an image or training sample in an external image library that is annotated with a true value frame; and the device 300 may further include: a halo processing unit, used to: expand the range of the true value frame corresponding to the traffic light according to the halo range data of the traffic light; wherein the halo range data is positively correlated with the expansion amplitude of the true value frame; the mixing unit 303 may be further used to: cut out a region of interest from the target image based on the true value frame of the target image; superimpose the region of interest on the background part of the training sample; or, add the pixel values ​​of the pixels at the same position of the target image and the training sample according to a fixed weight to form a new training sample.

[0057] Preferably, the sample generation unit 301 may be further configured to: determine a plurality of regions of interest covering the true value frame in the original image; wherein the true value frame is located at different positions in different regions of interest; and determine each region of interest as a training sample.

[0058] As a preferred solution, the flipping unit 302 can be further used to: when it is determined that the aspect ratio of the true value frame of the training sample does not meet the preset vertical distribution condition, not perform horizontal flipping on the image in the true value frame; the device 300 can further include: a supplementary augmentation unit, used to: after performing the mixing, use a preset splicing strategy to splice multiple training samples and perform affine transformation to form a new training sample; transform the current training sample into the hue, saturation, and brightness space, and perturb the hue, saturation and / or brightness components of the training sample to form a new training sample.

[0059] In addition, in an embodiment of the present invention, the training samples are used to train a preset traffic light recognition model, and the original image is further annotated with: the color and shape of the traffic light; the longitudinal distribution conditions include: the aspect ratio of the true value box is greater than a preset threshold; the environment types include: sunny daytime, rainy daytime, foggy daytime, sunny night, rainy night, foggy night; the similar environment conditions include: the environment types of the two images to be compared are both daytime or both nighttime.

[0060] According to the technical solution of an embodiment of the present invention, the region of interest is first cut out from the original image to obtain a training sample. Thereafter, before executing the data augmentation strategy of horizontal flipping, it is first determined whether the aspect ratio of the true value box is greater than a preset threshold. If so, it indicates that the current traffic light is vertically distributed, and horizontal flipping is performed to generate a new training sample. Otherwise, it indicates that the traffic light may be horizontally distributed, and horizontal flipping is not performed to prevent semantic errors in the training sample. When executing the data augmentation strategy of image mixing, it is first determined whether the environment types of the two images are similar. Only when they are similar can they be mixed, thereby avoiding semantic errors caused by mixing images with large environmental differences. When facing an image with a night environment type, the true value box range can be adaptively expanded according to the pre-labeled halo range data so that the final true value box can cover the halo, thereby ensuring the integrity of the color information of the traffic light. Through the above strategy, the quality of the training samples after data augmentation can be greatly improved, which helps to improve the performance of the traffic light positioning model or the traffic light recognition model.

[0061] Figure 4 An exemplary system architecture 400 is shown to which a data augmentation method or a data augmentation device for a traffic light according to an embodiment of the present invention may be applied.

[0062] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, and 403, a network 404, and a server 405 (this architecture is merely an example, and the components included in the specific architecture may be adjusted based on the specific application). Network 404 is used to provide a medium for communication links between terminal devices 401, 402, and 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0063] Users can use terminal devices 401, 402, 403 to interact with server 405 via network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as data augmentation applications (only as an example).

[0064] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0065] Server 405 may be a server that provides various services, such as a backend server (for example only) that supports data augmentation applications operated by users using terminal devices 401, 402, and 403. The backend server may process received data augmentation requests and feedback processing results (for example, augmented training samples—for example only) to terminal devices 401, 402, and 403.

[0066] It should be noted that the traffic light data augmentation method provided in the embodiment of the present invention is generally executed by the server 405 . Accordingly, the traffic light data augmentation device is generally disposed in the server 405 .

[0067] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0068] The present invention also provides an electronic device. The electronic device in an embodiment of the present invention includes: one or more processors; and a storage device configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the traffic light data augmentation method provided by the present invention.

[0069] Reference below Figure 5 , which shows a schematic structural diagram of a computer system 500 of an electronic device suitable for implementing an embodiment of the present invention. Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0070] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the computer system 500 are also stored in the RAM 503. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0071] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output section 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or a modem. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 510 as needed, so that computer programs read therefrom can be installed in the storage section 508 as needed.

[0072] In particular, according to embodiments disclosed herein, the processes described in the main step diagrams above can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via the communication section 509 and / or installed from removable media 511. When the computer program is executed by the central processing unit 501, the above-described functions defined in the system of the present invention are performed.

[0073] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0074] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0075] The units involved in the embodiments of the present invention may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as: a processor including a sample generation unit, a flipping unit, and a mixing unit. The names of these units do not, in some cases, limit the units themselves. For example, the sample generation unit may also be described as a "unit that provides training samples to the flipping unit and the mixing unit."

[0076] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not be assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device performs the following steps: obtaining at least one original image annotated with a truth box, determining a region of interest from the original image based on the truth box, and forming a training sample based on the region of interest; wherein the truth box is used to indicate the location of a traffic light in the original image, and the original image is further annotated with: the environment type of the original image; determining the aspect ratio of the truth box of the training sample, and if the aspect ratio meets a preset longitudinal distribution condition, performing a horizontal flip on the image in the truth box to form a new training sample; for any training sample, obtaining a target image that meets the preset environment conditions similar to the training sample based on the environment type of the training sample, and mixing the target image with the training sample to form a new training sample.

[0077] In the technical solution of the embodiment of the present invention, the region of interest is first cut out from the original image to obtain a training sample. Thereafter, before executing the data augmentation strategy of horizontal flipping, it is first determined whether the aspect ratio of the true value box is greater than a preset threshold. If so, it indicates that the current traffic light is vertically distributed, and horizontal flipping is performed to generate a new training sample. Otherwise, it indicates that the traffic light may be horizontally distributed, and horizontal flipping is not performed to prevent semantic errors in the training sample. When executing the data augmentation strategy of image mixing, it is first determined whether the environment types of the two images are similar. Only when they are similar can they be mixed, thereby avoiding semantic errors caused by mixing images with large environmental differences. When facing an image with a night environment type, the true value box range can be adaptively expanded according to the pre-labeled halo range data so that the final true value box can cover the halo, thereby ensuring the integrity of the color information of the traffic light. Through the above strategy, the quality of the training samples after data augmentation can be greatly improved, which helps to improve the performance of the traffic light positioning model or the traffic light recognition model.

[0078] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A traffic light data augmentation method, characterized in that: include: Obtain at least one original image annotated with a truth box, determine a region of interest from the original image based on the truth box, and form a training sample based on the region of interest; wherein the truth box is used to indicate the location of a traffic light in the original image, and the original image is further annotated with: the environment type of the original image; Determining the aspect ratio of the true value frame of the training sample, and performing horizontal flipping on the image in the true value frame to form a new training sample when the aspect ratio meets a preset vertical distribution condition; For any training sample, a target image that meets the preset environmental similarity conditions with the training sample is obtained based on the environmental type of the training sample, and the target image is mixed with the training sample to form a new training sample; the environmental similarity condition refers to the condition that can limit the environmental information between different images to be sufficiently similar and no semantic errors will occur after mixing with each other.

2. The method according to claim 1, characterized in that The original image is further annotated with: halo range data of the traffic light; and the method further comprises: The range of the true value frame corresponding to the traffic light is expanded according to the halo range data of the traffic light; wherein the halo range data is positively correlated with the expansion amplitude of the true value frame.

3. The method according to claim 1, characterized in that The target image is an image in an external image library and annotated with a true value box; and mixing the target image with the training sample to form a new training sample includes: Segmenting a region of interest from the target image according to a true value frame of the target image; The region of interest is superimposed on the background portion of the training sample.

4. The method according to claim 1, wherein The target image is a training sample; And, mixing the target image and the training sample into a new training sample comprises: The pixel values ​​of the pixels at the same position of the target image and the training sample are added according to fixed weights to form a new training sample.

5. The method according to claim 1, wherein The determining a region of interest from the original image according to the true value frame, and forming a training sample according to the region of interest, includes: Determining a plurality of regions of interest encompassing the ground truth box in the original image; wherein the ground truth box is located at different positions in different regions of interest; Each region of interest is identified as a training sample.

6. The method according to claim 3 or 4, characterized in that The method further comprises: When the aspect ratio of the true value frame of the training sample does not meet the preset vertical distribution condition, the image in the true value frame is not horizontally flipped; After performing the mixing, a preset splicing strategy is used to splice the multiple training samples and perform affine transformation to form a new training sample; The current training sample is transformed into the hue, saturation, and lightness space, and the hue, saturation, and / or lightness components of the training sample are perturbed to form a new training sample.

7. The method according to any one of claims 1 to 5, characterized in that: The training samples are used to train a preset traffic light recognition model, and the original image is further annotated with: the color and shape of the traffic light; The longitudinal distribution condition includes: the aspect ratio of the true value frame is greater than a preset threshold; The environment types include: sunny day, rainy day, foggy day, sunny night, rainy night, foggy night, and the similar environment condition includes: the environment types of the two images to be compared are both daytime or both nighttime.

8. A data augmentation device for a traffic light, characterized in that: include: A sample generation unit is configured to obtain at least one original image annotated with a truth box, determine a region of interest from the original image based on the truth box, and form a training sample based on the region of interest; wherein the truth box is used to indicate the location of the traffic light in the original image, and the original image is further annotated with the environment type of the original image; a flipping unit, configured to determine an aspect ratio of a truth frame of the training sample, and, if the aspect ratio meets a preset longitudinal distribution condition, perform horizontal flipping on the image in the truth frame to form a new training sample; The mixing unit is used to: for any training sample, obtain a target image that meets the preset environmental similarity conditions with the training sample based on the environment type of the training sample, and mix the target image with the training sample to form a new training sample; the environmental similarity condition refers to a condition that can limit the environmental information between different images to be sufficiently similar and no semantic errors will occur after mixing with each other.

9. The device according to claim 8, characterized in that The original image is further annotated with: halo range data of the traffic light, and the target image is an image or training sample in an external image library that is annotated with a true value frame; as well as, The device further comprises: a halo processing unit, configured to: expand the range of a truth frame corresponding to the traffic light according to halo range data of the traffic light; wherein the halo range data is positively correlated with the expansion amplitude of the truth frame; The mixing unit is further used to: cut out a region of interest from the target image based on the true value frame of the target image; superimpose the region of interest on the background part of the training sample; or add the pixel values ​​of the pixels at the same position of the target image and the training sample according to a fixed weight to form a new training sample.

10. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images