Target detection method and model, brightness enhancement network and device, vehicle and medium

Abstract

The invention discloses a target detection method for a vehicle, a brightness enhancement network, a detection model, an electronic device, a vehicle, a computer readable storage medium and a computer program product. The method comprises the following steps: determining a brightness enhancement image according to an acquired environment image; and based on a pre-trained brightness enhancement network, performing target detection according to the environment image and the brightness enhancement graph. Thus, when the vehicle is in the dark environment, the brightness enhancement graph representing the brightness information is obtained through the environment image, in the process of enhancing the environment image by using the brightness enhancement network, the brightness enhancement graph can assist in enhancing the brightness of the environment image, and when target detection is carried out, compared with direct target detection on the dark environment image, the brightness of the environment image is enhanced. The environment image brightness is enhanced based on the brightness enhancement network, so that the accuracy and reliability of a target detection result can be improved, the surrounding environment of the vehicle is sensed, and a safe and reasonable driving decision is made according to the detection result.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a target detection method, a brightness enhancement network, a detection model, an electronic device, a vehicle, a computer-readable storage medium, and a computer program product for a vehicle. Background Art

[0002] Perception technology is a key component of autonomous driving. It uses a variety of sensors, such as cameras, to acquire information about the surrounding scene and obstacles, and detects surrounding objects to make safe and appropriate driving decisions. However, in dimly lit environments, visually based environmental information is reduced due to the decrease in brightness, resulting in reduced accuracy and reliability of object detection results. Summary of the Invention

[0003] The present application provides a target detection method for a vehicle, a brightness enhancement network, a detection model, an electronic device, a vehicle, a computer-readable storage medium, and a computer program product.

[0004] The present application provides a method for detecting a target in a vehicle, comprising:

[0005] Determine a brightness enhancement map according to the acquired environment image;

[0006] Based on a pre-trained brightness enhancement network, target detection is performed according to the environment image and the brightness enhancement map.

[0007] In summary, in the target detection method for a vehicle provided in the embodiments of the present application, when the vehicle is in a dark environment, a brightness enhancement map representing brightness information is obtained from the environmental image. In the process of using the brightness enhancement network to enhance the environmental image, the brightness enhancement map can assist in enhancing the brightness of the environmental image. Specifically, it can assist in brightness enhancement of the dark areas of the environmental image. In this way, when performing target detection, compared to directly performing target detection on the dark environment image, in the embodiments of the present application, enhancing the brightness of the environmental image based on the brightness enhancement network can improve the accuracy and reliability of the target detection results, thereby perceiving the vehicle's surrounding environment and making safe and reasonable driving decisions based on the detection results.

[0008] In some embodiments, determining the brightness enhancement map based on the acquired environment image includes:

[0009] In a case where the ambient brightness of the environment where the vehicle is located is less than or equal to a first threshold, the brightness enhancement map is determined according to the environmental image.

[0010] In this way, when the ambient brightness of the vehicle's environment is less than or equal to the first threshold, a brightness enhancement map is determined based on the ambient image. The first ambient brightness threshold is used to distinguish between bright and dark environments. In dark environments, it is determined that brightness information enhancement of the ambient image is necessary, thereby improving the accuracy and reliability of target detection results.

[0011] In some embodiments, determining the brightness enhancement map based on the acquired environment image further includes:

[0012] When the ambient brightness is greater than the first threshold, target detection is performed based on the ambient image.

[0013] In this way, in bright light environments, the brightness information of the ambient image can support direct target detection with better accuracy and reliability, without the need for brightness enhancement through brightness enhancement maps, etc.

[0014] In some embodiments, determining the brightness enhancement map based on the acquired environment image further includes:

[0015] When the average value of image pixels of the environment image is less than or equal to a second threshold, the brightness enhancement map is determined according to the environment image.

[0016] In this way, the second threshold of the image pixel average value is used to distinguish between bright and dark environments. In a dark environment, it is determined that the brightness information of the environmental image needs to be enhanced, thereby improving the accuracy and reliability of the target detection result.

[0017] In some embodiments, determining the brightness enhancement map based on the acquired environment image further includes:

[0018] When the image pixel average value is greater than a second threshold, target detection is performed based on the environment image.

[0019] In this way, in bright light environments, the brightness information of the ambient image can support direct target detection with better accuracy and reliability, without the need for brightness enhancement through brightness enhancement maps, etc.

[0020] In some embodiments, determining the brightness enhancement map based on the acquired environment image further includes:

[0021] Performing color space conversion processing on the environment image to obtain target color space data;

[0022] The brightness enhancement map is determined according to the brightness channel data in the target color space data.

[0023] In this way, by performing color space conversion processing on the environment image, the environment image is converted from RGB to HSV space, wherein the V channel can represent the brightness information of the environment image, and then the brightness enhancement map can be determined.

[0024] In some embodiments, determining the brightness enhancement map based on the brightness channel data in the target color space data includes:

[0025] Normalization processing is performed on the brightness channel data, and the brightness enhancement map is determined based on the result of the normalization processing.

[0026] In this way, when converting the dark environment image from RGB to HSV space and processing the brightness channel data V, normalization processing is used to unify the numerical range, facilitate mathematical operations, avoid numerical variation, ensure processing consistency, and facilitate compatibility with subsequent processing, which meets the input requirements of the brightness enhancement network.

[0027] The embodiment of the present application further provides a brightness enhancement network, including:

[0028] an encoder configured to perform downsampling processing on an input image to determine an output image, wherein the input image is determined according to an environment image and the brightness enhancement map, and the brightness enhancement map is determined according to the environment image;

[0029] A decoder is configured to perform upsampling processing on the output image and determine a target brightness image, so that the target detection head of the brightness enhancement network cascade performs target detection based on the target brightness image.

[0030] In this way, the brightness enhancement map and the ambient image serve as input images for the brightness enhancement network. With the assistance of the brightness enhancement map, the brightness enhancement network performs brightness enhancement processing on the dark areas of the ambient image to determine the target brightness image. The brightness enhancement network includes an encoder, which downsamples the brightness enhancement map input image as an input image. The brightness enhancement map is the input image of the encoder and also the input image during the downsampling process, helping the encoder to improve the brightness of the enhanced ambient image to obtain an output image. The brightness enhancement network also includes a decoder cascaded with the encoder, which upsamples the output image to obtain a target brightness image with richer details and accurate brightness. In this way, compared to the low-light ambient image, the overall brightness of the target brightness image obtained after processing by the brightness enhancement network is enhanced, and the outline of the target object in the image is clearer and the detailed features are more obvious. Therefore, the brightness enhancement network provides higher-quality input to the cascaded target detection head, to a certain extent reducing the impact of insufficient ambient brightness on the target object detection results, and improving the accuracy and reliability of target detection.

[0031] In some embodiments, the input image includes a first input image, the encoder includes a first downsampling module, and the first downsampling module is configured to downsample the first input image to determine a first feature image, wherein the first input image is determined based on the ambient image and the brightness enhancement map, the ambient image and the brightness enhancement map have the same size, and the number of channels of the first feature image is twice that of the first input image.

[0032] In this way, matrix dot multiplication of the ambient image and the brightness enhancement map improves the brightness distribution of the downsampled object, making the dark region features prominent and the bright region features suppressed. The first downsampling module downsamples the first input image to obtain a first feature image with enhanced brightness and the features of the first downsampling process at that scale. This downsampling of the first input image by the first downsampling module improves the quality of the input image of the downsampling module cascaded with the first downsampling module.

[0033] In some embodiments, the input image also includes a second input image, and the encoder also includes a second downsampling module cascaded with the first downsampling module, and the second downsampling module is configured to downsample the second input image to determine a second feature image, wherein the second input image is determined based on the first feature image and the first brightness enhancement map, the first brightness enhancement map is obtained by downsampling the brightness enhancement map, the first feature image and the first brightness enhancement map have the same size, and the number of channels of the second feature image is twice that of the second input image.

[0034] In this way, matrix dot multiplication of the first feature image and the first brightness enhancement map improves the brightness distribution of the downsampled object, enhancing features in dark areas and suppressing features in bright areas. Downsampling the second input image by the second downsampling module yields a second feature image that combines the features of both the first and second downsampling processes, with enhanced brightness. Thus, downsampling the second input image by the second downsampling module improves the quality of the input image of the downsampling module cascaded with the second downsampling module.

[0035] In certain embodiments, the input image further includes a third input image, and the encoder further includes a third downsampling module cascaded with the second downsampling module, the third downsampling module being configured to downsample the third input image to determine a third feature image. The third input image is determined based on the second feature image and a second brightness enhancement map, the second brightness enhancement map is obtained by downsampling the brightness enhancement map, the second feature image and the second brightness enhancement map have the same size, and the third feature image has twice the number of channels as the third input image.

[0036] In this way, matrix dot multiplication of the second feature image and the second luminance image improves the luminance distribution of the downsampled object, enhancing features in dark regions and suppressing features in bright regions. Downsampling the third input image by the third downsampling module yields a third feature image with enhanced luminance, exhibiting the characteristics of primary, secondary, and tertiary downsampling. Thus, downsampling the third input image by the third downsampling module improves the quality of the input image of the downsampling module cascaded with the third downsampling module.

[0037] In some embodiments, the input image also includes a fourth input image, and the encoder also includes a fourth downsampling module cascaded with the third downsampling module, and the fourth downsampling module is configured to downsample the fourth input image to determine a fourth feature image, wherein the fourth input image is determined based on the third feature image and a third brightness enhancement map, and the third brightness enhancement map is obtained by downsampling the brightness enhancement map, the third feature map and the third brightness enhancement map have the same size, and the number of channels of the fourth feature image is the same as that of the fourth input image.

[0038] In this way, matrix dot multiplication of the third feature image and the third luminance image improves the luminance distribution of the downsampled object, enhancing the features in dark regions and suppressing the features in bright regions. Downsampling the fourth input image by the fourth downsampling module yields a fourth feature image with the characteristics of one, two, three, or four downsampling steps, and enhanced luminance. Thus, downsampling the fourth input image by the fourth downsampling module improves the quality of the input image of the downsampling module cascaded with the fourth downsampling module.

[0039] In some embodiments, the output image includes a first output image, the decoder includes a first upsampling module cascaded with the fourth downsampling module, and the first upsampling module is configured to perform upsampling processing on the first output image to determine a first luminance image, wherein the first output image is determined based on the fusion of the fourth feature image and the third feature image, the number of channels of the first output image is the sum of the number of channels of the fourth feature image and the third feature image, and the number of channels of the first luminance image is 1 / 2 of the first output image.

[0040] In this way, the fourth feature image and the third feature image undergo feature stitching processing, merging their different features. The first upsampling module then upsamples the first output image, obtaining a first target image with fused features and natural brightness enhancement. Thus, the first upsampling module upsamples the first output image, integrating multi-scale features, resulting in a more natural brightness enhancement for the first target image and improving the quality of the input image to the upsampling module cascaded with the first upsampling module.

[0041] In some embodiments, the output image also includes a second output image, and the decoder also includes a second upsampling module cascaded with the first upsampling module, and the second upsampling module is configured to perform upsampling processing on the second output image to determine a second brightness image, wherein the second output image is determined based on the fusion of the second feature image and the first brightness image, the number of channels of the second output image is the sum of the number of channels of the second feature image and the first brightness image, and the number of channels of the second brightness image is 1 / 2 of the second output image.

[0042] In this way, the second target image and the second feature image undergo feature stitching, merging their different features. The second upsampling module then upsamples the second output image to obtain a second target image with fused features and naturally enhanced brightness. Thus, the second upsampling module upsamples the second output image, integrating multi-scale features, resulting in a more natural brightness enhancement for the second target image and improving the quality of the input image to the upsampling module cascaded with the second upsampling module.

[0043] In some embodiments, the output image also includes a third output image, and the decoder also includes a third upsampling module cascaded with the second upsampling module, and the third upsampling module is configured to perform upsampling processing on the third output image to determine a third luminance image, wherein the third output image is determined based on the fusion of the first feature image and the second luminance image, the number of channels of the third output image is the sum of the number of channels of the first feature image and the second luminance image, and the number of channels of the third luminance image is 1 / 2 of the third output image.

[0044] In this way, the third feature image and the second brightness image undergo feature stitching, merging their different features. The third upsampling module then upsamples the third output image to obtain a third target image with fused features and natural brightness enhancement. This upsampling of the third output image by the third upsampling module fuses multi-scale features, resulting in a more natural brightness enhancement of the second target image and improving the quality of the input image to the upsampling module cascaded with the third upsampling module.

[0045] In some embodiments, the output image also includes a fourth output image, and the decoder also includes a channel compression module cascaded with a third upsampling module, and the channel compression module is configured to compress the number of channels of the third brightness image to determine the target brightness image, wherein the number of channels of the target brightness image is adapted to the number of channels of the target detection head.

[0046] Thus, the channel compression module is configured to perform channel compression processing on the fourth output image to obtain a target brightness image that incorporates multi-scale features. Thus, through processing by the channel compression module, the target brightness image incorporates multi-scale layer features, naturally enhances brightness, and compresses the number of channels to facilitate matching the number of channels in the cascaded target header.

[0047] In certain embodiments, the brightness enhancement network is trained by the following steps:

[0048] Fixedly configuring the weight of the target detection head according to a preset data set;

[0049] Randomly initializing the weights of the brightness enhancement network;

[0050] Based on the training loss function of the object detection head, the brightness enhancement network is trained according to a preset training set to determine the weights and bias values ​​of the brightness enhancement network.

[0051] In this way, by training the brightness enhancement network based on the object detection head's training loss function, its weights and bias values ​​are adjusted to generate images that are most conducive to object detection. This optimizes the brightness enhancement network, thereby improving image quality and detectability. Furthermore, because the brightness enhancement network is trained based on the object detection loss function, the enhanced images it generates can more accurately locate and identify objects, improving the accuracy and reliability of object detection. This entire process achieves a close integration and collaborative work between the brightness enhancement network and the object detection head.

[0052] An embodiment of the present application provides a detection model, including the above-mentioned brightness enhancement network or a target detection head cascaded with the brightness enhancement network, to implement the steps of the above-mentioned method.

[0053] An embodiment of the present application provides a vehicle, including the above-mentioned brightness enhancement network or target detection model, to implement the steps of the above-mentioned method.

[0054] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0055] An embodiment of the present application further provides a vehicle, comprising the above-mentioned vehicle target detection device and electronic device, to implement the steps of the above-mentioned method.

[0056] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by one or more processors, the steps of the above method are implemented.

[0057] An embodiment of the present application provides a computer program product, including a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0058] The electronic device, vehicle, computer-readable storage medium, and computer program product provided by the embodiments of the present application, when the vehicle is in a dark environment, obtain a brightness enhancement map by performing color space conversion processing and calculation on the environmental image. The brightness enhancement network is guided by the brightness enhancement map to enhance the brightness of the environmental image and perform target detection, which can effectively improve the accuracy and reliability of target detection results in dark environments.

[0059] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0061] Figure 1 This is one of the flowcharts of the target detection method for a vehicle according to certain embodiments of the present application;

[0062] Figure 2 is a schematic diagram of the structure of a brightness enhancement network in certain embodiments of the present application;

[0063] FIG3( a ) is a schematic diagram of the structure of a first downsampling module in an encoder of a brightness enhancement network according to certain embodiments of the present application;

[0064] FIG3( b ) is a schematic diagram of the structure of a second downsampling module in an encoder of a brightness enhancement network according to certain embodiments of the present application;

[0065] FIG3( c ) is a schematic diagram of the structure of the first downsampling module in the encoder of the brightness enhancement network according to certain embodiments of the present application;

[0066] FIG3( d ) is a schematic diagram of the structure of the first downsampling module in the encoder of the brightness enhancement network according to certain embodiments of the present application;

[0067] FIG3( e ) is a schematic diagram of the structure of the first upsampling module in the decoder of the brightness enhancement network according to certain embodiments of the present application;

[0068] FIG3( f ) is a schematic diagram of the structure of a second upsampling module in a decoder of a brightness enhancement network according to certain embodiments of the present application;

[0069] FIG3( g ) is a schematic diagram of the structure of the third upsampling module in the decoder of the brightness enhancement network according to certain embodiments of the present application;

[0070] FIG3( h ) is a schematic diagram of the structure of a channel compression module in a decoder of a brightness enhancement network according to certain embodiments of the present application;

[0071] Figure 4 Schematic diagram of the effect of matrix dot multiplication processing in certain embodiments of the present application;

[0072] Figure 5 is a schematic diagram of the structure of a brightness enhancement network in certain embodiments of the present application;

[0073] Figure 6 1 is a schematic diagram of a training process of a brightness enhancement network according to certain embodiments of the present application;

[0074] Figure 7 This is a second flow chart of a method for detecting an object in a vehicle according to certain embodiments of the present application;

[0075] Figure 8 This is a third flow chart of a method for detecting an object in a vehicle according to certain embodiments of the present application;

[0076] Figure 9 This is a fourth flow chart of a method for detecting an object in a vehicle according to certain embodiments of the present application;

[0077] Figure 10 This is a fifth flow chart of a method for detecting an object in a vehicle according to certain embodiments of the present application;

[0078] Figure 11 This is the sixth flow chart of a method for detecting an object in a vehicle according to certain embodiments of the present application;

[0079] Figure 12 This is the seventh flow chart of the target detection method for a vehicle in certain embodiments of the present application. DETAILED DESCRIPTION

[0080] The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and should not be understood as limiting the embodiments of the present application.

[0081] In related technologies, the target detection mode in the self-driving vehicle perception system is to detect the vehicle's surrounding environment, obtain surrounding scene and obstacle information, perceive the vehicle's surrounding environment, and thus make safe and reasonable driving decisions.

[0082] Target detection modes typically integrate multiple technologies, such as the YOLO family of algorithms, which can quickly detect real-time scenes. However, in current autonomous vehicle perception technology, a common problem faced by target detection modes is that low-light environments can affect the brightness, contrast, noise level, and other characteristic information of the detected environmental image, thereby reducing image quality and the recognizability of the target object, resulting in reduced accuracy and reliability of the target detection results. The alternation of day and night and weather changes in the natural environment inevitably place vehicles in low-light environments. Therefore, when the vehicle is in low-light environments, such as at night or in rainy weather, the environmental information is reduced due to the decrease in ambient brightness, which may weaken the detailed features of the target object due to insufficient ambient brightness and may also blur the boundaries of the target object. These factors may be combined to make it difficult for the target detection model to accurately identify the target object, or even result in misidentification, significantly reducing the accuracy and reliability of the target detection results.

[0083] Based on the above questions, please refer to Figure 1 , an embodiment of the present application provides a target detection method for a vehicle, the method comprising:

[0084] 01: Determine the brightness enhancement map based on the acquired environment image;

[0085] 02: Based on the pre-trained brightness enhancement network, target detection is performed according to the environment image and brightness enhancement map.

[0086] Embodiments of the present application provide a target detection device for a vehicle. The target detection method of the embodiments of the present application can be implemented by the target detection device of the embodiments of the present application. Specifically, the target detection device includes a brightness enhancement map determination module and a target detection module. The brightness enhancement map determination module is configured to determine a brightness enhancement image based on an acquired environmental image. The target detection module is configured to perform target detection based on the environmental image and the brightness enhancement image, based on a pre-trained brightness enhancement network.

[0087] The present application also provides an electronic device comprising a memory and a processor. The vehicle object detection method of the present application can be implemented by the electronic device of the present application. Specifically, the memory stores a computer program, and the processor is configured to determine a brightness enhancement map based on a captured environmental image; and to perform object detection based on the environmental image and the brightness enhancement map based on a pretrained brightness enhancement network.

[0088] Specifically, the environmental image in the embodiments of the present application refers to an image of the environment in which the vehicle is located, and the environmental image can be obtained by an image acquisition device such as an external camera of the vehicle. The embodiments of the present application are mainly aimed at target detection when the vehicle is in a dark light environment. In a dark light environment, the environmental image is the acquired dark light image, in which the object to be detected has defects such as low contrast and blurred outline due to reduced brightness. It can be understood that for such an environmental image, it is usually difficult to effectively identify the object to be detected by directly using a target detection head for detection.

[0089] The brightness enhancement map of the embodiment of the present application can be obtained by performing color space conversion and other processing on the environmental image. The brightness enhancement map is used to characterize the brightness information of the environmental image, wherein the closer the pixel brightness value in the brightness enhancement map is to 1, the darker the pixel of the corresponding environmental image is, and the closer it is to 0, the brighter the pixel of the corresponding environmental image is. Therefore, in the process of brightness enhancement processing, it is possible to clearly define the dark areas of the environmental image that need to be enhanced, increase the pixel brightness weights of the dark areas, thereby strengthening the outline and detail features of the objects to be detected (such as pedestrians at night, vehicles in the shadows) in the dark light environment, reducing the situation where the target objects are not recognized or are misidentified due to insufficient brightness, and improving the accuracy and reliability of the target detection results.

[0090] The brightness enhancement network is pre-trained. In the embodiments of the present application, the brightness enhancement map serves as the input or intermediate feature of the brightness enhancement network, informing the network of the dark areas that need to be enhanced. The brightness of the image to be detected is enhanced by the brightness enhancement network, thereby improving the accuracy and reliability of the target detection head cascaded with the network.

[0091] In practice, the environment image can first be processed, such as through color space conversion, to generate a brightness enhancement map representing the brightness of the environment image. This brightness enhancement map extracts and processes only the brightness channel, preserving information such as the hue and saturation of the environment image. This prevents color distortion (such as hue deviation and reduced saturation) during the enhancement process. During the brightness enhancement process, only the brightness of dark areas of the environment image is enhanced, meeting the input requirements of the target detection head in target detection tasks.

[0092] Then, based on the brightness enhancement network, the brightness enhancement map assists in enhancing the brightness of the environment image. Finally, target detection is performed based on the enhancement results of the brightness enhancement map and the environment image.

[0093] In summary, in the target detection method for a vehicle, the target detection device for a vehicle, and the electronic device provided in the embodiments of the present application, when the vehicle is in a dark environment, a brightness enhancement map representing brightness information is obtained through the environmental image. In the process of using the brightness enhancement network to enhance the environmental image, the brightness enhancement map can assist in enhancing the brightness of the environmental image. Specifically, it can assist in brightness enhancement of the dark areas of the environmental image. In this way, when performing target detection, compared to directly performing target detection on the dark environment image, in the embodiments of the present application, enhancing the brightness of the environmental image based on the brightness enhancement network can improve the accuracy and reliability of the target detection results, thereby perceiving the vehicle's surrounding environment and making safe and reasonable driving decisions based on the detection results.

[0094] See also Figure 2 The embodiment of the present application further provides a brightness enhancement network 100. The object detection method of the embodiment of the present application can be implemented based on the brightness enhancement network of the embodiment of the present application. Specifically, the brightness enhancement network 100 of the embodiment of the present application includes an encoder 110 and a decoder 120.

[0095] The encoder 110 is configured to perform downsampling processing on an input image to determine an output image, wherein the input image is determined based on an environment image and a brightness enhancement map, and the brightness enhancement map is determined based on the environment image.

[0096] The decoder 120 is configured to perform upsampling processing on the output image and determine a target brightness image so that the target detection head of the brightness enhancement network cascade performs target detection based on the target brightness map.

[0097] Specifically, the input image refers to the object to be processed that is input to the corresponding downsampling module of the encoder 110 during the downsampling process performed by the encoder 110. The downsampling process can extract features of the input image and reduce the size of the input image. In this way, the brightness of the ambient image in a low-light environment can be adjusted and optimized after downsampling, so that the overall brightness of the processed image is improved. For example, the overall brightness of the image can be increased and the details of objects in the image can be clearer.

[0098] The output image refers to the image obtained by processing the corresponding downsampling module of encoder 110 during the downsampling process in encoder 110. The output image also serves as the object to be processed by the corresponding upsampling module in decoder 120 cascaded with encoder 110. The upsampling process can integrate the processing information of the output image and restore the output image size. In this way, the output image can be enhanced with richer details and accurate brightness after upsampling.

[0099] The target brightness image is processed by the brightness enhancement network and is input to the target detection head for target detection. The target brightness image makes the details and outlines of the target object more obvious.

[0100] In this way, the brightness enhancement map and the environment image serve as the input images of the brightness enhancement network. With the assistance of the brightness enhancement map, the brightness enhancement network performs brightness enhancement processing on the dark areas of the environment image to determine the target brightness image. The brightness enhancement network includes an encoder, which downsamples the input image, enhances the brightness of the environment image, and obtains an output image. The brightness enhancement network also includes a decoder cascaded with the encoder, which upsamples the output image to obtain a target brightness image with richer details and accurate brightness. In this way, compared with the low-light environment image, the overall brightness of the target brightness image obtained after processing by the brightness enhancement network is enhanced, and the outline of the target object in the image is clearer and the detailed features are more obvious. Therefore, the brightness enhancement network provides higher quality input for the cascaded target detection head, which reduces the impact of insufficient ambient brightness on the target object detection results to a certain extent, and improves the accuracy and reliability of target detection.

[0101] 3( a ), in some embodiments, the encoder 110 includes a first downsampling module 111. The first downsampling module 111 is configured to perform downsampling processing on the first input image to determine a first feature image.

[0102] The input image includes a first input image, which is determined based on the environment image and the brightness enhancement map, wherein the environment image and the brightness enhancement map have the same size, and the number of channels of the first feature image is twice that of the first input image.

[0103] Specifically, the first input image is obtained by performing matrix dot multiplication processing on the ambient image and the brightness enhancement map, and is the processing object of the first downsampling module 111. The first feature image is the image obtained by downsampling the first input image, and contains feature information in the ambient image processed by the first downsampling module 111. In the image enhancement network, the ambient image and the brightness enhancement map are multiplied pixel by pixel to perform matrix dot multiplication processing. The matrix dot multiplication processing can amplify the brightness of the dark pixels in the ambient image after the dot multiplication processing, which is improved compared to the original brightness. The brightness of the bright pixels changes little after the dot multiplication processing, and their brightness may be sufficient in itself in the actual ambient image, and there is no need for excessive enhancement.

[0104] In one example, the specific structure of the first sampling module 111 includes a normalization layer (BN layer), a convolution layer (conv layer + LeakyRule layer + BN layer) and a pooling layer (max pooling layer), which is used to downsample the first input image and determine the first feature image. Among them, the first input image is obtained by matrix dot multiplication of the environment image and the brightness enhancement map. The size of the objects subjected to the matrix dot multiplication should be the same. Therefore, the size of the environment image and the brightness enhancement map is the same. The matrix dot multiplication can highlight the dark area features and improve the brightness distribution. The BN layer normalizes the first input image to make the training process more stable. The convolution layer performs convolution on the normalized result to extract features of the first input image such as the texture of the target object, and at the same time increases the number of image channels based on a convolution kernel of size 3*3. The pooling layer performs pooling on the convolution result to compress the size of the first input image to obtain the first feature image, halve the size, reduce the spatial resolution, and reduce the amount of calculation.

[0105] See also Figure 4 In one example, the figure shows an environment image, a brightness enhancement map, and a first input image determined after a matrix dot multiplication of the two. It can be seen that the environment image is an image of a dark environment, and the overall brightness of the image is low, so the vehicle outline in the image is blurred and almost merged with the background. In the brightness enhancement map corresponding to the environment image, the dark areas, such as the area where the vehicle is located and the vehicle background area in the figure, have high brightness values ​​(close to 1), and the bright areas (the area where the headlights are located and the sky area) have low brightness values ​​(close to 0). In the enhanced first input image, the overall brightness of the image is improved, especially the brightness of the dark areas (the area where the vehicle is located) is enhanced, and the bright areas, such as the area where the headlights are located and the sky area in the figure, are suppressed, wherein the vehicle details are clearer.

[0106] Regarding the changes in the size and number of channels of the environment image, the brightness enhancement map, and the corresponding feature image, in one example, the environment image has a size of 512*640 and a number of channels of 3, and the brightness enhancement map has a size of 512*640 and a number of channels of 3. After matrix dot multiplication of the environment image and the brightness enhancement map, they are processed by the first downsampling module 111 to obtain a first feature image. The first feature image has a size of 256*320 and a number of channels of 6. The first downsampling module 111 initially extracts low-level features such as edges and textures, and increases the number of channels to 6 in preparation for subsequent multi-scale processing.

[0107] In this way, matrix dot multiplication of the ambient image and the brightness enhancement map improves the brightness distribution of the downsampled object, making the dark region features prominent and the bright region features suppressed. The first downsampling module 111 downsamples the first input image, obtaining a first feature image with the features of the first downsampling process and enhanced brightness. Thus, the downsampling of the first input image by the first downsampling module 111 improves the quality of the input images of the downsampling modules cascaded with the first downsampling module 111.

[0108] 3( b ), in some embodiments, the encoder 110 includes a second downsampling module 112. The second downsampling module 112 is configured to perform downsampling processing on the second input image to determine a second feature image.

[0109] The input image includes a second input image, which is determined based on the first feature image and the first brightness enhancement map. The second downsampling module is cascaded to the first downsampling module. The first brightness enhancement map is determined by downsampling the brightness enhancement map to reduce its size. The first feature image and the first brightness enhancement map have the same size, and the second feature image has twice the number of channels as the second input image.

[0110] Specifically, the second input image is the result of matrix dot multiplication of the first feature image and the first brightness enhancement map, and is the processing object of the second downsampling module 112. The second feature image is the image obtained by downsampling the second input image, and contains feature information of the upper scale and the scale of the layer.

[0111] The specific structure of the second downsampling module 112 includes a normalization layer (BN layer), a convolution layer (conv layer + LeakyRule layer + BN layer) and a pooling layer (max pooling layer), which is used to downsample the matrix dot product result of the first feature image and the first brightness enhancement map to determine the second feature image. Among them, the second input image is the result of the matrix dot product processing of the first feature image and the first brightness enhancement map. The object size of the matrix dot product should be the same, so the size of the first feature image and the first brightness enhancement map should be the same. The matrix dot product can highlight the dark area features and improve the brightness distribution. The BN layer normalizes the dot product result to make the training process more stable. The convolution layer performs convolution processing on the normalized result to extract the features of the second input image such as the edge details of the target object, and at the same time enhances the image brightness based on the convolution kernel rule. The pooling layer performs pooling on the convolution result to compress the size of the second input image and determine the second feature image.

[0112] Continuing with the previous example, the first feature image has a size of 256*320 and 6 channels, and the first luminance image has a size of 256*320 and 6 channels. The first luminance image is determined by downsampling the luminance-enhanced image. The matrix dot product of the first feature image and the first luminance-enhanced image is processed by the second downsampling module 112 to output a second feature image. The second feature image has a size of 128*160 and 12 channels. The second downsampling module 112 further extracts mid-level features, doubling the number of channels to 12, thereby increasing feature complexity.

[0113] In this way, matrix dot multiplication of the first feature image and the first brightness enhancement map can improve the brightness distribution of the downsampled object, enhancing the features in dark areas and suppressing the features in bright areas. Downsampling the second input image by the second downsampling module yields a second feature image that has the features of both the first and second downsampling processes and enhanced brightness. Thus, downsampling the second input image by the second downsampling module 112 improves the quality of the input images of the downsampling modules cascaded with the second downsampling module 112.

[0114] 3( c ), in some embodiments, the encoder 110 includes a third downsampling module 113. The third downsampling module 113 is configured to perform downsampling processing on the third input image to determine a third feature image.

[0115] The input image includes a third input image, which is determined based on the second feature image and the second brightness enhancement map. The third downsampling module is cascaded to the second downsampling module. The second brightness enhancement map is determined by downsampling the brightness enhancement map to reduce its size. The second feature image and the second brightness enhancement map have the same size, and the third feature image has twice the number of channels as the third input image.

[0116] Specifically, the third input image is the result of matrix dot multiplication of the second feature image and the second brightness enhancement image, and is the processing object of the third downsampling module 113. The third feature image is the image obtained by downsampling the third input image, and contains feature information at three levels.

[0117] The specific structure of the third downsampling module 113 includes a normalization layer (BN layer), a convolution layer (conv layer + LeakyRule layer + BN layer) and a pooling layer (max pooling layer), which is used to downsample the third input image and determine the third feature image. Among them, the third input image is the result of the matrix dot multiplication of the second feature image and the second brightness enhancement map. The object size of the matrix dot multiplication should be the same, so the second feature image and the second brightness enhancement map should be the same. The dot multiplication can highlight the dark area features and improve the brightness distribution. The BN layer performs normalization processing on the third input image to make the training more stable. The convolution layer performs convolution processing on the normalized result to extract the third input image features such as other features of the target object, and enhances the image brightness based on the convolution kernel rule. The pooling layer performs pooling on the convolution result to compress the size of the third input image and obtain the third feature image.

[0118] Continuing with the previous example, the second feature image has a size of 128*160 and 12 channels, and the second luminance image has a size of 128*160 and 12 channels. The matrix dot product of the second feature image and the second luminance enhancement image is processed by the third downsampling module 113 to output a third feature map. The third feature map has a size of 64*80 and 24 channels. The third downsampling module 113 extracts high-level abstract features, and the number of channels is increased to 24.

[0119] Thus, in the third downsampling module 113, the matrix dot product of the second feature image and the second luminance image improves the luminance distribution of the downsampled object, thereby highlighting the features in the dark regions and suppressing the features in the bright regions. Through the downsampling process of the third input image by the third downsampling module 113, a third feature image is obtained that possesses the features of the first, second, and third downsampling processes, but with enhanced luminance. Thus, through the downsampling process of the third input image by the third downsampling module 113, the quality of the input images of the downsampling modules cascaded with the third downsampling module 113 is improved.

[0120] 3( d ), in some embodiments, the encoder 110 includes a fourth down-sampling module 114. The fourth down-sampling module 114 is configured to perform down-sampling processing on the fourth input image to determine a fourth feature image.

[0121] The input image includes a fourth input image, which is determined based on the third feature image and the third brightness enhancement map. The fourth downsampling module is cascaded to the third downsampling module. The third brightness enhancement map is determined by downsampling the brightness enhancement map to reduce its size. The third feature image and the third brightness enhancement map have the same size, and the fourth feature image has twice the number of channels as the fourth input image.

[0122] Specifically, the fourth input image is the result of matrix dot multiplication of the third feature image and the third brightness enhancement map, and is processed by the fourth downsampling module 114. The fourth feature image is the image obtained by downsampling the fourth input image, and contains feature information at four levels.

[0123] The specific structure of the fourth downsampling module 114 includes a normalization layer (BN layer) and a convolution layer (conv layer + LeakyRule layer + BN layer), which is used to downsample the fourth input image and determine the fourth feature image. It should be noted that, unlike the first downsampling module 111, the second downsampling module 112, and the third downsampling module 113, the fourth downsampling module 114 does not have a pooling layer. It can essentially be seen as an intermediate layer connecting the downsampling stage and the upsampling stage. Among them, the fourth input image is the result of matrix dot multiplication of the third feature image and the third brightness enhancement map. The object size of the matrix dot multiplication should be the same, so the size of the third feature image and the third brightness enhancement map should be the same. The matrix dot multiplication can highlight the dark area features and improve the brightness distribution. The BN layer normalizes the fourth input image to make the training more stable. The convolution layer convolutions the normalized result to extract the fourth input image features, such as other features of the target object, while enhancing the image brightness based on the convolution kernel rule to determine the fourth input image features. The fourth downsampling module 114 receives the high-level features output by the third downsampling module 113, strengthens the feature expression through convolution operation, and provides high-level feature input of the same scale for the subsequent upsampling stage to facilitate fusion.

[0124] Continuing with the previous example, the size of the third feature image is 64*80 and the number of channels is 24. The size of the third brightness image is 64*80 and the number of channels is 24. The matrix dot product of the third feature image and the third brightness enhancement image is processed by the fourth downsampling module 114 to output a fourth feature image. The fourth feature image has a size of 64*80 and the number of channels is 24. The input and output sizes of the fourth downsampling module 114 are both 64×80, and the number of channels remains unchanged at 24. That is, the number of input and output channels is 24. The spatial resolution and channel dimension of the feature map are not changed. Only the convolutional layer performs nonlinear transformation on the features. In the corresponding scale space, high-level semantic features, such as object contours and structures, are further abstracted and enhanced to avoid feature loss or degradation during the transition stage.

[0125] Therefore, during the entire downsampling process, feature images of different scales (512×640→256×320→128×160→64×80) and the environment image input into the first downsampling module 111 are respectively multiplied with the brightness enhancement map of the corresponding scale to achieve multi-scale pixel-level enhancement. While retaining details, the brightness is enhanced to avoid overexposure or detail loss caused by single-scale enhancement, making the brightness of the enhanced image more uniform and natural.

[0126] In this way, matrix dot multiplication of the third feature image and the third luminance image can improve the luminance distribution of the downsampled object, enhancing the features in dark areas and suppressing the features in bright areas. The fourth downsampling module 114 downsamples the fourth input image, obtaining a fourth feature image with the characteristics of one, two, three, or four downsampling steps and enhanced brightness. Thus, the downsampling of the fourth input image by the fourth downsampling module 114 improves the quality of the input images of the downsampling modules cascaded with the fourth downsampling module 114.

[0127] 3( e ), in some embodiments, the decoder 120 includes a first upsampling module 121. The first upsampling module 121 is configured to perform upsampling processing on the first output image to determine a first luminance image.

[0128] The output image includes a first output image, which is determined based on the fourth characteristic image and the third characteristic image. The first upsampling module is cascaded to the fourth downsampling module. The number of channels of the first output image is the sum of the number of channels of the fourth characteristic image and the third characteristic image, and the number of channels of the first luminance image is 1 / 2 of that of the first output image.

[0129] Specifically, after downsampling, upsampling is used to restore the spatial dimensions and fuse the multi-scale features from the downsampling stage. The first output image is the result of feature concatenation of the fourth and third feature images and is processed by the first upsampling module 121. The first luminance image is the concatenation of the outputs of the fourth downsampling module 114 and the third downsampling module 113, fusing feature information at different levels. This concatenation allows the network to comprehensively utilize these two levels of feature information in subsequent processing.

[0130] The specific structure of the first upsampling module 121 includes an upsampling layer and a convolution layer (conv layer + LeakyRule layer + BN layer), which is used to upsample the first output image and determine the first brightness image. Among them, the first output image is the result of splicing the features of the fourth feature image and the third feature image. The splicing operation can fuse features and supplement detail information, which helps to improve the brightness enhancement effect and image quality. During the downsampling process of the network, as the size of the feature map continues to shrink, some detail information will inevitably be lost. By reintroducing the features of the third feature image and splicing them with the features of the fourth feature image, this lost information can be restored to a certain extent, reducing information loss, so that subsequent processing can be based on richer features. The upsampling layer performs upsampling on the feature splicing result, restores the image size, and enhances image details. The convolution layer performs convolution processing to determine the first target image, which is used to optimize features and improve image quality. The number of channels is reduced by the convolution operation, thereby adjusting the feature dimension. Appropriately reducing the number of channels can reduce computational complexity while avoiding overfitting, making the network more efficient and stable. The input image size is restored to twice its current size to prepare for further feature fusion and processing, and gradually restored to the size of the original input image.

[0131] Continuing with the previous example, the size of the fourth feature image is 64*80 and the number of channels is 24, the size of the third feature image is 64*80 and the number of channels is 24, and the result of feature splicing of the fourth feature image and the third feature image is the first output image, with a size of 64*80 and a number of channels of 24+24=48. After upsampling processing by the first upsampling module 121, the first brightness image is output, and the size of the first brightness image is 128*160 and the number of channels is (24+24) / 2=24.

[0132] In this way, the fourth feature image and the third feature image undergo feature stitching processing, merging their different features. The first upsampling module 121 upsampling the first output image yields a first target image with fused features and naturally enhanced brightness. Thus, the first upsampling module 121 upsampling the first output image fuses multi-scale features, resulting in a more natural brightness enhancement for the first target image and improving the quality of the input image to the upsampling module cascaded with the first upsampling module 121.

[0133] 3( f ), in some embodiments, the decoder 120 includes a second upsampling module 122. The second upsampling module 122 is configured to perform upsampling processing on the second output image to determine a second luminance image.

[0134] The output image includes a second output image, which is determined based on the second feature image and the first luminance image. The second upsampling module is cascaded to the first upsampling module. The number of channels of the second output image is the sum of the number of channels of the second feature image and the first luminance image, and the number of channels of the second luminance image is 1 / 2 of that of the second output image.

[0135] Specifically, the second output image is the result of feature concatenation of the second feature image and the first luminance image, and is processed by the second upsampling module 122. The second luminance image combines the features output by the first upsampling module 121 with the features output by the second downsampling module 112 during the downsampling phase. This concatenation integrates features at different scales, enabling the network to comprehensively utilize features at different scales to better capture both the overall structure and local details of the image. The second feature image contains rich image details. Through concatenation and subsequent processing, it helps to restore details lost during the previous downsampling process, improving image clarity and quality.

[0136] The specific structure of the second upsampling module 122 includes an upsampling layer and a convolution layer (conv layer + LeakyRule layer + BN layer), which is used to upsample the second output image and determine the second brightness image. Among them, the second output image refers to the feature splicing result of the second feature image and the first brightness image. The feature splicing operation can fuse features and supplement detail information. The splicing operation enables features at different levels to be further fused. The upsampling layer upsamples the splicing result, restores the image size, and enhances image details. Specifically, the size of the spliced ​​input image is restored to the same size as after downsampling by the first downsampling module 111, providing a suitable spatial dimension for subsequent feature fusion. The convolution layer performs convolution processing to determine the second brightness image, which is used to optimize features, improve image quality, and reduce the number of channels.

[0137] Continuing with the previous example, the second feature image has a size of 128*160 and 12 channels, and the first luminance image has a size of 128*160 and 24 channels. The result of feature concatenation of the second feature image and the first luminance image is a second output image with a size of 128*160 and a number of channels of 24+12=36. After upsampling by the second upsampling module 122, the second luminance image is output. The size of the second luminance image is 256*320 and the number of channels is (24+12) / 2=18. Reducing the number of channels from 36 to 18 reduces computational complexity, avoids overfitting, and makes the network more efficient. At the same time, the upsampling operation expands the spatial size of the feature map, preparing for subsequent feature fusion and final output.

[0138] In this way, the second target image and the second feature image undergo feature stitching processing, merging their different features. The second upsampling module 122 upsamples the second output image to obtain a second target image with fused features and naturally enhanced brightness. Thus, the second upsampling module 122 upsamples the second output image, integrating multi-scale features, resulting in a more natural brightness enhancement for the second target image and improving the quality of the input image to the upsampling module cascaded with the second upsampling module 122.

[0139] 3( g ), in some embodiments, the decoder 120 includes a third upsampling module 123. The third upsampling module 123 is configured to perform upsampling processing on the third output image to determine a third luminance image.

[0140] The output image includes a third output image, which is determined based on the first feature image and the second luminance image. The third upsampling module is cascaded to the second upsampling module. The number of channels of the third output image is the sum of the number of channels of the first feature image and the second luminance image, and the number of channels of the third luminance image is 1 / 2 of that of the third output image.

[0141] Specifically, the third output image is the result of feature concatenation of the first feature image and the second luminance image, and is processed by the third upsampling module 123. The third luminance image concatenates the features output by the second upsampling module 122 with the features output by the first downsampling module 111 during the downsampling phase. The features of the second luminance image contain mid- and high-level feature information after multi-layer processing and fusion, while the features of the first feature image retain the original underlying features of the image, such as edges, textures, and other detailed information. This concatenation operation integrates feature information from different levels, providing a richer feature representation for subsequent processing.

[0142] The specific structure of the third upsampling module 123 includes an upsampling layer and a convolution layer (conv layer + LeakyRule layer + BN layer), which is used to upsample the third output image and determine the third brightness image. Among them, the third output image refers to the feature splicing result of the first feature image and the second brightness image. The splicing operation can fuse features and supplement detail information. The splicing operation enables features at different levels to be further integrated. The upsampling layer refers to upsampling the splicing result, restoring the image size, and enhancing image details. Specifically, the size of the spliced ​​input image is restored to the same size as the environment image. The convolution layer performs convolution processing to determine the third brightness image, which is used to optimize features, improve image quality, and reduce the number of channels.

[0143] Continuing with the previous example, the size of the first feature image is 256*320 and the number of channels is 18, the size of the second luminance image is 256*320 and the number of channels is 6, and the result of feature concatenation of the first feature image and the second luminance image is the third output image, which has a size of 256*320 and a number of channels of 18+6=24. After upsampling by the third upsampling module 123, the third luminance image is output. The size of the third luminance image is 512*640 and the number of channels is (18+6) / 2=12. The number of channels is reduced from 24 to 12, which reduces the dimension of the features and the amount of computation while avoiding overfitting, making the network more efficient and stable. At the same time, the upsampling operation restores the spatial size of the feature map to the same size as the original input image, preparing for the output of the final target luminance image.

[0144] In this way, the third feature image and the second brightness image undergo feature stitching processing, merging their different features. The third upsampling module 123 upsampling the third output image yields a third target image with fused features and naturally enhanced brightness. Thus, the upsampling module 123 upsampling the third output image fuses multi-scale features, resulting in a more natural brightness enhancement for the second target image and improving the quality of the input image to the upsampling module cascaded with the third upsampling module 123.

[0145] 3(h), in some embodiments, the decoder 120 includes a channel compression module 124. The channel compression module 124 is configured to compress the number of channels of the third luminance image to determine a target luminance image.

[0146] The output image includes a fourth output image, which is determined based on the third luminance image. The number of channels of the target luminance image is half that of the fourth output image, and the number of channels of the target luminance image is adapted to the number of channels of the target detection head. The channel compression module is cascaded to the third upsampling module.

[0147] Specifically, the fourth output image is the third luminance image, which is the processing target of the channel compression module 124. The target luminance image refers to an image that has undergone channel compression.

[0148] The specific structure of the channel compression module 124 includes a convolutional layer (conv layer + LeakyRule layer + BN layer), which is used to perform convolution processing on the third brightness image to determine the target brightness image. Among them, the convolutional layer performs convolution processing on the third brightness image to determine the target brightness image, which is used to compress the channel and adapt to the cascaded target detection head channel. The target brightness image is the feature map obtained by the previous multi-layer processing and feature fusion converted into the final RGB image after brightness enhancement, and the output meets the target detection requirements, completing the task of the entire brightness enhancement network.

[0149] In one example, channel compression module 124 includes two cascaded compression submodules. In other words, channel compression module 124 includes two consecutive convolutional layers. Using multiple convolutional layers for progressive compression effectively preserves brightness features and avoids detail loss caused by a sudden decrease in the number of channels.

[0150] Continuing with the above example, the third luminance image has a size of 512*640 and a number of channels of 12. After passing through the first compression submodule of the channel compression module 124, the output image has a size of 512*640 and a number of channels of 12 / 2=6. After passing through the second compression submodule, the target luminance image is output. The target luminance image has a size of 512*640 and a number of channels of 6 / 2=3.

[0151] Thus, through cross-layer stitching, a complete feature set is constructed from bottom-level edges to higher layers, ensuring that the brightness of objects of all scales in dark environments is appropriately enhanced. Each step integrates downsampled features of the corresponding scale to avoid blurring of details caused by direct upsampling. Furthermore, by compressing the number of channels at each layer, computational complexity is minimized while ensuring feature validity, making it suitable for real-time deployment in vehicles.

[0152] Thus, the channel compression module 124 is configured to perform channel compression processing on the fourth output image to obtain a target brightness image that incorporates multi-scale features. Thus, through processing by the channel compression module 124, the target brightness image incorporates multi-scale layer features, naturally enhances brightness, and compresses the number of channels to facilitate matching the number of channels in the cascaded target header.

[0153] See also Figure 5 , the following is a complete example to illustrate the processing flow of the brightness enhancement network:

[0154] Assuming that the size of the ambient image is 512*640 and the number of channels is 3, the size of the corresponding brightness enhancement map is 512*640 and the number of channels is 3, the matrix dot product result of the ambient image and the brightness enhancement map is processed by the first downsampling module 111, and the first feature image can be output. The size of the first feature image is 256*320 and the number of channels is 6.

[0155] Next, the size of the first brightness image is 256*320 and the number of channels is 6. The first brightness image is determined by downsampling the brightness enhancement image. The matrix dot product result of the first feature image and the first brightness enhancement image is processed by the second downsampling module 112 to output the second feature image. The size of the second feature image is 128*160 and the number of channels is 12.

[0156] The second luminance image has a size of 128*160 and a number of channels of 12. The matrix dot product of the second feature image and the second luminance enhancement map is processed by the third downsampling module 113 to output a third feature map. The third feature map has a size of 64*80 and a number of channels of 24. The third luminance image has a size of 64*80 and a number of channels of 24. The matrix dot product of the third feature image and the third luminance enhancement map is processed by the fourth downsampling module 114 to output a fourth feature image. The fourth feature image has a size of 64*80 and a number of channels of 24.

[0157] The feature splicing result of the fourth feature image and the third feature image is processed by the first upsampling module 121 to output a first brightness image. The size of the first brightness image is 128*160 and the number of channels is (24+24) / 2=24.

[0158] The feature splicing result of the first feature image and the second brightness image is processed by the third upsampling module 123 to output a third brightness image. The size of the third brightness image is 512*640, and the number of channels is (18+6) / 2=12.

[0159] Finally, the target brightness image is output through the channel compression module 124. The size of the target brightness image is 512*640, and the number of channels is 12 / 2 / 2=3.

[0160] The following Table 1 shows the specific data of the input and output of each layer module.

[0161] Table 1

[0162]

[0163] See also Figure 6 In some embodiments, the brightness enhancement network 100 is trained by the following steps:

[0164] 0101: Fixedly configure the weight of the target detection head according to the preset data set;

[0165] 0102: Randomly initialize the weights of the brightness enhancement network;

[0166] 0103: Based on the training loss function of the object detection head, the brightness enhancement network is trained according to the preset training set to determine the weights and bias values ​​of the brightness enhancement network.

[0167] Specifically, for the brightness enhancement network, random initialization allows the model to automatically learn appropriate parameter values ​​based on the input data during training. It is understandable that since this image brightness enhancement network is targeted at a specific dark environment image enhancement task, the network is initially unaware of the feature distribution of the data and the task requirements. Random initialization provides a starting point, allowing the model to gradually adjust parameters during training to adapt to the task. The initial weights of the image brightness enhancement network can be set to random numbers between 0 and 1. The setting of the initial weights will affect the convergence speed and final performance of the model. Randomly initializing the weights to values ​​between 0 and 1 allows the model to have a diverse initial state at the beginning of training, avoiding the same initial output for all neurons, thereby providing the model with the possibility of learning different features. Moreover, the range of 0-1 is relatively reasonable, and will not make the weights too large to cause gradient explosion, nor too small to cause gradient disappearance.

[0168] The object detection head can be trained based on a pre-set dataset to determine its weights. For example, using YOLOv8n as the object detection head, the weights officially trained on the COCO dataset can be used during training. The COCO dataset is a large-scale dataset for object detection, segmentation, and image captioning, containing a rich variety of images and object categories. YOLOv8n is an object detection model. The COCO dataset is a large-scale dataset for object detection, segmentation, and image captioning, containing a large number of images and object categories. Officials have conducted extensive and thorough training on this dataset, enabling the model to learn many common object features and detection patterns. Using these pre-trained weights allows YOLOv8n to quickly leverage this knowledge in new tasks, avoiding the significant time and computational resources required for training from scratch. This also helps the model converge faster and achieve better performance. It should be noted that in other examples, the object detection head can also be another model used for object detection. Its cascade with the brightness enhancement network and training methods are essentially the same as in this example. The choice of method can be tailored to your specific needs and will not be detailed here.

[0169] During training, the model's ultimate application scenario is dark environment object detection, so a dark environment dataset is used for training. Dark environment images, characterized by low lighting and contrast, differ significantly from images in normal lighting environments. Using a dark environment dataset allows the image brightness enhancement network to specifically learn how to effectively enhance the brightness of these images, thereby improving the accuracy of subsequent object detection.

[0170] In the embodiments of the present application, during the training process, the parameters of YOLOv8n can be fixed, and only the weights of the image brightness enhancement network are trained, that is, the weights of YOLOv8n are not updated. It is understandable that YOLOv8n is a relatively complex model with a large number of parameters. If the image brightness enhancement network and YOLOv8n are trained simultaneously, the amount of computation will increase significantly and the training time will be significantly extended. Fixing the parameters of YOLOv8n can reduce the computing resources and time required for training.

[0171] Furthermore, the image enhancement network's primary task is to improve image brightness and quality, while YOLOv8n's task is to detect objects. Initially, training the image enhancement network is focused on learning how to generate high-quality enhanced images. This prevents YOLOv8n parameter updates from interfering with the image enhancement results during training.

[0172] Furthermore, as mentioned above, since YOLOv8n has been extensively trained on the COCO dataset, its weights already incorporate a lot of useful object detection knowledge. Fixing these parameters ensures that when training the image brightness enhancement network, we can leverage YOLOv8n's existing detection capabilities while avoiding destroying this learned knowledge during training.

[0173] The goal of training is to enable the image enhancement network to convert dark images into images with appropriate brightness and good contrast, enabling subsequent YOLOv8n to more accurately detect objects. During training, the parameters of the image enhancement network are continuously adjusted so that the enhanced images achieve better detection results (such as higher precision and recall) after YOLOv8n detection. YOLOv8n's training loss function measures the difference between the enhanced image and the expected image, thereby measuring the performance of the image enhancement network. This loss is then used to train the network and update its parameters. Through this training strategy, the image enhancement network gradually learns how to effectively enhance dark images without affecting YOLOv8n's existing detection capabilities. Once the image enhancement network converges, the enhanced images it outputs provide better input to YOLOv8n, thereby improving the overall model's object detection performance in dark environments.

[0174] In this way, by training the brightness enhancement network based on the object detection head's training loss function, its weights and bias values ​​are adjusted to generate images that are most conducive to object detection. This optimizes the brightness enhancement network, thereby improving image quality and detectability. Furthermore, because the brightness enhancement network is trained based on the object detection loss function, the enhanced images it generates can more accurately locate and identify objects, improving the accuracy and reliability of object detection. This entire process achieves a close integration and collaborative work between the brightness enhancement network and the object detection head.

[0175] See also Figure 7 In some embodiments, step 01 (determining a brightness enhancement map based on the acquired environment image) includes:

[0176] 011: When the ambient brightness of the environment where the vehicle is located is less than or equal to a first threshold, determine a brightness enhancement map according to the environment image.

[0177] In some embodiments, the brightness enhancement map determining module is configured to determine the brightness enhancement map based on the environment image when the ambient brightness of the environment in which the vehicle is located is less than or equal to a first threshold.

[0178] In some embodiments, the processor is further configured to determine a brightness enhancement map based on the environment image when the ambient brightness of the environment in which the vehicle is located is less than or equal to a first threshold.

[0179] Specifically, the ambient light intensity is detected in real time through on-board sensors, such as illuminance sensors. When the brightness is lower than a preset threshold, such as a typical value of <10lux at night, the brightness enhancement network is triggered. The first threshold is a preset brightness value, which is a judgment condition for dynamically controlling the start and stop of the brightness enhancement network. In an example, the first threshold can be set to 10lux. When the ambient brightness is less than or equal to 10lux, it means that the vehicle is in a dark environment, and the ambient image needs to be processed according to the processor to obtain a brightness enhancement map. Under this ambient brightness condition, the brightness enhancement network needs to be started. It should be noted that the value of the first threshold is only a schematic illustration, and can be specifically determined based on the target detection results of the target detection head under different ambient brightness conditions.

[0180] In this way, when the ambient brightness of the vehicle's environment is less than or equal to the first threshold, a brightness enhancement map is determined based on the ambient image. The first ambient brightness threshold is used to distinguish between bright and dark environments. In dark environments, it is determined that brightness information enhancement of the ambient image is necessary, thereby improving the accuracy and reliability of target detection results.

[0181] See also Figure 8 In some embodiments, the target detection method further comprises:

[0182] 012: When the ambient brightness is greater than the first threshold, target detection is performed based on the ambient image.

[0183] In certain embodiments, the processor is further configured to perform target detection based on the environmental image when the ambient brightness of the vehicle's environment is greater than a first threshold. Specifically, continuing with the above example, when the ambient brightness is greater than 10 lux, indicating that the vehicle is in a normal environment, no processing of the environmental image is required, and target detection can be performed directly on the environmental image. The raw environmental image is directly input to the target detection head to avoid image distortion caused by excessive enhancement. For example, no processing is required under strong daylight.

[0184] In this way, in bright light environments, the brightness information of the ambient image can support direct target detection with better accuracy and reliability, without the need for brightness enhancement through brightness enhancement maps, etc.

[0185] See also Figure 9 In some embodiments, step 01 (determining a brightness enhancement map based on the acquired environment image) includes:

[0186] 013: When the average value of image pixels of the environment image is less than or equal to the second threshold, determine a brightness enhancement map according to the environment image.

[0187] In some embodiments, the brightness enhancement map determining module is configured to determine the brightness enhancement map based on the environment image when an average value of image pixels of the environment image is less than or equal to a second threshold.

[0188] In some embodiments, the processor is further configured to determine a brightness enhancement map based on the environmental image when an average value of image pixels of the environmental image is less than or equal to a second threshold.

[0189] Specifically, the image pixel average refers to the arithmetic mean of the brightness values ​​or color component values ​​of all pixels in the entire image. The image pixel average can intuitively reflect the overall brightness of the image. If the average is low, it means that the image is dark as a whole. If the average is high, it means that the image is bright as a whole. The second threshold refers to the preset pixel average. In some examples, the ambient brightness can be determined by analyzing the image captured by the vehicle's external camera. For example, by capturing images such as a front-view camera and a surround-view camera, the average grayscale value of the pixels in the picture is calculated (0-255, corresponding to brightness from dark to bright), and then the image pixel average is determined. In one example, the second threshold can be set to 128. When the image pixel average of the ambient image is less than or equal to 128, it means that the ambient image is too dark, and the ambient image needs to be processed according to the processor to obtain a brightness enhancement map. Under this ambient brightness condition, the brightness enhancement network needs to be started. It should be noted that the value of the second threshold is only a schematic illustration, and can be specifically determined based on the target detection results of the target detection head under different ambient brightness conditions.

[0190] Of course, in some other examples, the average brightness value of the input image is directly calculated, that is, the V channel mean converted to the HSV space. If it is lower than the threshold, such as V mean <0.3, corresponding to a dark environment), the brightness enhancement network is activated.

[0191] In this way, the second threshold of the image pixel average value is used to distinguish between bright and dark environments. In a dark environment, it is determined that the brightness information of the environmental image needs to be enhanced, thereby improving the accuracy and reliability of the target detection result.

[0192] See also Figure 10 In some embodiments, the target detection method further comprises:

[0193] 014: When the average value of the image pixels is greater than the second threshold, target detection is performed based on the environment image.

[0194] In some embodiments, the processor is further configured to perform target detection based on the environment image when the average value of the image pixels is greater than a second threshold.

[0195] Specifically, in one example, when the average value of the image pixels of the environmental image is greater than 128, it means that the vehicle is in a normal environment and there is no need to process the environmental image to obtain a brightness enhancement map. Target detection can be performed directly on the environmental image, and the original environmental image is directly input into the target detection head to avoid image distortion caused by excessive enhancement. For example, no processing is required under strong light during the day.

[0196] In this way, when converting the dark environment image from RGB to HSV space and processing the brightness channel data V, normalization processing is used to unify the numerical range, facilitate mathematical operations, avoid numerical variation, ensure processing consistency, and facilitate compatibility with subsequent processing, which meets the input requirements of the brightness enhancement network.

[0197] See also Figure 11 In some embodiments, step 01 (determining a brightness-enhanced image based on the acquired environment image) includes:

[0198] 015: Perform color space conversion on the environment image to determine the target color space data;

[0199] 016: Determine the brightness enhancement map based on the brightness channel data in the target color space data.

[0200] In some embodiments, the processor is further configured to perform color space conversion processing on the environment image to determine target color space data; and is further configured to determine a brightness enhancement map based on brightness channel data in the target color space data.

[0201] Specifically, color space conversion processing refers to converting the color data in the environment image to obtain the target color space data. Luminance extraction processing refers to extracting the luminance component in the target color space data to obtain a luminance enhancement map. In one example, the RGB data of the environment image can first be normalized to scale the original RGB values ​​(0-255) to the range [0,1], and then the luminance V is calculated using the formula V=max(R,G,B) to obtain the target color space data V. The luminance enhancement map is determined based on the luminance channel data V in the target color space data.

[0202] In this way, by performing color space conversion processing on the environment image, the environment image is converted from RGB to HSV space, wherein the V channel can represent the brightness information of the environment image, and then the brightness enhancement map can be determined.

[0203] See also Figure 12 In some embodiments, step 016 (determining a brightness enhancement map based on brightness channel data in the target color space data) includes:

[0204] 0161: Normalize the brightness channel data and determine the brightness enhancement map based on the normalization result.

[0205] Specifically, the target color space data V is first normalized to obtain V', V'=V / 255. Among them, the brightness V∈[0,255], V'∈[0,1]. Then, the brightness enhancement map is calculated by 1-V'. In the brightness enhancement map, the pixel value with lower brightness of the corresponding environment image is closer to 1. For example, the brightness value of a pixel V=50 is darker, and after normalization, it is V'=50 / 255≈0.196. The brightness value of another pixel V=200 is brighter, and after normalization, it is V'=200 / 255≈0.784. Normalization facilitates subsequent mathematical operations, such as subtracting the normalized value from 1, and unifies the data range to avoid excessively large or small values ​​affecting model training.

[0206] It can be understood that the darker the image pixel, the smaller the V' value of the dark pixel, the closer it is to 0, and the larger the value of 1-V', the closer it is to 1; conversely, the larger the V' value of the bright pixel, the closer it is to 1, and the smaller the value of 1-V', the closer it is to 0. For example: the dark pixel V'=0.1 corresponds to a value of 0.9 in the bright image enhancement map; another example: the bright pixel V'=0.9 corresponds to a value of 0.1 in the bright image enhancement map. The closer the value of the dark area (low brightness) is to 1 in the brightness enhancement map, the more it needs to be enhanced; the bright area (high brightness) has a value close to 0 in the brightness enhancement map, indicating "weak enhancement" or "no enhancement". In this way, the network can focus on improving the brightness of the dark area and avoid over-enhancement of the overly bright area.

[0207] In this way, when converting the dark environment image from RGB to HSV space and processing the brightness channel data V, normalization processing is used to unify the numerical range, facilitate mathematical operations, avoid numerical variation, ensure processing consistency, and facilitate compatibility with subsequent processing, which meets the input requirements of the brightness enhancement network.

[0208] The embodiments of the present application also provide a detection model, comprising the brightness enhancement network of the embodiments of the present application and a target detection head cascaded with the brightness enhancement network. The vehicle target detection method of the embodiments of the present application can be implemented using the detection model of the present application. The brightness enhancement network can be described in the aforementioned embodiments with reference to the explanation of the brightness enhancement network, which will not be repeated here. The target detection head cascaded with the brightness enhancement network can utilize any target detection algorithm.

[0209] Specifically, in one example, the target detection head can use the YOLOv8n model, which has a small amount of model calculation, meets the requirements, and has deployment experience. In this application, the brightness enhancement network is used as an independent pre-processing module, which is connected in series between the acquisition unit such as the camera and the perception model (target detection head). There is no need to modify the structure or parameters of the target detection head. For example, YOLOv8n can directly use the official pre-trained weights and has strong compatibility. Through multi-scale feature fusion and channel number compression, the computational overhead can be controlled to meet the low power consumption and low latency requirements of the vehicle system.

[0210] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the object detection method for a vehicle.

[0211] It is understood that a computer program includes computer program code. The computer program code may be in source code form, object code form, executable file, or some intermediate form. Computer-readable storage media may include any entity or device capable of carrying computer program code, recording media, USB flash drives, mobile hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media.

[0212] In the description of this specification, the descriptions with reference to the terms "particularly", "further", "particularly", "understandably", etc. are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms are not intended to refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0213] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code that includes one or more executable requests for implementing a specific logical function or step of a process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0214] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A target detection method for a vehicle, characterized in that: include: Determine a brightness enhancement map according to the acquired environment image; Based on a pre-trained brightness enhancement network, target detection is performed according to the environment image and the brightness enhancement map.

2. The method according to claim 1, characterized in that The step of determining the brightness enhancement map according to the acquired environment image includes: In a case where the ambient brightness of the environment where the vehicle is located is less than or equal to a first threshold, the brightness enhancement map is determined according to the environmental image.

3. The method according to claim 2, characterized in that The method further comprises: When the ambient brightness is greater than the first threshold, target detection is performed based on the ambient image.

4. The method according to claim 1, wherein The step of determining the brightness enhancement map according to the acquired environment image includes: In a case where the average value of image pixels of the environment image is less than or equal to a second threshold, the brightness enhancement map is determined according to the environment image.

5. The method according to claim 4, characterized in that The method further comprises: When the image pixel average value is greater than a second threshold, target detection is performed based on the environment image.

6. The method according to claim 1, characterized in that The step of determining the brightness enhancement map according to the acquired environment image includes: Performing color space conversion processing on the environment image to obtain target color space data; The brightness enhancement map is determined according to the brightness channel data in the target color space data.

7. The method according to claim 6, characterized in that The determining the brightness enhancement map according to the brightness channel data in the target color space data includes: Normalization processing is performed on the brightness channel data, and the brightness enhancement map is determined based on the result of the normalization processing.

8. A brightness enhancement network, characterized in that: include: an encoder configured to perform downsampling processing on an input image to determine an output image, wherein the input image is determined according to an environment image and the brightness enhancement map, and the brightness enhancement map is determined according to the environment image; A decoder is configured to perform upsampling processing on the output image and determine a target brightness image, so that the target detection head of the brightness enhancement network cascade performs target detection based on the target brightness image.

9. The brightness enhancement network according to claim 8, characterized in that: The input image includes a first input image, and the encoder includes a first downsampling module, which is configured to perform downsampling processing on the first input image to determine a first feature image, wherein the first input image is determined based on the environmental image and the brightness enhancement map, the environmental image and the brightness enhancement map have the same size, and the number of channels of the first feature image is twice that of the first input image.

10. The brightness enhancement network according to claim 9, characterized in that: The input image also includes a second input image, and the encoder also includes a second downsampling module cascaded with the first downsampling module, and the second downsampling module is configured to downsample the second input image to determine a second feature image, wherein the second input image is determined based on the first feature image and a first brightness enhancement map, the first brightness enhancement map is obtained by downsampling the brightness enhancement map, the first feature image and the first brightness enhancement map have the same size, and the number of channels of the second feature image is twice that of the second input image.

11. The brightness enhancement network according to claim 10, characterized in that: The input image also includes a third input image, and the encoder also includes a third downsampling module cascaded with the second downsampling module, and the third downsampling module is configured to downsample the third input image to determine a third feature image, wherein the third input image is determined according to the second feature image and a second brightness enhancement map, the second brightness enhancement map is obtained by downsampling according to the brightness enhancement map, the second feature image and the second brightness enhancement map have the same size, and the number of channels of the third feature image is twice that of the third input image.

12. The brightness enhancement network according to claim 11, characterized in that: The input image also includes a fourth input image, and the encoder also includes a fourth downsampling module cascaded with the third downsampling module, and the fourth downsampling module is configured to downsample the fourth input image to determine a fourth feature image, wherein the fourth input image is determined according to the third feature image and a third brightness enhancement map, and the third brightness enhancement map is obtained by downsampling according to the brightness enhancement map, the third feature map and the third brightness enhancement map have the same size, and the number of channels of the fourth feature image is the same as that of the fourth input image.

13. The brightness enhancement network according to claim 12, characterized in that: The output image includes a first output image, the decoder includes a first upsampling module cascaded with the fourth downsampling module, the first upsampling module is configured to perform upsampling processing on the first output image to determine a first luminance image, wherein the first output image is determined based on the fusion of the fourth feature image and the third feature image, the number of channels of the first output image is the sum of the number of channels of the fourth feature image and the third feature image, and the number of channels of the first luminance image is 1 / 2 of the first output image.

14. The brightness enhancement network according to claim 13, characterized in that: The output image also includes a second output image, and the decoder also includes a second upsampling module cascaded with the first upsampling module, and the second upsampling module is configured to perform upsampling processing on the second output image to determine a second luminance image, wherein the second output image is determined based on the fusion of the second feature image and the first luminance image, the number of channels of the second output image is the sum of the number of channels of the second feature image and the first luminance image, and the number of channels of the second luminance image is 1 / 2 of the second output image.

15. The brightness enhancement network according to claim 14, characterized in that: The output image also includes a third output image, and the decoder also includes a third upsampling module cascaded with the second upsampling module. The third upsampling module is configured to perform upsampling processing on the third output image to determine a third luminance image, wherein the third output image is determined based on the fusion of the first feature image and the second luminance image, the number of channels of the third output image is the sum of the number of channels of the first feature image and the second luminance image, and the number of channels of the third luminance image is 1 / 2 of the third output image.

16. The brightness enhancement network according to claim 15, characterized in that: The output image also includes a fourth output image, and the decoder also includes a channel compression module cascaded with a third upsampling module, and the channel compression module is configured to compress the number of channels of the third brightness image to determine the target brightness image, wherein the number of channels of the target brightness image is adapted to the number of channels of the target detection head.

17. The brightness enhancement network according to claim 9, characterized in that: The brightness enhancement network is trained by the following steps: Fixedly configuring the weight of the target detection head according to a preset data set; Randomly initializing the weights of the brightness enhancement network; Based on the training loss function of the object detection head, the brightness enhancement network is trained according to a preset training set to determine the weights and bias values ​​of the brightness enhancement network.

18. A detection model, characterized in that: The method comprises a brightness enhancement network as described in any one of claims 8 to 17 and a target detection head cascaded with the brightness enhancement network.

19. A vehicle, characterized in that: The vehicle is deployed with a brightness enhancement network as described in any one of claims 8-17 or a detection model as described in claim 18.

20. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

21. A vehicle, characterized in that: The vehicle includes the electronic device according to claim 20.

22. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 7 is implemented.

23. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images