Multi-module cooperative rain removal method and system for rainy day vehicle detection

Abstract

The invention discloses a multi-module cooperative rain removal method and system for rainy day vehicle detection. Rainy images input into the multi-module cooperative rain removal method are given, original noise filtering is completed through a convolutional layer, a progressive information sensing module is constructed, the progressive information sensing module is used for enriching extraction of spatial details and transmitting information with high priority, and a feature # imgabs0 # is output; the feature # imgabs1 # is reduced to a low resolution step by step in a coding unit through down-sampling, is mapped to an original high resolution by a decoding unit after being processed by an intermediate layer, and outputs a feature # imgabs2 #; the feature # imgabs3 is mapped by a 3 * 3 convolutional layer to generate a residual image, and the residual image is added to the # imgabs4 to obtain a final rain removal result.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a multi-module collaborative rain removal method and system for rainy day vehicle detection. Background Art

[0002] At present, the research on automobile intelligence is becoming the focus, and intelligent vehicle technology plays an important role in ensuring traffic safety and promoting urban development. Among the many key subsystems of intelligent vehicles, the vehicle detection system plays the role of identifying vehicle targets in traffic scenes and calculating their spatial positions. Therefore, ensuring the stable operation of the vehicle detection system is of great significance to the braking, obstacle avoidance and other functions of intelligent vehicles.

[0003] Among the many methods of vehicle detection, the application of vehicle detection methods based on computer vision is more extensive than other methods due to its low cost. However, the actual traffic environment is relatively complex and changeable, so it is necessary to continuously optimize the performance of benchmark vehicle detection methods represented by SSD and YOLO series models to ensure the safe driving of smart cars under various working conditions. In order to consolidate and improve the performance of vehicle detection methods, the current conventional practice is to use a general and powerful benchmark vehicle detection method as the underlying architecture, and then continuously optimize the various components of the benchmark vehicle detection method to meet the needs of various detection tasks. Liu et al. improved the accuracy of SSD detection of long-distance vehicles by designing feature fusion. Xu et al. improved the vehicle detection accuracy of YOLOX-S based on the strategy of integrating attention mechanism and optimizing loss function. At the same time, some scholars are committed to improving the accuracy of benchmark vehicle detection methods in specific scenarios. Ning et al. improved the vehicle detection performance of YOLOv3 in foggy scenes by integrating DenseNet and attention mechanism. Zhang et al. improved the vehicle detection accuracy of RetinaNet in night scenes based on the method of optimizing loss function and network structure. Although the above optimization method improves the detection performance of the baseline vehicle detection method, it may not be the optimal solution for smart cars. It is well known that the images generated by on-board cameras in complex scenes may be degraded to varying degrees, which fundamentally restricts the vehicle feature information contained in the image. In rainy scenes, raindrops attached to the lens or window glass will directly interfere with the visibility of the background, and raindrops suspended in the air will scatter the light. Even when the rainfall is heavier, dense rain lines will gather into obvious rain streaks and rain fog in the air, thereby blocking various objects in the vehicle background layer. These interference factors will cause problems such as loss of texture details and degradation of edge information in rainy images, which will lead to reduced vehicle detection accuracy and directly threaten the safe driving of smart cars. Therefore, strengthening image deraining research to enrich the feature information of the vehicle background layer is of great significance to improving the performance of vehicle detection methods.

[0004] Image deraining is a classic image restoration task. With the increasingly stringent requirements of industries such as smart logistics and smart cars on image quality, this work is becoming a hot topic. Most of the early image deraining methods were model-driven, that is, the deraining model was designed based on the prior knowledge of the image, such as the direction and density of rain streaks. Among them, the methods include discriminative sparse coding and Gaussian mixture model, but these methods do not consider other factors such as rain and fog. Benefiting from the vigorous development of artificial intelligence, data-driven deraining methods have become mainstream. Li et al. proposed a non-local enhanced encoder-decoder architecture for learning complex features, thereby accurately modeling rain streaks while retaining image details. Zhang et al. constructed a deraining method based on rain density perception. Ren et al. comprehensively considered the network structure and loss function and built a simple and efficient baseline deraining method. Although the above methods can achieve image deraining, there are residual rain layers caused by incomplete deraining and background damage induced by excessive deraining. The MPRNet proposed by Zamir et al. decomposes the rain removal process into multiple stages to achieve progressive rain removal. At the same time, it extracts deep features based on channel attention and establishes a cross-stage feature fusion mechanism to strengthen the information interaction between each stage. Therefore, MPRNet can effectively remove rain while alleviating the residual rain layer and background damage. However, the complex structural design of MPRNet leads to an increase in the number of network parameters and a longer running time.

[0005] In summary, the various defects of existing rain removal methods that cannot be ignored limit their direct application in rainy day vehicle detection. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a multi-module collaborative rain removal method and system for rainy day vehicle detection in view of the deficiencies in the above-mentioned prior art, so as to solve the technical problem that the detection accuracy of the benchmark vehicle detection method declines in rainy days.

[0007] The present invention adopts the following technical solutions: A multi-module collaborative rain removal method for rainy day vehicle detection includes the following steps: S1. Given a rainy image with input multi-module collaborative deraining method, the original noise is filtered through the convolution layer, and a progressive information perception module is constructed to enrich the extraction of spatial details and transmit high-priority information, and the output features ; S2. Features The encoding unit is downsampled and gradually reduced to a low resolution. After being processed by the intermediate layer, it is mapped to the original high resolution by the decoding unit, and the output feature ; S3. Features After being mapped by a 3×3 convolutional layer, a residual image is generated and compared with the input image Add them together to get the final rain removal result.

[0008] Preferably, the feature for:

[0009] in, Represents a 3×3 convolutional layer, responsible for inputting the image Dimensional operations, represents two consecutive progressive information perception modules, It is the feature output by the shallow feature extraction unit.

[0010] Preferably, the progressive information perception module includes: Low-level detail information extraction module, embedded features After layer normalization, the standard features are obtained , and then the convolutional layer completes the cross-dimensional information interaction. Perform equal splitting at channel level; Feature extraction is done based on the convolutional layer. It is passed directly to the next layer without being processed. and Perform channel-level integration and refinement through convolutional layers; generate attention maps based on GeLU and obtain representative detail feature representations; obtain the output features of the low-level detail information extraction module by element-wise addition ; High-level semantic information extraction module, output features After layer normalization, the canonical features are obtained , followed by multiple convolutional layers to complete the deep feature extraction. SCA adaptively calibrates the rain layer information with higher priority; the features are further refined through the convolution layer and the output features are obtained by element-wise addition. .

[0011] Preferably, the output feature for:

[0012] in, Represents the characteristics of attention feedback, Represents a 1×1 convolutional layer.

[0013] Preferably, the feature for:

[0014] in, is the feature after processing in the middle layer, It is the decoding unit in the encoder-decoder architecture.

[0015] Preferably, a multi-level feature capture module is embedded in each level of the encoder-decoder architecture, and the information flow process is as follows: The embedded features are preprocessed based on layer normalization to stabilize the training process; then the convolutional layers are used to Perform dimensionality transformation; use depthwise separable convolutional layers for deep feature extraction, aggregate 3D attention to adaptively guide the retention and transfer of higher priority rain layer information, and further refine it by the convolutional layer.

[0016] Preferably, the multi-level feature capture module aggregates the dual-branch structure under the premise of acceptable computational overhead to explore the rain layer features of different levels and forms, as follows:

[0017] in, represents a 3×3 depthwise separable convolutional layer, represents GeLU activation, and are the output features of the upper branch and the lower branch respectively, represents the addition of elements, It is the feature after the fusion of the two branches.

[0018] Preferably, high quality rain-free images for: .

[0020] Preferably, the Charbonnier loss and edge loss are integrated to complete the multi-module collaborative deraining network training and define the total loss function for:

[0021] in, According to experience, it is set to 0.05. is the edge loss, For the loss of Charbonnier, represents the predicted rain-free image, Represents the original rain-free image.

[0022] In a second aspect, an embodiment of the present invention provides a multi-module collaborative rain removal system for rainy day vehicle detection, comprising: The shallow feature extraction unit, given the rainy image of the multi-module collaborative deraining method, completes the original noise filtering through the convolution layer, builds a progressive information perception module, which is used to extract rich spatial details and transmit high-priority information, and outputs features ; Deep feature extraction unit, features The encoding unit is downsampled and gradually reduced to a low resolution. After being processed by the intermediate layer, it is mapped to the original high resolution by the decoding unit, and the output feature ; Deraining result output unit, features After being mapped by a 3×3 convolutional layer, a residual image is generated. Add them together to get the final rain removal result.

[0023] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned multi-module collaborative rain removal method for rainy day vehicle detection when executing the computer program.

[0024] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned multi-module collaborative rain removal method for rainy day vehicle detection.

[0025] In a fifth aspect, a chip comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the multi-module collaborative rain removal method for rainy day vehicle detection are implemented.

[0026] In a sixth aspect, an embodiment of the present invention provides an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-mentioned multi-module collaborative rain removal method for rainy day vehicle detection.

[0027] Compared with the prior art, the present invention has at least the following beneficial effects: A multi-module collaborative rain removal method for rainy day vehicle detection can efficiently remove the rain layer and restore the vehicle in the image, and can effectively improve the rainy day detection accuracy of various benchmark vehicle detection algorithms.

[0028] Furthermore, given a rainy image with input multi-module collaborative deraining method, the original noise is filtered through the convolution layer, and a progressive information perception module is constructed to enrich the extraction of spatial details and transmit high-priority information, and the output features , the progressive information perception module works together with multi-scale processing and attention mechanism to enhance the model's ability to recognize rain streak features. Multi-scale processing enables the model to capture raindrop and rain line features of different scales, which behave differently at different resolutions. Multi-scale analysis can provide a more comprehensive understanding of the distribution and morphology of rain streaks. The attention mechanism further enhances the model's ability to focus on key information. It can identify and emphasize the most important features in the image, such as the edges and shapes of raindrops, while suppressing less important background information. The benefit of this design is that it not only improves the deraining effect, but also preserves the details and texture information of the image. By progressively fusing features of different scales, the model is able to reconstruct rain-free images more accurately while reducing artifacts and blur that may occur during the deraining process. In addition, the priority delivery of high-priority information means that the model can respond and process the most critical features faster, which is particularly important for real-time or near-real-time deraining applications.

[0029] Furthermore, the benefit of setting the progressive information perception module is that the progressive information perception module includes a low-level detail information extraction module and a high-level semantic information extraction module, which mainly contribute to representative feature representation.

[0030] Further, the characteristics The encoding unit is downsampled and gradually reduced to a low resolution. After being processed by the intermediate layer, it is mapped to the original high resolution by the decoding unit, and the output feature , through downsampling, the spatial dimension of the data can be reduced, which not only reduces the computational complexity of the model, but also helps the model extract more abstract features, which are crucial for understanding the global structure of the data. At the same time, the receptive field is expanded during the downsampling process, allowing the model to capture a wider range of contextual information, which is especially important for tasks such as image recognition. Secondly, during the downsampling process, although the spatial resolution is reduced, the model retains the important features and spatial information of the input data through parameter sharing and convolution operations. The retention of this information is crucial for subsequent tasks such as image segmentation or object detection. After processing the intermediate layer, the decoding unit maps the low-resolution feature map back to the original high resolution. This step allows the model to fuse feature information of different scales and enhances the model's ability to capture details. This feature fusion not only improves the performance of the model, but also helps to recover spatial details that may be lost during the downsampling process.

[0031] Furthermore, the multi-level feature capture module can explore the contextual information of the rain layer and its bidirectional disturbance with the vehicle background layer in a large receptive field.

[0032] Furthermore, the total loss function can enhance the tolerance to errors and further improve the image quality after rain removal.

[0033] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0034] In summary, the present invention can efficiently eliminate the rain layer and restore the vehicle in the image, and effectively improve the rainy day detection accuracy of various benchmark vehicle detection methods.

[0035] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a multi-level feature capture module diagram; Figure 3 It is a diagram of the progressive information perception module; Figure 4 It is a simplified channel attention map; Figure 5 The gain effect diagram of the proposed algorithm over the baseline YOLOv8m in rain streak and rain fog scenes; Figure 6 A schematic diagram of a computer device provided by an embodiment of the present invention; Figure 7 The block diagram of a chip provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0039] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0040] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0041] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0042] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0043] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0044] The present invention provides a multi-module collaborative deraining method (MultipleBlocks Collaborative Deraining Net, MBCDNet) for rainy day vehicle detection, which builds a multi-level feature capture module to provide insight into the contextual information of the rain layer in a large receptive field; constructs a progressive information perception module to contribute to the representation of representative features; and achieves efficient identification and removal of the rain layer, and clear positioning and restoration of the vehicle background layer. In terms of image deraining, comprehensive evaluation results in the Test1200 and Test2800 public synthetic datasets show that the PSNR and SSIM of the image after deraining by the method of the present invention have some advantages over MPRNet, and the average running time is reduced by 30.5% and 33.6% respectively compared with MPRNet, which proves the advancedness and real-time performance of the method of the present invention. In terms of vehicle detection in rainy days, the test results in the Rain-Vehicle self-built dataset show that the mAP@0.5 gains of the method of the present invention over the benchmarks SSD, YOLOv5m, YOLOv8n and YOLOv8m are 6.1%, 8.8%, 12.4% and 7.8% respectively, which proves the effectiveness of the method of the present invention for vehicle detection in rainy days and has certain practical value.

[0045] According to the lightweight design principle, the present invention completes the construction of the core network module based on only two modules in the single-stage architecture. Considering the various types of rain layer shapes and complex and changeable positions in rainy images, MBCDNet has built a multi-level feature capture module to gain insight into the contextual information of the rain layer in the large-scale receiving domain, and constructed a progressive information perception module to strengthen the representation of representative features in the cumbersome information flow. The two modules cooperate with each other, embed different units and flexibly and collaboratively represent the contextual information and spatial features of the rain layer, thereby guiding MBCDNet to complete the efficient identification and removal of the rain layer and the clear positioning and restoration of the vehicle background layer.

[0046] The present invention provides a multi-module collaborative rain removal method for rainy day vehicle detection, which includes a shallow feature extraction unit and a deep feature extraction unit. The shallow feature extraction unit enhances the retention of the spatial features of the rain layer. The deep feature extraction unit enhances the accurate perception and efficient transmission of the contextual information and structural features of the rain layer. In addition, the corresponding levels of the encoding unit (Encoder) and the decoding unit (Decoder) establish interaction through long jump connections, and thereby enrich the spatial features; the specific steps are as follows: S1. Given a rainy image with input multi-module collaborative deraining method, in the shallow feature extraction unit, the convolution layer completes the original noise filtering, and the progressive information perception module is used to extract rich spatial details and transmit higher priority information; Features output by shallow feature extraction unit for: (1) in, Represents a 3×3 convolutional layer, responsible for inputting the image Dimensional operations, represents two consecutive progressive information perception modules, It is the feature output by the shallow feature extraction unit.

[0047] Progressive Information Perception Module The rainy images captured by the camera contain a variety of complex rain layers and vehicle background layers. At the same time, the manifestation and flow patterns of rain streaks and rain fog in the air are significantly different, so the disturbances they cause to the vehicle background layer are also different. Therefore, the network's ability to recognize and perceive rain layers of different forms should be enhanced to avoid residual rain layers and excessive rain removal caused by confusion between rain layers and vehicle background layers, which in turn leads to damage to the detailed information of the vehicle background layer. In order to enable the network to efficiently explore and remove the redundant rain layer information in rainy images while perfectly retaining the fine details of the vehicle background layer, the present invention designs a progressive information perception module, such as Figure 3 shown.

[0048] The progressive information perception module includes a low-level detail information extraction module and a high-level semantic information extraction module. It is a feature extraction module similar to the Transformer architecture. Given that the multi-level feature capture module has strong context information extraction capabilities, this module is used as a supplement and mainly contributes to representative feature representation. Conventional Transformers use a self-attention mechanism to calculate relevance, but its inference cost is difficult to ignore and there is feature redundancy.

[0049] In particular, in order to improve the operation efficiency of the network, this module does not choose the aggregated self-attention mechanism and its variants, but incorporates the basic concept of feature capture in the spatial adaptive modulation mechanism and Res2Net, thereby establishing a low-level detail information extraction module. At the same time, the simplified channel attention (SCA) is aggregated to construct a high-level semantic information extraction module. Given the embedded features of the progressive information perception module, the detailed mapping process is expressed as follows (8) in, is the embedding feature, and are the output features of the low-level detail information extraction module and the high-level semantic information extraction module, respectively. , Representative feature operations that represent low-level detail information and high-level semantic information, respectively.

[0050] Specifically, in the low-level detail information extraction module of the progressive information perception module, the feature embedding After layer normalization, the standard features are obtained , and then the convolutional layer completes the cross-dimensional information interaction (9) in, Represents the features mapped by the 1×1 convolutional layer.

[0051] Secondly, Perform equal split at channel level (10) in, represents channel splitting, and Represents the features of the split.

[0052] then, Feature extraction is done based on the convolutional layer. Passed directly to the next layer without processing (11) in, Represents the features mapped by a 3×3 depthwise separable convolutional layer.

[0053] Afterwards, and Perform channel-level integration and refinement through convolutional layers (12) in, represents channel-level splicing, Represents the features after refinement by the 1×1 convolutional layer.

[0054] Afterwards, the attention map is generated based on GeLU and representative detail feature representation is obtained (13) Finally, the output features of the low-level detail information extraction module are obtained by element-wise addition.

[0055] (14) In particular, in order to enhance the ability of the progressive information perception module to screen and transmit key rain layer features, the present invention designs a high-level semantic information extraction module. In this module, After layer normalization, the canonical features are obtained , followed by multiple convolutional layers to complete deep feature extraction (15) in, Represents deep semantic features.

[0056] Then, SCA adaptively calibrates the rain layer information with higher priority (16) in, and Represent the characteristics of attention correction and its feedback respectively. The SCA network structure is as follows Figure 4 shown.

[0057] Finally, the features are further refined through the convolutional layer and the output features are obtained by element-wise addition. (17) Based on the above structure and mapping process, the progressive information perception module, as a supplement to the multi-level feature capture module, realizes the exploration and perception of low-level detail information and high-level semantic information of representative rain layers.

[0058] S2. In the deep feature extraction unit, The encoding unit is downsampled to a lower resolution, and after being processed by the intermediate layer, it is mapped to the original high resolution by the decoding unit. (2) in, and Represent the encoding unit and decoding unit in the encoder-decoder architecture, respectively. and are the input and output features of the encoding unit, respectively. is the feature after processing in the middle layer, is the feature output by the decoding unit, Represents the middle layer consisting of six consecutive progressive information perception modules.

[0059] Multi-level feature capture module There are complex spatial relationships in rainy images. For example, the rain layers in the same area may contain different depth information. Therefore, the limited receptive field may not be able to mine and transmit sufficient information for the network to identify the relationship between rain layers and the interaction between rain layers and vehicle background layers, which will result in residual rain marks and damage to the details of the vehicle background layer in the restored image, which may eventually cause disadvantages that cannot be ignored in subsequent vehicle detection tasks. In short, the complete removal of the rain layer and the complete restoration of the vehicle background layer must comprehensively consider multi-scale information. Therefore, in order to further explore the contextual information of the rain layer in a large range of receptive fields and its bidirectional perturbation with the vehicle background layer, See also Figure 2, a multi-level feature capture module is used to embed each level of the encoder-decoder architecture, providing insight into the context of the rain layer in multi-scale space, while also dynamically adjusting the spatial features passed from the encoding unit to the decoding unit through skip connections. Given the embedded features, the information flow in this module is as follows: First, given that the effectiveness of layer normalization has been demonstrated in multiple image restoration studies, the embedded features are preprocessed based on layer normalization to stabilize the training process. (4) in, is the embedding feature, and They represent the layer normalization operation and its normalized features respectively.

[0060] Then use the convolutional layer to Perform dimension transformation (5) in, and They represent the features of the 1×1 convolutional layer and its mapping, respectively.

[0061] Next, in order to meet the lightweight design requirements, a deep separable convolutional layer is used to extract deep features. At the same time, the module aggregates a dual-branch structure to explore the characteristics of rain layers at different levels and in different forms under the premise of acceptable computational overhead. (6) in, represents a 3×3 depthwise separable convolutional layer, represents GeLU activation, and are the output features of the upper branch and the lower branch respectively, represents the addition of elements, It is the feature after the fusion of the two branches.

[0062] Subsequently, the 3D attention is aggregated to adaptively guide the retention and transfer of the higher priority rain layer information, which is further refined by the convolutional layer. (7) in, represents the three-dimensional attention mechanism, The rain layer features are finally mined by the multi-level feature capture module.

[0063] S3. In the deraining result output unit, After being mapped by a 3×3 convolutional layer, a residual image is generated, which is then Add them together to get the final rain removal result.

[0064] High quality rain-free images for: (3) in, This is the high-quality rain-free image finally output by the method of the present invention.

[0065] Loss Function In order to further improve the image quality after rain removal, the Charbonnier loss and edge loss are combined to complete the network training. , edge loss They are defined as (18) (19) in, represents the Laplace operator, represents the predicted rain-free image, represents the original rain-free image, Set to 1×10 -3 .

[0066] Total loss function Defined as (20) in, It is empirically set to 0.05.

[0067] In yet another embodiment of the present invention, a multi-module collaborative rain removal system for rainy day vehicle detection is provided. The system can be used to implement the above-mentioned multi-module collaborative rain removal method for rainy day vehicle detection. Specifically, the multi-module collaborative rain removal system for rainy day vehicle detection includes a shallow feature extraction unit, a deep feature extraction unit and a rain removal result output unit.

[0068] Among them, the shallow feature extraction unit, given the rainy image of the multi-module collaborative deraining method, completes the original noise filtering through the convolution layer, builds a progressive information perception module, which is used to extract rich spatial details and transmit high-priority information, and outputs the feature ; Deep feature extraction unit, features The encoding unit is downsampled and gradually reduced to a low resolution. After being processed by the intermediate layer, it is mapped to the original high resolution by the decoding unit, and the output feature ; Deraining result output unit, features After being mapped by a 3×3 convolutional layer, a residual image is generated. Add them together to get the final rain removal result.

[0069] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a multi-module collaborative deraining method for rainy vehicle detection, including: Given a rainy image with input multi-module collaborative deraining method, the original noise is filtered through the convolution layer, and a progressive information perception module is constructed to enrich the extraction of spatial details and transmit high-priority information, and the output features ;feature The encoding unit is downsampled and gradually reduced to a low resolution. After being processed by the intermediate layer, it is mapped to the original high resolution by the decoding unit, and the output feature ;feature After being mapped by a 3×3 convolutional layer, a residual image is generated. Add them together to get the final rain removal result.

[0070] See also Figure 6 , the terminal device is a computer device. The computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the multi-module collaborative rain removal method for rainy day vehicle detection in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the multi-module collaborative rain removal system for rainy day vehicle detection in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0071] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 6 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0072] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0073] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0074] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0075] See also Figure 7 , the terminal device is a chip, and the chip 600 of this embodiment includes a processor 622, which may be one or more, and a memory 632 for storing a computer program executable by the processor 622. The computer program stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processor 622 may be configured to execute the computer program to execute the above-mentioned multi-module collaborative rain removal method for rainy vehicle detection.

[0076] In addition, the chip 600 may further include a power supply component 626 and a communication component 650, wherein the power supply component 626 may be configured to perform power management of the chip 600, and the communication component 650 may be configured to implement communication, for example, wired or wireless communication, of the chip 600. In addition, the chip 600 may further include an input / output interface 658. The chip 600 may operate based on an operating system stored in the memory 632.

[0077] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (Non-VolatileMemory), such as at least one disk memory.

[0078] The processor may load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multi-module collaborative rain removal method for rainy vehicle detection in the above embodiment; the processor may load and execute the following steps: Given a rainy image with input multi-module collaborative deraining method, the original noise is filtered through the convolution layer, and a progressive information perception module is constructed to enrich the extraction of spatial details and transmit high-priority information, and the output features ;feature The encoding unit is downsampled and gradually reduced to a low resolution. After being processed by the intermediate layer, it is mapped to the original high resolution by the decoding unit, and the output feature ;feature After being mapped by a 3×3 convolutional layer, a residual image is generated. Add them together to get the final rain removal result.

[0079] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0080] The multi-module collaborative rain removal method proposed in this paper and the integrated benchmark vehicle detection method were used for relevant experiments on RTX 3090. The batch size and epoch size of the proposed method were set to 4 and 400 respectively during training. The Adam optimizer was used and the initial learning rate was set to 2×10 -4 , using the Cosine Annealing strategy, decaying to 1×10 -6 until.

[0081] Experimental verification of the advancedness of the multi-module collaborative rain removal method: This section follows the literature [20,21] The method of [13,17,34-36] The provided dataset is repartitioned as shown in Table 1.

[0082] Table 1 Division and renaming of the rain removal dataset

[0083] The effectiveness and generalization verification experiment of the multi-module collaborative deraining method: Since the focus of this invention is to consolidate and enhance the vehicle detection accuracy of the benchmark vehicle detection method on rainy days by designing a deraining method that is suitable for various rain layer forms, a Rain-Vehicle dataset for rainy day vehicle detection experiments is constructed. The data in this dataset mainly comes from the COCO dataset and the vehicle images collected by the author. Specifically, the training set and the validation set are 2150 and 500 normal clear images respectively, and the test set is 696 degraded rainy images. The distribution of rain layers mainly includes synthetic and real rain streaks and rain fog. Since this dataset contains a variety of scenes such as cities, suburbs and villages at home and abroad, the generalization and practicality of the multi-module collaborative deraining method can be fully verified. In the rainy day vehicle detection experiment in Chapter 4, data containing various rain layer distribution forms are used for training to obtain a deraining model that can be used for rainy day vehicle detection. The Rain-Vehicle dataset is used to complete the training of four benchmark vehicle detection methods, and the trained weights are used to conduct comparative experiments in the test set.

[0084] To prove the advancedness of MBCDNet proposed in this paper, this section compares the deraining performance with other excellent deraining methods in five public data sets. The deraining performance evaluation indicators are Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). Among them, the higher the PSNR, the closer the SSIM is to 1, indicating that the corresponding method has better deraining performance. The specific results are shown in Table 2, where the bold ones are the best results and the underlined ones are the suboptimal results.

[0085] Table 2 Comparison of rain removal performance of different rain removal methods

[0086] As shown in Table 2, compared with other classic excellent deraining methods RESCAN, PReNet and MSPFN, MBCDNet has achieved the best PSNR and SSIM on Test100, Rain100H, Rain100L, Test2800 and Test1200 datasets.

[21] In comparison, MBCDNet has the best PSNR performance in the Test1200 dataset, and the best SSIM performance in the Test100, Rain100L and Test1200 datasets. Taking the average of the five datasets, MBCDNet's PSNR is 32.56dB and SSIM is 0.924, which are both superior to MPRNet. In short, the above comparison objectively proves that MBCDNet has excellent deraining performance.

[0087] In order to fully illustrate the advancement and real-time performance of the proposed method MBCDNet, this section conducts a comprehensive evaluation with the main competing method MPRNet in the Test1200 and Test2800 datasets with rich scene content and diverse rain streak types as shown in Table 3, where the best results are displayed in bold.

[0088] Table 3 Comparison of the proposed method and MPRNet in terms of rain removal performance, average running time, and number of parameters

[0089] As shown in Table 3, the PSNR and SSIM of MBCDNet are close to those of MPRNet in the Test1200 and Test2800 datasets. The average running time is 0.098s and 0.079s, respectively, which is 30.5% and 33.6% lower than that of MPRNet. At the same time, the number of parameters of MBCDNet is 1.18M, ​​which is 67.6% lower than that of MPRNet. The above experiments fully demonstrate that MBCDNet has achieved a balance between deraining performance and computational efficiency.

[0090] To illustrate the advantages of MBCDNet over recent excellent deraining methods in terms of running efficiency, 100 rainy images with image sizes of 256×256, 384×384, and 512×512 were selected and named Image256, Image384, and Image512, respectively. The average running time of recent excellent deraining methods was compared on the above three datasets. The test results are shown in Table 4, where the best results are bolded and the suboptimal results are underlined.

[0091] Table 4 Comparison of average running time of different deraining methods

[0092] As shown in Table 4, in the Image256 dataset, when the image size is 256×256, the average running time of MBCDNet is reduced by 21.4% compared with MPRNet, and the average running time of MBCDNet is reduced by 21.4% compared with Restormer. [ The processing speed of a single image by the rain removal method Restormer is 0.216s in the Image384 dataset, which is 384×384 in size, and cannot meet the real-time requirements. However, the processing speed of the proposed method is 0.067s, which still has low real-time performance. In summary, for rainy images of various sizes, MBCDNet has a significant speed advantage over recent excellent rain removal methods, and therefore it is easier to meet the real-time rain removal requirements under various conditions.

[0093] The comprehensive advantages of the proposed method MBCDNet in terms of rain removal performance and average running time have been fully demonstrated, and therefore it is more suitable for rainy day vehicle detection tasks than other excellent rain removal methods.

[0094] Furthermore, to prove the effectiveness of this method for rainy day vehicle detection, the benchmark vehicle detection method SSD is incorporated [3] , YOLOv5 and YOLOv8.

[0095] Solution 1 is the result of the benchmark vehicle detection method directly detecting vehicles in the test set (without rain removal). Solution 2 is the vehicle detection results of the benchmark vehicle detection method after the test set is first processed by MBCDNet for rain removal.

[0096] The evaluation index is mAP@0.5. The higher the value, the better. The detailed comparison is shown in Table 5.

[0097] Table 5 Gain results of the proposed method over the benchmark vehicle detection method

[0098] As shown in Table 5, the mAP@0.5 gains of MBCDNet over the benchmark SSD, YOLOv5m, YOLOv8n and YOLOv8m in rainy day vehicle detection are 6.1%, 8.8%, 12.4% and 7.8% respectively, which fully proves that the MBCDNet proposed in this paper can improve the detection accuracy of various benchmark vehicle detection methods in rainy days.

[0099] To more intuitively illustrate the generalization and effectiveness of MBCDNet, five traffic scene images with different rainfall levels are selected for visualization. In the synthetic rain streak and rain fog scenes (a), (b) and (c) and the real rain streak and rain fog scenes (d) and (e), the gain results of MBCDNet over the baseline YOLOv8m are shown in the figure. Figure 5 shown.

[0100] Depend on Figure 5 It is observed that for the synthetic rainfall scenes (a), (b) and (c), although the rain falling angle and the number of rain streaks are significantly different, MBCDNet can effectively remove the rain streaks. Not only is the vehicle background layer in the figure clearly restored, but the restored image is also real and natural. Specifically, from scene (a), due to the interference of dense rain streaks, the first scheme using only the baseline YOLOv8m misdetects roadside plants as vehicles, while the second scheme introduces MBCDNet. Benefiting from the excellent image deraining ability of this method, the false detection phenomenon not only disappears, but also the confidence of the vehicle detection result is improved from 0.84 to 0.94. From scene (c), thanks to MBCDNet, the baseline YOLOv8m avoids missed detections, and the detection results have a higher confidence. For the real rainfall scenes (d) and (e), the rain streaks in scene (e) are more complex, but MBCDNet still performs well in restoring scene (e). After removing most of the rain streaks in the image, the confidence of vehicle detection increases from 0.88 to 0.94.

[0101] In summary, the multi-module collaborative rain removal method and system for rainy day vehicle detection of the present invention has the following characteristics: (1) The MBCDNet rain removal method designed by the present invention has a simple architecture. The multi-level feature capture module and the progressive information perception module complement and cooperate with each other, realizing the accurate perception and capture of the position and shape of the rain layer in the rainy image and the degree of interference with the vehicle background layer, thereby guiding the overall network to complete the core task of removing the rain layer and restoring the vehicle background layer. The test results on multiple synthetic rain datasets show that the method has excellent rain removal performance. The test results on the Test1200 and Test2800 synthetic rain datasets show that the method has the comprehensive advantages of excellent rain removal performance and faster average running time compared with the recent excellent rain removal method MPRNet.

[0102] (2) Based on the driving environment of smart cars, we constructed the Rain-Vehicle dataset, fused MBCDNet with a variety of benchmark vehicle detection methods and completed the test on this dataset. The results show that the mAP@0.5 gains of the proposed method over the benchmarks SSD, YOLOv5m, YOLOv8n and YOLOv8m are 6.1%, 8.8%, 12.4% and 7.8% respectively, which proves the generalization and effectiveness of the method.

[0103] The method of the present invention can efficiently and accurately restore vehicle targets in rainy images, thereby effectively improving the detection accuracy of the benchmark vehicle detection method in smart cars on rainy days. The results of the present invention can provide a reference for improving the detection accuracy of the vehicle detection system of smart cars on rainy days and even in other complex weather conditions. However, the rain removal method of the present invention still has room for optimization, such as the processing speed should be further improved to achieve real-time rain removal in high-resolution images. In addition, since the rainy images collected by the on-board camera in the night scene are easily affected by various light sources, the distribution of rain streaks is more complex and diverse. Therefore, in subsequent work, research on rain removal methods for night scenes should be carried out.

[0104] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0105] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0106] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0107] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0110] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0111] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0114] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A multi-module collaborative rain removal method for rainy day vehicle detection, characterized in that: The following steps are involved: S1. Given a rainy image with input multi-module collaborative deraining method, the original noise is filtered through the convolution layer, and a progressive information perception module is constructed to enrich the extraction of spatial details and transmit high-priority information, and the output features ; S2. Features The encoding unit is downsampled and gradually reduced to a low resolution. After being processed by the intermediate layer, it is mapped to the original high resolution by the decoding unit, and the output feature ; S3. Features After being mapped by a 3×3 convolutional layer, a residual image is generated and compared with the input image Add them together to get the final rain removal result.

2. The multi-module collaborative rain removal method for rainy day vehicle detection according to claim 1 is characterized in that: feature for: in, Represents a 3×3 convolutional layer, responsible for inputting the image Dimensional operations, represents two consecutive progressive information perception modules, It is the feature output by the shallow feature extraction unit.

3. The multi-module collaborative rain removal method for rainy day vehicle detection according to claim 1 is characterized in that: The progressive information perception module includes: Low-level detail information extraction module, embedding features After layer normalization, the standard features are obtained , and then the convolutional layer completes the cross-dimensional information interaction. Perform equal splitting at the channel level; Feature extraction is done based on the convolutional layer. It is passed directly to the next layer without being processed. and Perform channel-level integration and refine through convolutional layers; generate attention maps based on GeLU and obtain representative detail feature representations; obtain the output features of the low-level detail information extraction module by element-wise addition ; High-level semantic information extraction module, output features After layer normalization, the canonical features are obtained , followed by multiple convolutional layers to complete the deep feature extraction. SCA adaptively calibrates the rain layer information with higher priority; the features are further refined through the convolution layer and the output features are obtained by element-wise addition. .

4. The multi-module collaborative rain removal method for rainy day vehicle detection according to claim 3 is characterized in that: Output characteristics for: in, Represents the characteristics of attention feedback, Represents a 1×1 convolutional layer.

5. The multi-module collaborative rain removal method for rainy day vehicle detection according to claim 1, characterized in that: feature for: in, is the feature after processing in the middle layer, It is the decoding unit in the encoder-decoder architecture.

6. The multi-module collaborative rain removal method for rainy day vehicle detection according to claim 5, characterized in that: Each level of the encoder-decoder architecture embeds a multi-level feature capture module, and the information flow process is as follows: The embedded features are preprocessed based on layer normalization to stabilize the training process; then the convolutional layers are used to Perform dimensional transformation; Depthwise separable convolutional layers are used for deep feature extraction, and three-dimensional attention is aggregated to adaptively guide the retention and transfer of higher priority rain layer information, which is further refined by the convolutional layers.

7. The multi-module collaborative rain removal method for rainy day vehicle detection according to claim 6 is characterized in that: The multi-level feature capture module aggregates the dual-branch structure under the premise of acceptable computational overhead to explore the characteristics of rain layers at different levels and in different forms, as follows: in, represents a 3×3 depthwise separable convolutional layer, represents GeLU activation, and are the output features of the upper branch and the lower branch respectively, represents the addition of elements, It is the feature after the fusion of the two branches.

8. The multi-module collaborative rain removal method for rainy day vehicle detection according to claim 1, characterized in that: High quality rain-free images for: 。 9. The multi-module collaborative rain removal method for rainy day vehicle detection according to claim 1, characterized in that: Fusion of Charbonnier loss and edge loss completes multi-module collaborative deraining network training and defines the total loss function for: in, According to experience, it is set to 0.

05. is the edge loss, For the loss of Charbonnier, represents the predicted rain-free image, Represents the original rain-free image.

10. A multi-module collaborative rain removal system for rainy day vehicle detection, characterized in that: include: The shallow feature extraction unit, given the rainy image of the multi-module collaborative deraining method, completes the original noise filtering through the convolution layer, builds a progressive information perception module, which is used to extract rich spatial details and transmit high-priority information, and outputs features ; Deep feature extraction unit, features The encoding unit is downsampled and gradually reduced to a low resolution. After being processed by the intermediate layer, it is mapped to the original high resolution by the decoding unit, and the output feature ; Deraining result output unit, features After being mapped by a 3×3 convolutional layer, a residual image is generated. Add them together to get the final rain removal result.

Images