Control method of foreign matter removing equipment, vehicle controller and vehicle

Abstract

The invention relates to the technical field of vehicles, in particular to a control method of foreign matter removing equipment, a vehicle controller and a vehicle. The method comprises the following steps: acquiring a current video frame image of a windshield of a vehicle at a current moment; performing image segmentation on the current video frame image, and determining a segmentation image corresponding to each image category; wherein the image category comprises any one of raindrops, snowflakes and dust; determining a control parameter corresponding to each image category in the image categories corresponding to the non-empty segmented images based on the image categories corresponding to the non-empty segmented images when determining that the non-empty segmented images exist; wherein the control parameters comprise control parameters used for controlling the foreign matter removing equipment; and controlling the foreign matter removing equipment to operate according to the control parameters to remove foreign matters on the windshield.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle technology, and in particular to a control method for a foreign matter removing device, a vehicle controller and a vehicle. Background Art

[0002] When a driver encounters rain, snow or bad weather while driving a vehicle, the driver is often required to manually adjust the wipers and / or washer according to the actual weather conditions, which distracts the driver's attention to a certain extent and affects driving safety.

[0003] Therefore, how to make the wipers automatically adjust based on the actual weather conditions to reduce the driver's operation time of the wipers and / or sprayers has become an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides a control method for a foreign object removal device, a vehicle controller and a vehicle, which are used to solve the problem of how to make the wipers automatically adjust based on the actual weather conditions to reduce the driver's operation time of the wipers.

[0005] In a first aspect, the present application provides a control method for a foreign object removal device, comprising: obtaining a current video frame image of a vehicle's windshield at a current moment; performing image segmentation on the current video frame image, and determining a segmentation map corresponding to each image category; wherein the image category includes any one of raindrops, snowflakes, and dust; when it is determined that there is a non-empty segmentation map, based on the image category corresponding to the non-empty segmentation map, determining control parameters corresponding to each image category in the image category corresponding to the non-empty segmentation map; wherein the control parameters include control parameters for controlling the foreign object removal device; controlling the foreign object removal device to operate according to the control parameters to remove foreign objects from the windshield; wherein the foreign object removal device includes one or more of a wiper and a sprinkler, and the foreign objects include one or more of raindrops, snowflakes, and dust.

[0006] In some feasible examples, before obtaining the current video frame image of the windshield of the vehicle at the current moment, the control method of the foreign object removal device provided by the present disclosure also includes: obtaining training sample data and the marking result of the training sample data; wherein the training sample data includes at least one first video frame image of the windshield containing foreign objects and at least one second video frame image not containing foreign objects, and the marking result includes whether the first video frame image and the second video frame image contain foreign objects; inputting the training sample data into the deep learning architecture for learning, and obtaining the prediction result of the deep learning architecture for the training sample data; wherein the deep learning architecture includes an encoder and a decoder, the encoder includes at least two first feature extraction layers and at least three second feature extraction layers, the first feature extraction layer includes a first feature extraction module and a first attention mechanism module for enhancing the performance of the convolutional neural network, and the second feature extraction layer includes a first feature fusion module and a first attention mechanism module for enhancing A second attention mechanism module for improving the performance of a convolutional neural network, the decoder includes at least five second feature fusion modules, the first feature extraction module adopts a deep separable convolution and suppresses its maximum value activation function, the output of the decoder is equal to the sum of the first actual output of the target attention mechanism module of the encoder and the second actual output determined by the decoder based on the output of the encoder, the target attention mechanism module includes a first attention mechanism module and a second attention mechanism module; the loss value of the prediction result and the labeling result is calculated based on the target loss function; wherein the target loss function includes a binary cross entropy loss with a scaling factor introduced; when the loss value is less than or equal to the target value, the deep learning architecture is determined to converge and an image segmentation model is obtained; the current video frame image is segmented to determine the segmentation map corresponding to each image category, including: inputting the current video frame image into the image segmentation model for image segmentation to determine the segmentation map corresponding to each image category.

[0007] In some feasible examples, the first feature extraction module includes a 1×1 first convolution kernel, a 3×3 depthwise separable convolution kernel, and a 1×1 second convolution kernel, the input of the first convolution kernel includes training sample data or the input of the first convolution kernel includes the output of the first feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer, the input of the depthwise separable convolution kernel includes the output of the first convolution kernel, the input of the second convolution kernel includes the output of the depthwise separable convolution kernel, the output of the second convolution kernel includes the output of the feature extraction module, and the activation functions of both the depthwise separable convolution kernel and the second convolution kernel are activation functions that suppress their maximum values.

[0008] In some feasible examples, the input of the first feature fusion module includes the output of the feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer, the input of the second feature fusion module includes the output of the first attention mechanism module in the target extraction layer of the same layer as the second feature fusion module and the output of the second feature fusion module of the previous layer, the target extraction layer includes any one of the first feature extraction layer and the second feature extraction layer, or the input of the second feature fusion module includes the output of the first attention mechanism module in the second feature extraction layer of the same level as the second feature fusion module, the output of the first feature fusion module includes the input of the first feature fusion module in the first feature extraction layer of the next layer, or the output of the first feature fusion module includes the input of the first attention mechanism module of the same level as the first feature fusion module, and the output of the second feature fusion module includes the input of the second feature fusion module of the next layer.

[0009] In some implementable examples, the first feature fusion module and the second feature fusion module both include a second feature extraction module, a dual-channel feature separation module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, a sixth feature extraction module, a channel number superposition module, a seventh feature extraction module and a third attention mechanism module for enhancing the performance of the convolutional neural network; wherein the input of the second feature extraction module includes the output of the feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer, or the input of the second feature extraction module includes the output of the first attention mechanism module in the first feature extraction layer of the previous layer and the output of the second feature fusion module of the previous layer, or the input of the second feature extraction module includes the output of the first attention mechanism module in the second feature extraction layer of the same level as the second feature extraction module; the dual-channel feature separation module is used to perform feature separation on the output of the second feature extraction module to obtain a first separation feature and a second separation feature, the first separation feature is the input of the channel number superposition module, the second separation feature is the input of the channel number superposition module, and the second separation feature The feature is the input of the third feature extraction module; the third feature extraction module is used to process the second separation feature output by the dual-channel feature separation module to obtain the first result; the fourth feature extraction module is used to process the first result to obtain the second result; the fifth feature extraction module is used to process the second result output by the fourth feature extraction module to obtain the third result; the sixth feature extraction module is used to process the third result output by the fifth feature extraction module to obtain the fourth result; the channel number superposition module is used to perform feature splicing on the first separation feature and the second separation feature output by the dual-channel feature separation module and the fourth result output by the sixth feature extraction module to obtain the fifth result; the seventh feature extraction module is used to process the fifth result output by the channel number superposition module to obtain the sixth result; the third attention mechanism module is used to process the sixth result output by the seventh feature extraction module to obtain the seventh result; the seventh result is the input of the first feature fusion module in the first feature extraction layer of the next layer, or the seventh result is the input of the first attention mechanism module at the same level as the first feature fusion module, and the seventh result is the input of the second feature fusion module of the next layer.

[0010] In some feasible examples, the foreign object removing device includes a wiper; when it is determined that there is a non-empty segmentation map, based on the image category corresponding to the non-empty segmentation map, determine the control parameters corresponding to each image category in the image category corresponding to the non-empty segmentation map, including: when it is determined that there is a non-empty segmentation map and the image category corresponding to the non-empty segmentation map is raindrops, determine the first actual area of ​​the minimum enclosing area containing all raindrops in the segmentation map; determine the first actual density based on the ratio of the first actual area to the second actual area of ​​the windshield; calculate the actual similarity between the segmentation map of the current video frame image and the segmentation map of the previous video frame image; determine the first theoretical parameter based on the first actual density and the actual similarity; use the first theoretical parameter as the control parameter corresponding to the image category is raindrops; control the foreign object removing device to operate according to the control parameters to remove foreign objects from the windshield, including: control the wiper to operate according to the control parameters to remove foreign objects from the windshield.

[0011] In some feasible examples, the foreign object removing device includes a wiper; when it is determined that there is a non-empty segmentation map, based on the image category corresponding to the non-empty segmentation map, determine the control parameters corresponding to each image category in the image category corresponding to the non-empty segmentation map, including: when it is determined that there is a non-empty segmentation map, and when the image category corresponding to the non-empty segmentation map is snowflakes, determine the third actual area of ​​the minimum enclosing area containing all snowflakes in the segmentation map; based on the ratio of the second actual area and the second actual area of ​​the windshield, determine the second actual density; based on the second actual density, determine the second theoretical parameter; use the second theoretical parameter as the control parameter corresponding to the image category is snowflakes; control the foreign object removing device to operate according to the control parameters to remove foreign objects from the windshield, including: control the wiper to operate according to the control parameters to remove foreign objects from the windshield.

[0012] In some feasible examples, the foreign object removal device includes a water sprayer and a wiper; when it is determined that there is a non-empty segmentation map, based on the image category corresponding to the non-empty segmentation map, the control parameters corresponding to each image category in the image category corresponding to the non-empty segmentation map are determined, including: when it is determined that there is a non-empty segmentation map, and the image category corresponding to the non-empty segmentation map is dust, the control parameter of the water sprayer is determined to be a first preset parameter, and the control parameter of the wiper is determined to be a second preset parameter; controlling the foreign object removal device to operate according to the control parameters to remove foreign objects from the windshield, including: controlling the water sprayer to operate according to the first preset parameter, and controlling the wiper to operate according to the second preset parameter.

[0013] In a second aspect, the present application provides a vehicle controller, comprising: an acquisition module, used to acquire a current video frame image of a vehicle's windshield at a current moment; a processing module, used to perform image segmentation on the current video frame image acquired by the acquisition module, and determine a segmentation map corresponding to each image category; wherein the image category includes any one of raindrops, snowflakes, and dust; the processing module is also used to determine, when it is determined that there is a non-empty segmentation map, based on the image category corresponding to the non-empty segmentation map, the control parameters corresponding to each image category in the image category corresponding to the non-empty segmentation map; wherein the control parameters include control parameters for controlling a foreign object removal device; the processing module is also used to control the foreign object removal device to operate according to the control parameters to remove foreign objects from the windshield; wherein the foreign object removal device includes one or more of a wiper and a sprinkler, and the foreign objects include one or more of raindrops, snowflakes, and dust.

[0014] In a third aspect, the present application provides a vehicle, the vehicle comprising the vehicle controller as described above.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above method.

[0016] Compared with the prior art, the technical solution provided by the present invention has the following advantages:

[0017] The control method of the foreign object removal device provided by the present disclosure is as follows: by obtaining the current video frame image of the windshield of the vehicle at the current moment; performing image segmentation on the current video frame image to determine the segmentation graph corresponding to each image category; when it is determined that there is a segmentation graph that is not empty, based on the image category corresponding to the segmentation graph that is not empty, determining the control parameters corresponding to each image category in the image category corresponding to the segmentation graph that is not empty; and controlling the foreign object removal device to operate according to the control parameters to remove foreign objects from the windshield. In this way, the driver does not need to operate the wiper and / or the water sprayer during driving the vehicle, but the controller of the vehicle can determine whether to control the foreign object removal device to remove foreign objects from the windshield based on the current video frame image of the windshield at the current moment, thereby shortening the driver's operation time of the wiper and / or the water sprayer, and solving the problem of how to make the wiper automatically adjust based on the actual weather conditions to reduce the driver's operation time of the wiper and / or the water sprayer. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 One of the flow charts of a control method of a foreign matter removal device provided in the first embodiment is exemplarily shown;

[0021] Figure 2 2 is a flow chart of a control method for a foreign matter removal device provided in the first embodiment;

[0022] Figure 3 Schematic diagram 3 of a flow chart of a control method for a foreign matter removal device provided in the first embodiment is exemplarily shown;

[0023] Figure 4 4 is a flow chart of a control method for a foreign matter removal device provided in the first embodiment;

[0024] Figure 5 Schematic diagram 5 of a flow chart of a control method for a foreign matter removal device provided in the first embodiment is exemplarily shown;

[0025] Figure 6 6 is a flow chart of a control method for a foreign matter removal device provided in the first embodiment;

[0026] Figure 7 FIG. 7 exemplarily shows a flow chart of a control method of a foreign matter removal device provided in the first embodiment;

[0027] Figure 8 One of the structural diagrams of the vehicle controller provided in the second embodiment is exemplarily shown in FIG.

[0028] Fig. 9 FIG. 2 exemplarily shows a second structural diagram of the vehicle controller provided in the second embodiment. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0031] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0032] In some examples, the U-Net model in the embodiments of the present disclosure is an excellent semantic segmentation model, which is mainly used for pixel-level classification of images.

[0033] In some examples, ReLU6 in the disclosed embodiments is an activation function, which is a variant of the ReLU (Rectified Linear Unit) function.

[0034] In some examples, Focal Loss in the embodiments of the present disclosure is a loss function designed specifically for solving the problems of class imbalance and imbalance between difficult and easy samples, and has demonstrated its unique advantages and broad application potential in multiple deep learning fields.

[0035] In some examples, Conv in the embodiments of the present disclosure refers to a convolution operation, which is used to extract features in deep learning.

[0036] In some examples, CBAM in the embodiments of the present disclosure refers to Convolutional Block Attention Module, which is a lightweight attention module designed to enhance the representation capability of convolutional neural networks (CNNs).

[0037] In some examples, c2f in the embodiments of the present disclosure refers to a module for feature extraction and enhancement in the target detection algorithm (You Only Look Once, YOLO) algorithm, which is widely used in the Backbone and Head parts of the network.

[0038] Embodiment 1

[0039] Figure 1 A flow chart of a control method for removing foreign matter equipment is shown in FIG. 1 . The execution subject of this example may be a vehicle controller, such as Figure 1 As shown, the method includes:

[0040] S11. Obtain a current video frame image of the windshield of the vehicle at the current moment.

[0041] In some examples, an image acquisition device for acquiring a current image of the windshield is provided in the cabin of the vehicle. After the vehicle is turned on, the image acquisition device is powered on, and the image acquisition device can acquire a current video frame image of the windshield at the current moment in real time, so that the vehicle controller can acquire the current video frame image of the windshield at the current moment through the image acquisition device.

[0042] In some examples, the image acquisition device may be a camera.

[0043] In some examples, since the foreign object removal device can only remove foreign objects from a partial area of ​​the windshield when cleaning foreign objects from the windshield, in order to ensure that the foreign object removal device can remove foreign objects from this partial area, the current video frame image at the current moment captured by the image acquisition device may be an image corresponding to the partial area of ​​the windshield that can be cleaned by the foreign object removal device. This can avoid the foreign object removal device from working frequently due to the fact that the current video frame image at the current moment captured by the acquisition device contains images of other areas of the windshield except this partial area.

[0044] In some examples, in order to reduce the occupancy of computing resources of the vehicle controller, the current video frame image may be acquired periodically, such as acquiring the current video frame image at a period of 4 seconds.

[0045] S12, performing image segmentation on the current video frame image to determine a segmentation map corresponding to each image category, wherein the image category includes any one of raindrops, snowflakes, and dust.

[0046] In some examples, the current video frame image may contain one or more of raindrops, snowflakes, and dust. In order to ensure that the windshield does not affect the driver's driving of the vehicle, the current video frame image is analyzed to determine the foreign matter contained in the current video frame image, such as: performing image segmentation on the current video frame image to determine the foreground image and background image in the current video frame image. Performing image recognition on the foreground image to determine the foreign matter contained in the foreground image, such as: when the foreign matter contained in the foreground image is raindrops, the foreground image is used as a segmentation map of the image category (raindrops); or when the foreign matter contained in the foreground image is snowflakes, the foreground image is used as a segmentation map of the image category (snowflakes); when the foreign matter contained in the foreground image is dust, the foreground image is used as a segmentation map of the image category (dust). Alternatively, when the foreign objects contained in the foreground image are multiple items among raindrops, snowflakes and dust, the foreground image is used as a segmentation map of the image category (each of the multiple items among raindrops, snowflakes and dust). For example, when the foreign objects contained in the foreground image are raindrops and snowflakes, the foreground image is used as a segmentation map of the image category (snowflakes), and the foreground image is also used as a segmentation map of the image category (raindrops).

[0047] Alternatively, the current video frame image is input into an image segmentation model for image segmentation to determine a segmentation map corresponding to each image category.

[0048] S13. When it is determined that there is a segmentation map that is not empty, based on the image category corresponding to the segmentation map that is not empty, determine the control parameters corresponding to each image category in the image category corresponding to the segmentation map that is not empty; wherein the control parameters include control parameters for controlling a foreign matter removal device.

[0049] In some examples, when the foreign objects are raindrops, the image category of the segmentation map containing the raindrops is raindrops; when the foreign objects are snowflakes, the image category of the segmentation map containing the raindrops is snowflakes; when the foreign objects are dust, the image category of the segmentation map containing the raindrops is dust.

[0050] In some examples, in order to prevent the foreign body removal device from being started frequently, when it is determined that there is a non-empty segmentation map and the frequency of occurrence of the image category corresponding to the non-empty segmentation map is greater than the preset frequency, the control parameters corresponding to each image category in the image category corresponding to the non-empty segmentation map are determined. Among them, the frequency of occurrence is equal to the ratio of the total number of any image category corresponding to the non-empty segmentation map in the target number of video frame images obtained continuously to the target number, such as: the preset frequency is 40%, 20 consecutive video frame images, 10 of which contain segmentation maps with image category of raindrops, then the frequency of occurrence of the image category of raindrops is determined to be 50%. Since 50% is greater than 40%, it is necessary to determine the control parameters of the image category of raindrops.

[0051] In some examples, the segmentation map is an image corresponding to the smallest circumscribed area containing the same foreign object in the current video frame image.

[0052] In some examples, when the segmentation map is empty, that is, the pixel values ​​of all pixels of the segmentation map are 0, it means that the current video frame image does not contain any of raindrops, snowflakes and dust, that is, there is no foreign matter on the windshield.

[0053] In some examples, when the segmentation map is not empty, that is, there are pixels with non-zero pixel values ​​in the segmentation map, it indicates that the current video frame image may contain one or more of raindrops, snowflakes, and dust, that is, there are foreign objects on the windshield. When cleaning different foreign objects on the windshield, different foreign object cleaning devices are used for different foreign objects, and different cleaning strategies are used for the foreign object cleaning devices, that is, different image categories correspond to different control parameters of the foreign object cleaning devices.

[0054] In some examples, the vehicle memory pre-stores the correspondence relationship of the control parameters of the foreign object removal device corresponding to each image category. Therefore, the vehicle controller can read the correspondence relationship stored in the vehicle memory, and then query the correspondence relationship based on the image category corresponding to the segmentation map that is not empty, so as to obtain the control parameter corresponding to each image category in the image category corresponding to the segmentation map that is not empty.

[0055] Alternatively, the vehicle's memory stores a calculation method for control parameters of a foreign object removal device corresponding to each image category, such as: the foreign object removal device includes a wiper. When it is determined that there is a non-empty segmentation map, and the image category corresponding to the non-empty segmentation map is raindrops, the calculation method corresponding to the image category (raindrops) is determined to be method one: wherein method one includes: calculating a first actual area of ​​a minimum enclosing area containing all raindrops in the segmentation map; determining a first actual density based on a ratio of the first actual area to a second actual area of ​​the windshield; calculating an actual similarity between a segmentation map of a current video frame image and a segmentation map of a previous video frame image; determining a first theoretical parameter based on the first actual density and the actual similarity; and using the first theoretical parameter as a control parameter corresponding to the image category being raindrops.

[0056] Alternatively, the foreign object removal device includes a wiper. When it is determined that there is a non-empty segmentation map, and the image category corresponding to the non-empty segmentation map is snowflakes, the calculation method corresponding to the image category (snowflakes) is determined to be method two: wherein method two includes: calculating the third actual area of ​​the minimum enclosing area containing all snowflakes in the segmentation map; determining the second actual density based on the ratio of the second actual area and the second actual area of ​​the windshield; determining the second theoretical parameter based on the second actual density; and using the second theoretical parameter as the control parameter corresponding to the image category being snowflakes.

[0057] Alternatively, the foreign object removal device includes a water sprayer and a wiper; when it is determined that there is a non-empty segmentation map, and the image category corresponding to the non-empty segmentation map is dust, the calculation method corresponding to the image category (dust) is determined to be method three: wherein method three includes: determining that the control parameter of the water sprayer is the first preset parameter (such as turning on the water sprayer and spraying water on the windshield), and the control parameter of the wiper is the second preset parameter (such as controlling the wiper to operate according to a preset gear).

[0058] It should be noted that the above example is based on the example that the current video frame image contains only one image category of foreign objects. When the current video frame image contains foreign objects of multiple image categories, a segmentation map corresponding to the foreign objects of each image category will be obtained, that is, the same segmentation map only contains foreign objects of the same category. Afterwards, according to the priority of foreign object cleaning, the control parameters corresponding to the image category to which the foreign object belongs are used in turn to control the foreign object removal device to operate according to the control parameters to remove foreign objects from the windshield.

[0059] For example, assuming that the dust cleaning priority is higher than the raindrop cleaning priority, the raindrop cleaning priority is higher than the snowflake cleaning priority, and the dust cleaning priority is higher than the snowflake cleaning priority. The current video frame image contains foreign objects of two image categories, namely raindrops and snowflakes. At this time, the current video frame image is segmented to obtain the segmentation corresponding to the image category (raindrops). Figure 1 , and the segmentation corresponding to the image category (snowflakes) Figure 2 Due to the segmentation Figure 1 Not empty, so based on split Figure 1 Image category, determine the segmentation Figure 1 The corresponding control parameter of the foreign body removal device is 1. Similarly, due to the segmentation Figure 2 Not empty, so based on split Figure 2 Image category, determine the segmentation Figure 2 The corresponding control parameter 2 of the foreign body removal device. Since the cleaning priority of raindrops is higher than the cleaning priority of snowflakes, the vehicle controller first controls the wiper to operate according to control parameter 1 to remove raindrops from the windshield. Afterwards, when the vehicle determines that the preset conditions are met, the vehicle controller controls the wiper to operate according to control parameter 2 to remove snowflakes from the windshield. Among them, the preset conditions include: the vehicle controller controls the wiper to operate according to control parameter 1 for a preset time (such as 30s), or after the vehicle controller controls the wiper to operate according to control parameter 1, the corresponding video frame image is obtained for image segmentation, and it is determined that the image category of the segmentation map that is not empty does not contain raindrops.

[0060] S14. Control the foreign object removal device to operate according to the control parameters to remove foreign objects from the windshield; wherein the foreign object removal device includes one or more of a wiper and a sprinkler, and the foreign objects include one or more of raindrops, snowflakes, and dust.

[0061] As can be seen from the above, the control method of the foreign object removal device provided by the embodiment of the present disclosure obtains the current video frame image of the windshield of the vehicle at the current moment; performs image segmentation on the current video frame image to determine the segmentation map corresponding to each image category; when it is determined that there is a segmentation map that is not empty, based on the image category corresponding to the segmentation map that is not empty, determines the control parameters corresponding to each image category in the image category corresponding to the segmentation map that is not empty; controls the foreign object removal device to operate according to the control parameters to remove foreign objects from the windshield. In this way, the driver does not need to operate the wipers and / or the water sprayer while driving the vehicle, but the vehicle controller can determine whether to control the foreign object removal device to remove foreign objects from the windshield based on the current video frame image of the windshield at the current moment.

[0062] In some possible implementation examples, combined with Figure 1 ,like Figure 2 As shown, before executing S11, the control method of the foreign matter removing device provided by the embodiment of the present disclosure further includes: S15-S18, and the above S12 can be specifically implemented by the following S120.

[0063] S15. Obtain training sample data and marking results of the training sample data; wherein the training sample data includes at least one first video frame image of the windshield containing foreign objects and at least one second video frame image not containing foreign objects, and the marking results include whether the first video frame image and the second video frame image contain foreign objects.

[0064] S16. Input the training sample data into the deep learning architecture for learning, and obtain the prediction results of the deep learning architecture for the training sample data; wherein, the deep learning architecture includes an encoder and a decoder, the encoder includes at least two first feature extraction layers and at least three second feature extraction layers, the first feature extraction layer includes a first feature extraction module and a first attention mechanism module for enhancing the performance of the convolutional neural network, the second feature extraction layer includes a first feature fusion module and a second attention mechanism module for enhancing the performance of the convolutional neural network, the decoder includes at least five second feature fusion modules, the first feature extraction module adopts a deep separable convolution and suppresses its maximum value activation function, the output of the decoder is equal to the sum of the first actual output of the target attention mechanism module of the encoder and the second actual output determined by the decoder based on the output of the encoder, and the target attention mechanism module includes a first attention mechanism module and a second attention mechanism module.

[0065] In some examples, the deep learning architecture can be a U-Net model.

[0066] In some examples, the first feature extraction module, the second feature extraction module, the third feature extraction module, the fourth feature extraction module, the fifth feature extraction module, the sixth feature extraction module and the seventh feature extraction module are all Conv.

[0067] In some examples, the first attention mechanism module, the second attention mechanism module, and the third attention mechanism module are all CBAMs.

[0068] In some examples, both the first feature fusion module and the second feature fusion module are C2f.

[0069] For example, taking the encoder including two first feature extraction layers and three second feature extraction layers, and the decoder including five second feature fusion modules as an example, the network structure of the U-Net model is as follows: Figure 3 As shown, the U-Net model includes an encoder 1 and a decoder 2. The encoder 1 includes a first feature extraction layer 1, a first feature extraction layer 2, a second feature extraction layer 1, a second feature extraction layer 2, and a second feature extraction layer 3. The decoder 2 includes a second feature fusion module 1, a second feature fusion module 2, a second feature fusion module 3, a second feature fusion module 4, and a second feature fusion module 5.

[0070] Exemplarily, the first feature extraction layer includes a first feature extraction module and a first attention mechanism module for enhancing the performance of the convolutional neural network, the second feature extraction layer includes a first feature fusion module and a second attention mechanism module for enhancing the performance of the convolutional neural network, the decoder includes at least five second feature fusion modules, and the network structure of the U-Net model is as follows: Figure 4As shown, the U-Net model includes an encoder 1 and a decoder 2. The encoder 1 includes a first feature extraction layer 1, a first feature extraction layer 2, a second feature extraction layer 1, a second feature extraction layer 2, and a second feature extraction layer 3. The first feature extraction layer 1 includes a first feature extraction module (Conv1-1) and a first attention mechanism module (CBAM1-1). The first feature extraction layer 2 includes a first feature extraction module (Conv2-1) and a first attention mechanism module (CBAM2-1). The second feature extraction layer 1 includes a first feature fusion module (C2f1-1) and a second attention mechanism module (CBAM1-2). The second feature extraction layer 2 includes a first A feature fusion module (C2f2-1) and a second attention mechanism module (CBAM2-2), the third feature extraction layer 3 includes a first feature fusion module (C2f3-1) and a second attention mechanism module (CBAM3-2), the decoder 2 includes a second feature fusion module 1, a second feature fusion module 2, a second feature fusion module 3, a second feature fusion module 4 and a second feature fusion module 5, the second feature fusion module 1 includes C2f1-3, the second feature fusion module 2 includes C2f2-3, the second feature fusion module 3 includes C2f3-3, the second feature fusion module 4 includes C2f4-3 and the second feature fusion module 5 includes C2f5-3.

[0071] In some examples, the maximum suppression activation function includes ReLu6.

[0072] S17. Calculate the loss values ​​of the prediction results and the labeling results based on the target loss function; wherein the target loss function includes a binary cross entropy loss that introduces a scaling factor.

[0073] In some examples, the objective loss function includes Focal Loss.

[0074] S18. When the loss value is less than or equal to the target value, determine that the deep learning architecture converges and obtain an image segmentation model.

[0075] In some examples, when the number of times the loss value is continuously less than or equal to the target value is greater than or equal to the convergence value (such as 200 times), it is determined that the deep learning architecture has converged, and the converged deep learning architecture is used as the image segmentation model.

[0076] S120: Input the current video frame image into an image segmentation model for image segmentation, and determine a segmentation map corresponding to each image category.

[0077] In some feasible examples, the first feature extraction module includes a 1×1 first convolution kernel, a 3×3 depthwise separable convolution kernel, and a 1×1 second convolution kernel, the input of the first convolution kernel includes training sample data or the input of the first convolution kernel includes the output of the first feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer, the input of the depthwise separable convolution kernel includes the output of the first convolution kernel, the input of the second convolution kernel includes the output of the depthwise separable convolution kernel, the output of the second convolution kernel includes the output of the feature extraction module, and the activation functions of both the depthwise separable convolution kernel and the second convolution kernel are activation functions that suppress their maximum values.

[0078] In some examples, such as Figure 4 As shown, the first feature extraction module (Conv1-1) includes a 1×1 first convolution kernel (Conv1 1×1), a 3×3 depth-separable convolution kernel (Dwise 3×3), the activation function of Dwise 3×3 is the maximum value suppression activation function BN Relu6, and a 1×1 second convolution kernel (Conv2 1×1), and the activation function of Conv2 1×1 is the maximum value suppression activation function BN Relu6, which further reduces the computational complexity of the U-Net model.

[0079] In some examples, the previous layer and the next layer in the embodiments of the present disclosure refer to the next module of the current module along the data flow direction, that is, the next layer. The previous module of the current module against the data flow direction is the previous layer. Figure 4 , the input of the first feature extraction module (Conv1-1) includes the training sample data, and the input of the first feature extraction module (Conv2-1) includes the output of the first feature extraction module (Conv1-1) in the first feature extraction layer (i.e., the first feature extraction layer 1) of the previous layer and the output of the first attention mechanism module (CBAM1-1) in the first feature extraction layer (i.e., the first feature extraction layer 1) of the previous layer. Similarly, the output of the first feature extraction module (Conv1-1) in the first feature extraction layer 1 is the input of the first feature extraction module (Conv2-1) of the first feature extraction layer 2 of the next layer, and the output of the first attention mechanism module (CBAM1-1) in the first feature extraction layer 1 is the input of the first feature extraction module (Conv2-1) of the first feature extraction layer 2 of the next layer.

[0080] It should be noted that the above example is based on the network structure of the first feature extraction module (Conv1-1). Since the network structure of the first feature extraction module (Conv2-1) is the same as that of the first feature extraction module Conv1-1, it will not be repeated here.

[0081] It can be seen that the control method for removing foreign matter equipment provided by the embodiment of the present disclosure uses a depth-separable convolution to replace the existing convolution operation. First, the feature dimension is expanded by 3 times using the first 1×1 convolution kernel (Conv1 1×1), and then the feature information in each channel is extracted using the 3×3 depth-separable convolution kernel (Dwise 3×3). Then, the dimension is reduced by the second 1×1 convolution kernel (Conv2 1×1) to fuse the information between different channels. On the one hand, in order to enable the model to be deployed on the vehicle computer and achieve real-time detection capabilities, the computational cost of the model should be reduced as much as possible. On the other hand, the information interaction capability between different channels can be enhanced, and the feature extraction capability of the network can be improved.

[0082] In some feasible examples, the input of the first feature fusion module includes the output of the feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer, the input of the second feature fusion module includes the output of the first attention mechanism module in the target extraction layer of the same layer as the second feature fusion module and the output of the second feature fusion module of the previous layer, the target extraction layer includes any one of the first feature extraction layer and the second feature extraction layer, or the input of the second feature fusion module includes the output of the first attention mechanism module in the second feature extraction layer of the same level as the second feature fusion module, the output of the first feature fusion module includes the input of the first feature fusion module in the first feature extraction layer of the next layer, or the output of the first feature fusion module includes the input of the first attention mechanism module of the same level as the first feature fusion module, and the output of the second feature fusion module includes the input of the second feature fusion module of the next layer.

[0083] In some implementable examples, the first feature fusion module and the second feature fusion module both include a second feature extraction module, a dual-channel feature separation module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, a sixth feature extraction module, a channel number superposition module, a seventh feature extraction module and a third attention mechanism module for enhancing the performance of the convolutional neural network; wherein the input of the second feature extraction module includes the output of the feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer, or the input of the second feature extraction module includes the output of the first attention mechanism module in the first feature extraction layer of the previous layer and the output of the second feature fusion module of the previous layer, or the input of the second feature extraction module includes the output of the first attention mechanism module in the second feature extraction layer of the same level as the second feature extraction module; the dual-channel feature separation module is used to perform feature separation on the output of the second feature extraction module to obtain a first separation feature and a second separation feature, the first separation feature is the input of the channel number superposition module, the second separation feature is the input of the channel number superposition module, and the second separation feature The feature is the input of the third feature extraction module; the third feature extraction module is used to process the second separation feature output by the dual-channel feature separation module to obtain the first result; the fourth feature extraction module is used to process the first result to obtain the second result; the fifth feature extraction module is used to process the second result output by the fourth feature extraction module to obtain the third result; the sixth feature extraction module is used to process the third result output by the fifth feature extraction module to obtain the fourth result; the channel number superposition module is used to perform feature splicing on the first separation feature and the second separation feature output by the dual-channel feature separation module and the fourth result output by the sixth feature extraction module to obtain the fifth result; the seventh feature extraction module is used to process the fifth result output by the channel number superposition module to obtain the sixth result; the third attention mechanism module is used to process the sixth result output by the seventh feature extraction module to obtain the seventh result; the seventh result is the input of the first feature fusion module in the first feature extraction layer of the next layer, or the seventh result is the input of the first attention mechanism module at the same level as the first feature fusion module, and the seventh result is the input of the second feature fusion module of the next layer.

[0084] In some examples, the dual-channel feature separation module is used to split the output of the second feature extraction module into two parts, such as feature 1 and feature 2, feature 1 and feature 2 are different, and feature 1 and feature 2 are equal to the output of the second feature extraction module after splicing.

[0085] Exemplarily, the dual-channel feature separation module (Chunk) in the first feature fusion module (C2f1-1) in the second feature extraction layer 1 divides the output of the second feature extraction module (Conv1) in the second feature extraction layer 1 into two parts.

[0086] In some examples, the same level in the embodiments of the present disclosure refers to Figure 4 The modules of the same layer in the U-Net model shown, such as: the first feature extraction layer 1 and the second feature fusion module 5 are at the same level, that is, the first feature extraction module Conv1-1 and the first attention mechanism module (CBAM1-1) in the first feature extraction layer 1 are at the same level as the second feature fusion module 5 (C2f5-3). Similarly, the first feature extraction layer 2 and the second feature fusion module 4 are at the same level, that is, the first feature extraction module (Conv2-1) and the first attention mechanism module (CBAM2-1) in the first feature extraction layer 2 are at the same level as the second feature fusion module 4 (C2f4-3). Similarly, the second feature extraction layer 1 and the second feature fusion module 3 are at the same level, that is, the first feature fusion module (C2f1-1) and the second attention mechanism module (CBAM1-2) in the second feature extraction layer 1 are at the same level as the second feature fusion module 3 (C2f3-3).

[0087] In some examples, such as Figure 4 As shown, the first feature fusion module (C2f3-1) includes a second feature extraction module (Conv1), a dual-channel feature separation module (Chunk), a third feature extraction module (Conv2), a fourth feature extraction module (Conv3), a fifth feature extraction module (Conv4), a sixth feature extraction module (Conv5), a channel number superposition module (Concat), a seventh feature extraction module (Conv6) and a third attention mechanism module (CBAM) for enhancing the performance of convolutional neural networks.

[0088] It should be noted that the above example is an example of the network structure of the first feature fusion module (C2f3-1). Since the network structures of the first feature fusion module (C2f1-1), the first feature fusion module (C2f2-1), the second feature fusion module 1 (C2f1-3), the second feature fusion module 2 (C2f2-3), the second feature fusion module 3 (C2f3-3), the second feature fusion module 4 (C2f4-3), and the second feature fusion module 5 (C2f5-3) are the same as the network structure of the first feature fusion module (C2f3-1), they will not be repeated here.

[0089] In some examples, in the decoder, the output of the decoder is equal to the sum of the first actual output of the target attention mechanism module of the encoder (including the first attention mechanism module and the second attention mechanism module) and the second actual output determined by the decoder based on the output of the encoder. Figure 3, the output of the decoder is equal to the sum of the output of the first attention mechanism module in the first feature extraction layer 1 of the encoder, the output of the first attention mechanism module in the first feature extraction layer 2 of the encoder, the output of the first attention mechanism module in the second feature extraction layer 1 of the encoder, the output of the first attention mechanism module in the second feature extraction layer 2 of the encoder, and the second actual output determined by the decoder based on the output of the first attention mechanism module in the second feature extraction layer 3 of the encoder.

[0090] Specifically, the second feature fusion module 1 of the decoder determines the first processing result based on the output of the first attention mechanism module in the second feature extraction layer 3 of the encoder. Afterwards, the second feature fusion module 2 of the decoder determines the second processing result based on the first processing result. The second feature fusion module 2 of the decoder determines the third processing result based on the second processing result and the sum of the output of the first attention mechanism module in the second feature extraction layer 2 of the encoder. Afterwards, the second feature fusion module 3 of the decoder determines the fourth processing result based on the third processing result. The second feature fusion module 3 of the decoder determines the fifth processing result based on the fourth processing result and the sum of the output of the first attention mechanism module in the second feature extraction layer 1 of the encoder. Afterwards, the second feature fusion module 4 of the decoder determines the sixth processing result based on the fifth processing result. The second feature fusion module 4 of the decoder determines the seventh processing result based on the sixth processing result and the sum of the output of the first attention mechanism module in the first feature extraction layer 2 of the encoder. Afterwards, the second feature fusion module 5 of the decoder determines the eighth processing result based on the seventh processing result. The second feature fusion module 5 of the decoder determines the ninth processing result based on the sum of the eighth processing result and the output of the first attention mechanism module in the first feature extraction layer 1 of the encoder. Among them, the ninth processing result is the output of the decoder. In this way, the pixel superposition method add operation can be used instead of the original channel increase method concat to reduce the number of parameters of the U-Net model.

[0091] Exemplarily, assuming that the eighth processing result is [1, 2, 3, 4], the output of the first attention mechanism module in the first feature extraction layer 1 of the encoder is [1, 0, 0, 1], the sum of the eighth processing result and the output of the first attention mechanism module in the first feature extraction layer 1 of the encoder is equal to [2, 2, 3, 5], that is, the ninth processing result is [2, 2, 3, 5].

[0092] It should be noted that the above example is explained by taking the calculation process of calculating the sum of the eighth processing result and the output of the first attention mechanism module in the first feature extraction layer 1 of the encoder as an example. The calculation process of calculating the sum of the second processing result and the output of the first attention mechanism module in the second feature extraction layer 2 of the encoder, calculating the sum of the fourth processing result and the output of the first attention mechanism module in the second feature extraction layer 1 of the encoder, and calculating the sum of the sixth processing result and the output of the first attention mechanism module in the first feature extraction layer 2 of the encoder is the same as the calculation process of calculating the sum of the eighth processing result and the output of the first attention mechanism module in the first feature extraction layer 1 of the encoder, and will not be repeated here.

[0093] In some possible examples, the foreign object removal device includes a wiper; Figure 1 ,like Figure 5 As shown, the above S13 can be specifically implemented through the following S130-S134, and the above S14 can be specifically implemented through the following S140.

[0094] S130: when it is determined that there is a segmentation graph that is not empty, and when the image category corresponding to the segmentation graph that is not empty is raindrops, determining a first actual area of ​​a minimum enclosing region that includes all raindrops in the segmentation graph;

[0095] S131, determining a first actual density based on a ratio of the first actual area to a second actual area of ​​the windshield;

[0096] S132: Calculate the actual similarity between the segmentation graph of the current video frame image and the segmentation graph of the previous video frame image.

[0097] In some examples, the cosine similarity between the segmentation graph of the current video frame image and the segmentation graph of the previous video frame image can be used as the actual similarity. Alternatively, the Euclidean distance between the segmentation graph of the current video frame image and the segmentation graph of the previous video frame image can be used as the actual similarity.

[0098] S133. Determine a first theoretical parameter based on the first actual density and the actual similarity.

[0099] In some examples, different first actual densities and actual similarities correspond to different first theoretical parameters, such as a first parameter relationship table between the first actual density, the actual similarity, and the first theoretical parameter is pre-stored in the memory of the vehicle. Afterwards, the vehicle controller can query the first actual density and the actual similarity based on the first parameter relationship table in the memory to determine the first theoretical parameter corresponding to the first actual density and the actual similarity.

[0100] Alternatively, a preset parameter table of numerical intervals and first theoretical parameters is pre-stored in the memory of the vehicle. Afterwards, the vehicle controller can obtain the first numerical value based on the preset parameter table in the memory of the vehicle and the product of the first actual density and the actual similarity. The preset parameter table is queried based on the first numerical value to determine the first theoretical parameter corresponding to the numerical interval into which the first numerical value falls.

[0101] S134. Using the first theoretical parameter as a control parameter corresponding to when the image category is raindrops.

[0102] S140: Control the wipers to operate according to control parameters to remove foreign matter from the windshield.

[0103] In some possible examples, the foreign object removal device includes a wiper; Figure 1 ,like Figure 6 As shown, the above S13 can be specifically implemented through the following S135-S138, and the above S14 can be specifically implemented through the following S140.

[0104] S135. When it is determined that there is a segmentation graph that is not empty, and when the image category corresponding to the segmentation graph that is not empty is snowflakes, determine a third actual area of ​​the minimum enclosing region that contains all snowflakes in the segmentation graph.

[0105] S136: Determine a second actual density based on a ratio of the second actual area to the second actual area of ​​the windshield.

[0106] S137. Determine a second theoretical parameter based on the second actual density.

[0107] In some examples, different second actual densities correspond to different second theoretical parameters, such as a second parameter relationship table between the second actual density and the first theoretical parameter is pre-stored in the memory of the vehicle. Afterwards, the vehicle controller can query the second actual density based on the second parameter relationship table in the memory to determine the first theoretical parameter corresponding to the second actual density.

[0108] S138. Use the second theoretical parameter as a control parameter corresponding to the image category of snowflakes.

[0109] S140: Control the wipers to operate according to control parameters to remove foreign matter from the windshield.

[0110] In some embodiments, the foreign object removal device includes a water sprayer and a windshield wiper; Figure 1 ,like Figure 7 As shown, the above S13 can be specifically implemented through the following S139, and the above S14 can be specifically implemented through the following S141.

[0111] S139: When it is determined that there is a segmentation graph that is not empty, and when the image category corresponding to the segmentation graph that is not empty is dust, determine that the control parameter of the sprinkler is the first preset parameter, and the control parameter of the wiper is the second preset parameter.

[0112] In some examples, the first preset parameter may be that the sprinkler is turned on and sprays water for a first duration (e.g., 20 seconds) at intervals of a target duration (e.g., 30 seconds) until the sprinkler is turned off after spraying water three times.

[0113] In some examples, the second preset parameter may be that the wiper is turned on, and the wiper is turned off after running at a preset gear for a specified time (such as 2 minutes).

[0114] S141. Control the sprinkler to operate according to the first preset parameter, and control the wiper to operate according to the second preset parameter.

[0115] Embodiment 2

[0116] Figure 8 The structural diagram of the vehicle controller provided in the second embodiment of the present application is exemplarily shown in FIG. Figure 8 As shown, the vehicle controller includes: an acquisition module 81 and a processing module 82.

[0117] An acquisition module is used to acquire a current video frame image of a vehicle's windshield at a current moment; a processing module is used to perform image segmentation on the current video frame image acquired by the acquisition module, and determine a segmentation map corresponding to each image category; wherein the image category includes any one of raindrops, snowflakes, and dust; the processing module is also used to determine, when it is determined that there is a non-empty segmentation map, based on the image category corresponding to the non-empty segmentation map, the control parameters corresponding to each image category in the image category corresponding to the non-empty segmentation map; wherein the control parameters include control parameters for controlling a foreign object removal device; the processing module is also used to control the foreign object removal device to operate according to the control parameters to remove foreign objects from the windshield; wherein the foreign object removal device includes one or more of a wiper and a sprinkler, and the foreign objects include one or more of raindrops, snowflakes, and dust.

[0118] In some implementable examples, the acquisition module is further used to acquire training sample data and labeling results of the training sample data; wherein the training sample data includes at least one first video frame image of the windshield containing foreign objects and at least one second video frame image that does not contain foreign objects, and the labeling results include whether the first video frame image and the second video frame image contain foreign objects; the processing module is further used to input the training sample data acquired by the acquisition module into the deep learning architecture for learning, and obtain the prediction results of the deep learning architecture for the training sample data; wherein the deep learning architecture includes an encoder and a decoder, the encoder includes at least two first feature extraction layers and at least three second feature extraction layers, the first feature extraction layer includes a first feature extraction module and a first attention mechanism module for enhancing the performance of the convolutional neural network, and the second feature extraction layer includes a first feature fusion module and a second attention mechanism module for enhancing the performance of the convolutional neural network Mechanism module, the decoder includes at least five second feature fusion modules, the first feature extraction module adopts deep separable convolution and suppresses its maximum value activation function, the output of the decoder is equal to the sum of the first actual output of the target attention mechanism module of the encoder and the second actual output determined by the decoder based on the output of the encoder, and the target attention mechanism module includes the first attention mechanism module and the second attention mechanism module; the processing module is also used to calculate the loss value of the prediction result and the labeling result obtained by the acquisition module based on the target loss function; wherein the target loss function includes a binary cross entropy loss introducing a scaling factor; the processing module is also used to determine the convergence of the deep learning architecture when the loss value is less than or equal to the target value, and obtain the image segmentation model; the processing module is specifically used to input the current video frame image obtained by the acquisition module into the image segmentation model for image segmentation, and determine the segmentation map corresponding to each image category.

[0119] In some feasible examples, the first feature extraction module includes a 1×1 first convolution kernel, a 3×3 depthwise separable convolution kernel, and a 1×1 second convolution kernel, the input of the first convolution kernel includes training sample data or the input of the first convolution kernel includes the output of the first feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer, the input of the depthwise separable convolution kernel includes the output of the first convolution kernel, the input of the second convolution kernel includes the output of the depthwise separable convolution kernel, the output of the second convolution kernel includes the output of the feature extraction module, and the activation functions of both the depthwise separable convolution kernel and the second convolution kernel are activation functions that suppress their maximum values.

[0120] In some feasible examples, the input of the first feature fusion module includes the output of the feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer, the input of the second feature fusion module includes the output of the first attention mechanism module in the target extraction layer of the same layer as the second feature fusion module and the output of the second feature fusion module of the previous layer, the target extraction layer includes any one of the first feature extraction layer and the second feature extraction layer, or the input of the second feature fusion module includes the output of the first attention mechanism module in the second feature extraction layer of the same level as the second feature fusion module, the output of the first feature fusion module includes the input of the first feature fusion module in the first feature extraction layer of the next layer, or the output of the first feature fusion module includes the input of the first attention mechanism module of the same level as the first feature fusion module, and the output of the second feature fusion module includes the input of the second feature fusion module of the next layer.

[0121] In some implementable examples, the first feature fusion module and the second feature fusion module both include a second feature extraction module, a dual-channel feature separation module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, a sixth feature extraction module, a channel number superposition module, a seventh feature extraction module and a third attention mechanism module for enhancing the performance of the convolutional neural network; wherein the input of the second feature extraction module includes the output of the feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer, or the input of the second feature extraction module includes the output of the first attention mechanism module in the first feature extraction layer of the previous layer and the output of the second feature fusion module of the previous layer, or the input of the second feature extraction module includes the output of the first attention mechanism module in the second feature extraction layer of the same level as the second feature extraction module; the dual-channel feature separation module is used to perform feature separation on the output of the second feature extraction module to obtain a first separation feature and a second separation feature, the first separation feature is the input of the channel number superposition module, the second separation feature is the input of the channel number superposition module, and the second separation feature The feature is the input of the third feature extraction module; the third feature extraction module is used to process the second separation feature output by the dual-channel feature separation module to obtain the first result; the fourth feature extraction module is used to process the first result to obtain the second result; the fifth feature extraction module is used to process the second result output by the fourth feature extraction module to obtain the third result; the sixth feature extraction module is used to process the third result output by the fifth feature extraction module to obtain the fourth result; the channel number superposition module is used to perform feature splicing on the first separation feature and the second separation feature output by the dual-channel feature separation module and the fourth result output by the sixth feature extraction module to obtain the fifth result; the seventh feature extraction module is used to process the fifth result output by the channel number superposition module to obtain the sixth result; the third attention mechanism module is used to process the sixth result output by the seventh feature extraction module to obtain the seventh result; the seventh result is the input of the first feature fusion module in the first feature extraction layer of the next layer, or the seventh result is the input of the first attention mechanism module at the same level as the first feature fusion module, and the seventh result is the input of the second feature fusion module of the next layer.

[0122] In some feasible examples, the foreign object removing device includes a wiper; a processing unit, specifically used to determine a first actual area of ​​a minimum enclosing area containing all raindrops in the segmentation map when it is determined that there is a non-empty segmentation map and the image category corresponding to the non-empty segmentation map is raindrops; the processing unit, specifically used to determine a first actual density based on a ratio of the first actual area to a second actual area of ​​the windshield; the processing unit, specifically used to calculate an actual similarity between a segmentation map of a current video frame image and a segmentation map of a previous video frame image; the processing unit, specifically used to determine a first theoretical parameter based on the first actual density and the actual similarity; the processing unit, specifically used to use the first theoretical parameter as a control parameter corresponding to when the image category is raindrops; the processing unit, specifically used to control the wiper to operate according to the control parameter to remove foreign objects from the windshield.

[0123] In some feasible examples, the foreign object removing device includes a wiper; a processing unit, specifically used to determine a third actual area of ​​the minimum enclosing area containing all snowflakes in the segmentation map when it is determined that there is a non-empty segmentation map and the image category corresponding to the non-empty segmentation map is snowflake; the processing unit, specifically used to determine a second actual density based on a ratio of the second actual area and the second actual area of ​​the windshield; the processing unit, specifically used to determine a second theoretical parameter based on the second actual density; the processing unit, specifically used to use the second theoretical parameter as a control parameter corresponding to the image category being snowflake; the processing unit, specifically used to control the wiper to operate according to the control parameter to remove foreign objects from the windshield.

[0124] In some feasible examples, the foreign matter removal device includes a water sprayer and a wiper; a processing unit is specifically used to determine, when it is determined that there is a non-empty segmentation map, based on the image category corresponding to the non-empty segmentation map, the control parameters corresponding to each image category in the image category corresponding to the non-empty segmentation map, including: a processing unit is specifically used to determine that the control parameter of the water sprayer is a first preset parameter, and the control parameter of the wiper is a second preset parameter when it is determined that there is a non-empty segmentation map and the image category corresponding to the non-empty segmentation map is dust; a processing unit is specifically used to control the water sprayer to operate according to the first preset parameter, and control the wiper to operate according to the second preset parameter.

[0125] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and its role will not be repeated here.

[0126] Of course, the vehicle controller provided by the embodiment of the present invention includes but is not limited to the above modules, for example, the vehicle controller may also include a storage module 83. The storage module 83 may be used to store the program code of the vehicle controller, and may also be used to store data generated during the operation of the vehicle controller, such as diagnostic data.

[0127] Fig. 9 A schematic diagram of the structure of a vehicle controller provided by an embodiment of the present invention is shown in FIG. Fig. 9 As shown, the vehicle controller may include: at least one processor 51 , a memory 52 , a communication interface 53 and a communication bus 54 .

[0128] Combine the following Fig. 9 A detailed introduction to the various components of the vehicle controller:

[0129] The processor 51 is the control center of the vehicle controller, and may be a processor or a general term for multiple processing elements. For example, the processor 51 is a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more DSPs, or one or more field programmable gate arrays (FPGAs).

[0130] In a specific implementation, as an embodiment, the processor 51 may include one or more CPUs, such as Fig. 9 Also, as an embodiment, the vehicle controller may include multiple processors, such as Fig. 9 51 and 55 are shown in FIG. Each of these processors may be a single-core processor (Single-CPU) or a multi-core processor (Multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0131] The memory 52 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 52 may exist independently and be connected to the processor 51 via a communication bus 54. The memory 52 may also be integrated with the processor 51.

[0132] In a specific implementation, the memory 52 is used to store the data of the present invention and execute the software program of the present invention. The processor 51 can execute various functions of the air conditioner by running or executing the software program stored in the memory 52 and calling the data stored in the memory 52.

[0133] The communication interface 53 uses any transceiver-like device to communicate with other devices or communication networks, such as Radio Access Network (RAN), Wireless Local Area Networks (WLAN), terminals, the cloud, etc. The communication interface 53 may include an acquisition module to implement the acquisition function.

[0134] The communication bus 54 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0135] As an example, combining Figure 8The functions implemented by the acquisition module 81 of the vehicle controller are similar to Fig. 9 The function of the communication interface 53 in the vehicle controller is the same as that of the processing module 82 in the vehicle controller. Fig. 9 The functions of the processor 51 in the vehicle controller are the same as those of the storage module 83 in the vehicle controller. Fig. 9 The function of the memory 52 in is the same.

[0136] An embodiment of the present application also provides a vehicle, which may include the vehicle controller in any embodiment.

[0137] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method in any embodiment.

[0138] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A control method for removing foreign matter equipment, characterized in that: include: Obtain the current video frame image of the vehicle's windshield at the current moment; Performing image segmentation on the current video frame image to determine a segmentation map corresponding to each image category; wherein the image category includes any one of raindrops, snowflakes, and dust; When it is determined that there is a segmentation graph that is not empty, based on the image category corresponding to the segmentation graph that is not empty, determine the control parameters corresponding to each image category in the image category corresponding to the segmentation graph that is not empty; wherein the control parameters include control parameters for controlling the foreign matter removing device; The foreign object removal device is controlled to operate according to the control parameters to remove foreign objects from the windshield; wherein the foreign object removal device includes one or more of a wiper and a sprinkler, and the foreign objects include one or more of raindrops, snowflakes, and dust.

2. The control method of the foreign matter removal device according to claim 1, characterized in that: Before obtaining the current video frame image of the windshield of the vehicle at the current moment, the method further includes: Acquire training sample data and labeling results of the training sample data; wherein the training sample data includes at least one first video frame image of the windshield containing foreign matter and at least one second video frame image not containing foreign matter, and the labeling results include whether the first video frame image and the second video frame image contain foreign matter; The training sample data is input into a deep learning architecture for learning, and a prediction result of the deep learning architecture on the training sample data is obtained; wherein the deep learning architecture includes an encoder and a decoder, the encoder includes at least two first feature extraction layers and at least three second feature extraction layers, the first feature extraction layer includes a first feature extraction module and a first attention mechanism module for enhancing the performance of a convolutional neural network, the second feature extraction layer includes a first feature fusion module and a second attention mechanism module for enhancing the performance of a convolutional neural network, the decoder includes at least five second feature fusion modules, the first feature extraction module uses a depthwise separable convolution and suppresses its maximum value activation function, the output of the decoder is equal to the sum of a first actual output of a target attention mechanism module of the encoder and a second actual output determined by the decoder based on the output of the encoder, and the target attention mechanism module includes a first attention mechanism module and a second attention mechanism module; Calculating loss values ​​of the prediction result and the labeling result based on a target loss function; wherein the target loss function includes a binary cross entropy loss that introduces a scaling factor; When the loss value is less than or equal to the target value, determining that the deep learning architecture converges to obtain an image segmentation model; The performing image segmentation on the current video frame image to determine a segmentation map corresponding to each image category includes: The current video frame image is input into the image segmentation model for image segmentation, and a segmentation map corresponding to each image category is determined.

3. The control method of the foreign matter removal device according to claim 2, characterized in that: The first feature extraction module includes a 1×1 first convolution kernel, a 3×3 depthwise separable convolution kernel, and a 1×1 second convolution kernel. The input of the first convolution kernel includes training sample data or the input of the first convolution kernel includes the output of the first feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer. The input of the depthwise separable convolution kernel includes the output of the first convolution kernel, the input of the second convolution kernel includes the output of the depthwise separable convolution kernel, the output of the second convolution kernel includes the output of the feature extraction module, and the activation functions of both the depthwise separable convolution kernel and the second convolution kernel are activation functions that suppress their maximum values.

4. The control method of the foreign matter removal device according to claim 2, characterized in that: The input of the first feature fusion module includes the output of the feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer. The input of the second feature fusion module includes the output of the first attention mechanism module in the target extraction layer of the same layer as the second feature fusion module and the output of the second feature fusion module of the previous layer. The target extraction layer includes any one of the first feature extraction layer and the second feature extraction layer, or the input of the second feature fusion module includes the output of the first attention mechanism module in the second feature extraction layer of the same level as the second feature fusion module, and the output of the first feature fusion module includes the input of the first feature fusion module in the first feature extraction layer of the next layer, or the output of the first feature fusion module includes the input of the first attention mechanism module of the same level as the first feature fusion module, and the output of the second feature fusion module includes the input of the second feature fusion module of the next layer.

5. The control method of the foreign matter removal device according to claim 2, characterized in that: The first feature fusion module and the second feature fusion module both include a second feature extraction module, a dual-channel feature separation module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, a sixth feature extraction module, a channel number superposition module, a seventh feature extraction module and a third attention mechanism module for enhancing the performance of the convolutional neural network; wherein the input of the second feature extraction module includes the output of the feature extraction module in the first feature extraction layer of the previous layer and the output of the first attention mechanism module in the first feature extraction layer of the previous layer, or the input of the second feature extraction module includes the output of the first attention mechanism module in the first feature extraction layer of the previous layer and the output of the second feature fusion module of the previous layer, or the input of the second feature extraction module includes the output of the first attention mechanism module in the second feature extraction layer of the same level as the second feature extraction module; the dual-channel feature separation module is used to perform feature separation on the output of the second feature extraction module to obtain a first separation feature and a second separation feature, the first separation feature is the input of the channel number superposition module, the second separation feature is the input of the channel number superposition module, and the second separation feature is the output of the third feature extraction module Input; the third feature extraction module is used to process the second separation feature output by the dual-channel feature separation module to obtain a first result; the fourth feature extraction module is used to process the first result to obtain a second result; the fifth feature extraction module is used to process the second result output by the fourth feature extraction module to obtain a third result; the sixth feature extraction module is used to process the third result output by the fifth feature extraction module to obtain a fourth result; the channel number superposition module is used to perform feature splicing on the first separation feature and the second separation feature output by the dual-channel feature separation module and the fourth result output by the sixth feature extraction module to obtain a fifth result; the seventh feature extraction module is used to process the fifth result output by the channel number superposition module to obtain a sixth result; the third attention mechanism module is used to process the sixth result output by the seventh feature extraction module to obtain a seventh result; the seventh result is the input of the first feature fusion module in the first feature extraction layer of the next layer, or the seventh result is the input of the first attention mechanism module at the same level as the first feature fusion module, and the seventh result is the input of the second feature fusion module of the next layer.

6. The control method of the foreign matter removal device according to claim 1, characterized in that: The foreign matter removal device includes a windshield wiper; The method of determining, when it is determined that there is a segmentation graph that is not empty, based on the image category corresponding to the segmentation graph that is not empty, a control parameter corresponding to each image category in the image category corresponding to the segmentation graph that is not empty comprises: When it is determined that there is a segmentation map that is not empty, and when the image category corresponding to the segmentation map that is not empty is raindrops, determining a first actual area of ​​a minimum enclosing region that contains all raindrops in the segmentation map; determining a first actual density based on a ratio of the first actual area to a second actual area of ​​the windshield; Calculate the actual similarity between the segmentation map of the current video frame image and the segmentation map of the previous video frame image; Determining a first theoretical parameter based on the first actual density and the actual similarity; Using the first theoretical parameter as a control parameter corresponding to the image category being raindrops; The controlling the foreign matter removing device to operate according to the control parameters to remove foreign matter from the windshield includes: The wiper is controlled to operate according to the control parameters to remove foreign matter from the windshield.

7. The control method of the foreign matter removal device according to claim 1, characterized in that: The foreign matter removal device includes a windshield wiper; The method of determining, when it is determined that there is a segmentation graph that is not empty, based on the image category corresponding to the segmentation graph that is not empty, a control parameter corresponding to each image category in the image category corresponding to the segmentation graph that is not empty comprises: When it is determined that there is a segmentation graph that is not empty, and the image category corresponding to the segmentation graph that is not empty is snowflakes, determining a third actual area of ​​the minimum enclosing area that contains all snowflakes in the segmentation graph; determining a second actual density based on a ratio of the second actual area to a second actual area of ​​the windshield; Based on the second actual density, determining a second theoretical parameter; The second theoretical parameter is used as a control parameter corresponding to the image category of snowflakes; The controlling the foreign matter removing device to operate according to the control parameters to remove foreign matter from the windshield includes: The wiper is controlled to operate according to the control parameters to remove foreign matter from the windshield.

8. The control method of the foreign matter removal device according to claim 1, characterized in that: The foreign matter removal equipment includes a water sprayer and a wiper; The method of determining, when it is determined that there is a segmentation graph that is not empty, based on the image category corresponding to the segmentation graph that is not empty, a control parameter corresponding to each image category in the image category corresponding to the segmentation graph that is not empty comprises: When it is determined that there is a segmentation graph that is not empty, and the image category corresponding to the segmentation graph that is not empty is dust, determining that the control parameter of the water sprinkler is the first preset parameter, and the control parameter of the wiper is the second preset parameter; The controlling the foreign matter removing device to operate according to the control parameters to remove foreign matter from the windshield includes: The sprinkler is controlled to operate according to the first preset parameters, and the wiper is controlled to operate according to the second preset parameters.

9. A vehicle controller, characterized in that: include: An acquisition module, used to acquire a current video frame image of the windshield of the vehicle at a current moment; A processing module, configured to perform image segmentation on the current video frame image acquired by the acquisition module, and determine a segmentation map corresponding to each image category; wherein the image category includes any one of raindrops, snowflakes, and dust; The processing module is further used to determine, when it is determined that there is a non-empty segmentation graph, based on the image category corresponding to the non-empty segmentation graph, a control parameter corresponding to each image category in the image category corresponding to the non-empty segmentation graph; wherein the control parameter includes a control parameter for controlling the foreign matter removing device; The processing module is also used to control the foreign object removal device to operate according to the control parameters to remove foreign objects from the windshield; wherein the foreign object removal device includes one or more of a wiper and a sprinkler, and the foreign objects include one or more of raindrops, snowflakes, and dust.

10. A vehicle, characterized in that: The vehicle includes the vehicle controller as claimed in claim 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images