Image acquisition method, obstacle detection method, related device, vehicle and medium

Abstract

The invention relates to an image acquisition method, an obstacle detection method, a related device, a vehicle and a medium. The image acquisition method comprises the following steps: determining that a windscreen wiper of a vehicle is started; when a camera arranged in the vehicle collects a current frame of scene image, the current exposure position of the camera in the current frame of scene image and the current position of the windscreen wiper in the current frame of scene image are determined; and adjusting the sweeping state of the windscreen wiper according to the current exposure position and the current position of the windscreen wiper in the current frame scene image so as to reduce the area of the windscreen wiper in the current frame scene image. Thus, the influence of the windscreen wiper on the image can be reduced, the accuracy of image acquisition is improved, and then the precision of obstacle detection and the safety of automatic driving are improved. In addition, the rain and snow on the windshield can be removed by starting the windscreen wiper, so that the image acquisition accuracy of the camera arranged in the vehicle is further improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and particularly to an image acquisition method, an obstacle detection method, related devices, a vehicle, and a medium. Background Art

[0002] With the development of computer technology and the automotive industry, autonomous driving technology has gradually become a current research hotspot. An autonomous driving system needs to perform panoramic perception of the surrounding environment, including the detection of obstacles to help the vehicle avoid obstacles. At the same time, it is necessary to detect lane lines and drivable areas and provide the decision-making system with information including: the position of obstacles, whether the road is drivable, the position information of lane lines, etc. This is the key to planning the vehicle driving route.

[0003] Currently, obstacle detection mainly calculates the occupancy state of grids in three-dimensional space based on scene images, that is, the position of obstacles and corresponding other information, such as the category and moving speed of obstacles. However, most current obstacle detection schemes mainly target normal weather and use the scene images of vehicle driving obtained in normal weather to detect obstacles. Among them, the quality of the scene images collected by the camera in normal weather is relatively high, that is, the accuracy of image acquisition can be ensured in normal weather. In abnormal weather, such as rainy and snowy days, raindrops or snow on the front windshield and the windshield wipers that clear raindrops or snow on the front windshield will affect the quality of the collected images, resulting in poor quality of the scene images collected by the camera in abnormal weather, that is, the accuracy of image acquisition is relatively low in abnormal weather, thus affecting the accuracy of obstacle detection and the safety of autonomous driving. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides an image acquisition method, an obstacle detection method, related devices, a vehicle, and a medium.

[0005] According to the first aspect of the embodiments of the present disclosure, an image acquisition method is provided, including:

[0006] Determine that the windshield wiper of the vehicle has been started;

[0007] When the camera disposed in the vehicle acquires the current frame of scene image, determine the current exposure position of the camera in the current frame of scene image and the current position of the windshield wiper in the current frame of scene image;

[0008] Adjust the sweeping state of the windshield wiper according to the current exposure position and the current position of the windshield wiper in the current frame of scene image to reduce the area of the windshield wiper in the current frame of scene image.

[0009] According to a second aspect of the embodiments of the present disclosure, there is provided an obstacle detection method, the method comprising:

[0010] Obtaining a scene image collected by a camera, the scene image being obtained by the image collection method according to any one of the first aspects of the present disclosure;

[0011] Inputting the scene image into a pre-trained detection model to obtain a detection result output by the detection model, the detection result including position information of an obstacle.

[0012] According to a third aspect of the embodiments of the present disclosure, there is provided an image collection device, the image collection device comprising:

[0013] A first determination module configured to determine that a windshield wiper of a vehicle has been started;

[0014] A second determination module configured to determine a current exposure position of the camera in the current frame scene image and a current position of the windshield wiper in the current frame scene image when the camera disposed in the vehicle is collecting the current frame scene image;

[0015] An adjustment module configured to adjust a sweeping state of the windshield wiper according to the current exposure position and the current position of the windshield wiper in the current frame scene image to reduce an area of the windshield wiper in the current frame scene image.

[0016] According to a fourth aspect of the embodiments of the present disclosure, there is provided an obstacle detection device, the obstacle detection device comprising:

[0017] An acquisition module configured to acquire a scene image collected by a camera, the scene image being obtained by the image collection method according to any one of the first aspects of the present disclosure;

[0018] An input module configured to input the scene image into a pre-trained detection model to obtain a detection result output by the detection model, the detection result including position information of an obstacle.

[0019] According to a fifth aspect of the embodiments of the present disclosure, there is provided a vehicle, comprising:

[0020] A windshield wiper, a camera disposed in the vehicle;

[0021] A processor;

[0022] A memory for storing executable instructions of the processor;

[0023] Wherein, when the processor is configured to execute the executable instructions, it implements the steps of the image collection method according to any one of the first aspects of the present disclosure, or implements the steps of the obstacle detection method according to the second aspect of the present disclosure.

[0024] According to a sixth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, and when the program instructions are executed by a processor, the steps of the image acquisition method described in any one of the first aspects of the present disclosure are implemented, or the steps of the obstacle detection method described in the second aspect of the present disclosure are implemented.

[0025] By adopting the above technical solution, according to the current exposure position of the camera in the current frame scene image and the current position of the windshield wiper in the current frame scene image, the sweeping state of the windshield wiper is adjusted to reduce the number of pixel rows affected by the windshield wiper in the image, and further reduce the area of the windshield wiper in the current frame scene image. In this way, the influence of the windshield wiper on the image can be reduced, the accuracy of image acquisition can be improved, and further the accuracy of obstacle detection and the safety of autonomous driving can be improved. In addition, since starting the windshield wiper can clear rain and snow on the windshield, the accuracy of image acquisition by the camera installed in the vehicle is further improved.

[0026] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0028] Figure 1 is a scene image of a vehicle driving in a rainy day scene generated by the simulator Diffusion.

[0029] Figure 2A is a scene image of a vehicle driving in a rainy day scene collected by the camera.

[0030] Figure 2B is the algorithm based on the current latest neural network for Figure 2A the image obtained after denoising the shown scene image.

[0031] Figure 3A and 3B are scene images collected by the camera when using the windshield wiper.

[0032] Figure 4 is a flowchart of an image acquisition method shown according to an exemplary embodiment.

[0033] Figure 5 is a schematic diagram of the position and exposure position of a windshield wiper in a current frame scene image shown according to an exemplary embodiment.

[0034] Figure 6It is a schematic diagram showing the position of another windshield wiper and the exposure position in the current frame scene image according to an exemplary embodiment.

[0035] Figure 7 It is a schematic diagram showing the position of another windshield wiper and the exposure position in the current frame scene image according to an exemplary embodiment.

[0036] Figure 8 It is a flowchart of an obstacle detection method according to an exemplary embodiment.

[0037] Figure 9 It is a block diagram of an image acquisition device according to an exemplary embodiment.

[0038] Figure 10 It is a block diagram of an obstacle detection device according to an exemplary embodiment.

[0039] Figure 11 It is a block diagram of a vehicle according to an exemplary embodiment. Detailed implementation manners

[0040] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0041] It should be noted that all actions of obtaining signals, information, or data in this application are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where it is located and obtaining the authorization given by the owner of the corresponding device.

[0042] In the related art, the accuracy of obstacle detection in abnormal weather is mainly improved through the following two methods. The first is data-driven. Its main approach is to collect more data under various weather conditions, or generate simulated data under abnormal conditions through simulation means. Let the neural network "learn" more about these situations to increase the robustness of the algorithm. The data-driven method can effectively enhance the stability of the obstacle detection algorithm in rainy weather scenarios. However, there are also many problems. Among them, the resources and funds required for collecting and annotating data are very large, and the simulation effect is difficult to conform to the real scenario. Figure 1 It is a scene image of a vehicle driving in a rainy weather scenario generated by the simulator Diffusion.

[0043] The second is denoising. The main approach is to use algorithms to calculate which information is noise, such as raindrops, snowflakes, fog, etc., and then remove it. The advantage of this type of method is that it does not require as large a data volume as the above-mentioned method. However, whether it is a heuristic-based algorithm or an algorithm optimized based on statistical learning, it is very difficult to completely remove noise. Moreover, after the noise is removed, the useful information is not enhanced, and even useful information may be accidentally deleted. Figure 2A is a scene image of a vehicle driving in a rainy day scene collected by a camera, Figure 2B and Figure 2A is an image obtained by denoising the scene image shown above using the algorithm of the latest neural network. Referring to Figure 2B , using the algorithm of the latest neural network for denoising, it is still impossible to completely retain the image information while removing raindrops.

[0044] To avoid raindrops, fog, snowflakes, etc. in the scene image collected by the camera, in abnormal weather, the raindrops, fog, and snowflakes on the windshield can be wiped off by the windshield wiper. However, the appearance of the windshield wiper will affect the visual signal. Figure 3A and 3B are the scene images collected by the camera when using the windshield wiper. As shown in Figure 3A and Figure 3B , starting the windshield wiper will briefly clear the raindrops on the front windshield. However, if the frequency of the windshield wiper is low, it will cause the raindrops on the front windshield to stay for a long time, affecting the image collected by the camera. On the contrary, if the frequency of the windshield wiper is too high, a large amount of camera input signals will be blocked by the windshield wiper, that is, it will affect the image collected by the camera. Therefore, in rainy and snowy weather, although starting the windshield wiper can clear the raindrops on the front windshield, the windshield wiper attached to the front windshield will also affect the image collected by the camera, resulting in a poor quality of the collected scene image.

[0045] Thus, using the solutions in the related art, the accuracy of image collection is low, and a relatively accurate image cannot be obtained. Furthermore, when performing obstacle detection based on this image, the detection accuracy is poor, resulting in low safety of autonomous driving.

[0046] In view of this, the present disclosure provides an image collection method, an obstacle detection method, related devices, a vehicle, and a medium to solve the influence of the windshield wiper on the image collected by the camera, improve the accuracy of image collection, improve the quality of the collected image, and further improve the accuracy of obstacle detection and the safety of autonomous driving.

[0047] Figure 4 is a flowchart of an image collection method shown according to an exemplary embodiment. This method can be applied to an electronic device with processing capabilities, such as an in-vehicle processor, a controller, a microcontroller unit (MCU), etc. As shown inFigure 1 As shown, the method may include the following steps:

[0048] In step S41, it is determined that the windshield wiper of the vehicle has been activated.

[0049] Exemplarily, the micro control unit may determine whether the windshield wiper has been activated by detecting the state of the windshield wiper switch. If the windshield wiper switch is in the closed state, it is determined that the windshield wiper has been activated. If the windshield wiper switch is in the open state, it is also determined that the windshield wiper has been activated.

[0050] It should be understood that when the windshield wiper is working, on the one hand, the windshield wiper blocks the camera input signal and affects image acquisition. On the other hand, the movement of the windshield wiper also forms a smear in the image, affecting the image quality.

[0051] In step S42, when the camera installed in the vehicle is collecting the current frame scene image, the current exposure position of the camera in the current frame scene image and the current position of the windshield wiper in the current frame scene image are determined.

[0052] Among them, the scene image refers to the image of the vehicle driving scene captured by the camera through the front windshield of the vehicle.

[0053] It should be understood that in the present disclosure, the installation position of the windshield wiper on the vehicle and the position of the camera are both known. Therefore, the relative position between the windshield wiper and the camera is known, and the internal parameters of the camera are known. Therefore, based on the internal and external parameters of the camera, the current position of the windshield wiper in the current frame scene image can be determined. Among them, the external parameter of the camera is the relative position between the windshield wiper and the camera.

[0054] In step S43, according to the current exposure position and the current position of the windshield wiper in the current frame scene image, the sweeping state of the windshield wiper is adjusted to reduce the area of the windshield wiper in the current frame scene image.

[0055] Currently, the cameras equipped in mainstream autonomous driving systems are mainly rolling shutter cameras, which use the rolling shutter exposure method for progressive exposure. The rolling shutter camera has several advantages. The most important one is that it can obtain a faster imaging speed at a lower cost. Its working principle is different from that of the global shutter camera. In the rolling shutter camera, the exposure time of each pixel is asynchronous. Generally, it is the progressive exposure of pixel rows from top to bottom. After the exposure of the last pixel row is completed, the entire image will be output. The cameras used in autonomous vehicles are usually 30Hz, that is, the imaging time of each frame is about 33.3ms.

[0056] The working frequency of the windshield wiper is approximately 20 - 60 times per minute, which is not in the same order of magnitude as the exposure time of the rolling shutter camera. Calculated at the highest sweeping speed of the windshield wiper, the camera can capture approximately 30 images within the duration when the windshield wiper sweeps across the entire glass surface. Therefore, it is not easy to synchronize the working frequencies of the windshield wiper and the camera. Among them, the microcontroller unit (MCU) can control the image acquisition of the camera and the operation of the windshield wiper respectively by sending trigger signals. In addition, since the working frequency of the camera is determined by the camera parameters, its working frequency is relatively stable, and it is difficult to change its exposure sequence. While the windshield wiper can control its sweeping state by controlling the motor that drives the sweeping. Therefore, in the present disclosure, by adjusting the sweeping state of the windshield wiper according to the current exposure position and the current position of the windshield wiper in the current frame of the scene image, the camera will not capture the windshield wiper, thereby reducing the area of the windshield wiper in the current frame of the scene image.

[0057] It should be understood that since the windshield wiper has a certain size, and when the windshield wiper sweeps on the windshield, it is neither horizontally placed nor vertically placed, but also has a certain inclination angle. Therefore, in the present disclosure, the position of the windshield wiper in the current frame of the scene image refers to multiple positions of the windshield wiper within the image.

[0058] Adopting the above technical solution, according to the current exposure position of the camera in the current frame of the scene image and the current position of the windshield wiper in the current frame of the scene image, the sweeping state of the windshield wiper is adjusted to reduce the number of pixel rows affected by the windshield wiper in the image, thereby reducing the area of the windshield wiper in the current frame of the scene image. In this way, the influence of the windshield wiper on the image can be reduced, the accuracy of image acquisition can be improved, and further the accuracy of obstacle detection and the safety of autonomous driving can be improved. In addition, since starting the windshield wiper can clear the rain and snow on the windshield, the accuracy of image acquisition by the camera installed in the vehicle is further improved.

[0059] In one embodiment, the exposure position is the exposure pixel row. The above step S43 of adjusting the sweeping state of the windshield wiper according to the current exposure position and the current position of the windshield wiper in the current frame of the scene image may include:

[0060] First, obtain the sweeping speed of the windshield wiper and the exposure speed of the camera.

[0061] Among them, the sweeping speed of the windshield wiper includes the sweeping direction and the sweeping rate, and the sweeping speed of the windshield wiper can be controlled by a corresponding control switch. For example, in a non-autonomous vehicle, the driver can adjust the sweeping speed of the windshield wiper by moving the operating lever where the windshield wiper switch is located up or down. The exposure speed of the camera also includes the exposure direction and the exposure rate. Among them, when the camera uses rolling shutter exposure, the exposure direction is usually downward, and the exposure rate can refer to the number of pixel rows exposed per unit time. The exposure speed of the camera is related to the parameters of the camera, and when the camera is determined, its exposure speed is fixed.

[0062] Next, based on the current exposure position, the exposure speed, the current position of the windshield wiper in the current frame scene image, and the sweeping speed, it is determined whether there is a position in the position of the windshield wiper at the next moment in the current frame scene image that is located in the exposure pixel row of the camera at the next moment.

[0063] In one implementation manner, based on the current exposure position, the current position of the windshield wiper in the current frame scene image, the exposure speed, and the sweeping speed, the position of the windshield wiper at the next moment in the current frame scene image and the exposure pixel row of the camera at the next moment in the current frame scene image are respectively estimated; based on the position of the windshield wiper at the next moment in the current frame scene image and the exposure pixel row of the camera at the next moment in the current frame scene image, it is determined whether there is a position in the position of the windshield wiper at the next moment in the current frame scene image that is located in the exposure pixel row of the camera at the next moment.

[0064] Exemplarily, based on the current exposure position and the exposure speed, the exposure pixel row of the camera at the next moment in the current frame scene image is estimated. And, based on the current position of the windshield wiper in the current frame scene image and the sweeping speed, the position of the windshield wiper at the next moment in the current frame scene image is estimated. For example, the duration between the current moment and the next moment can be determined, and then, based on the product of the duration and the exposure speed and the current exposure position, the exposure pixel row of the camera at the next moment in the current frame scene image is determined. Similarly, based on the product of the duration and the sweeping speed and the current position of the windshield wiper in the current frame scene image, the position of the windshield wiper at the next moment in the current frame scene image is determined. Based on the position of the windshield wiper at the next moment in the current frame scene image and the exposure pixel row of the camera at the next moment in the current frame scene image, it is determined whether there is a position in the position of the windshield wiper at the next moment in the current frame scene image that is located in the exposure pixel row of the camera at the next moment.

[0065] Exemplarily, Figure 5 is a schematic diagram of the position of the windshield wiper in the current frame scene image and the exposure position shown according to an exemplary embodiment. As Figure 5As shown, the horizontal solid-line rectangular box represents the current exposure pixel row, the inclined solid-line rectangular box represents the current position of the wiper in the current-frame scene image. Assuming the exposure direction is downward exposure, the horizontal dashed-line rectangular box represents the exposure pixel row at the next moment. Assuming the sweeping direction of the wiper is upward sweeping, the inclined dashed-line rectangular box represents the position of the wiper at the next moment in the current-frame scene image. In Figure 5 the position of the wiper at the next moment in the current-frame scene image, there is a position located in the exposure pixel row of the camera at the next moment.

[0066] For example, in Figure 5 assuming the coordinate system is the coordinate system shown in Figure 5 it is possible to determine whether there is a position coordinate in the position coordinates of the wiper at the next moment in the current-frame scene image whose ordinate is the same as the ordinate of the exposure pixel row of the camera at the next moment. If so, it is determined that there is a position of the wiper at the next moment in the current-frame scene image located in the exposure pixel row of the camera at the next moment.

[0067] In another implementation, according to the exposure speed and the sweeping speed, determine the target relative distance change amount during the period from the current moment to the next moment; if the minimum difference between the exposure position and the position of the wiper in the current-frame scene image is less than or equal to the target relative distance change amount, it is determined that there is a position of the wiper at the next moment in the current-frame scene image located in the exposure pixel row of the camera at the next moment; if the sweeping direction of the wiper is the same as the exposure direction of the camera, the target relative distance change amount is the minimum relative distance change amount; if the sweeping direction of the wiper is opposite to the exposure direction of the camera, the target relative distance change amount is the maximum relative distance change amount.

[0068] Exemplarily, according to the exposure speed and the sweeping speed, determine the relative speed between the camera exposure and the wiper sweeping. Then, determine the product of the duration from the current moment to the next moment and the relative speed as the relative distance change amount. For example, assume the duration from the current moment to the next moment is T, the exposure direction is downward exposure, the exposure rate is v1, the sweeping direction is upward sweeping, and the sweeping rate is v2. Then the relative speed is v1 + v2, or if the sweeping direction is downward sweeping, the relative speed is v1 - v2. Determine the product of the relative speed and T as the relative distance change amount.

[0069] In the present disclosure, a target relative distance change amount during a period from the current moment to the next moment is determined, where the target relative distance change amount refers to the relative distance change amount between the top of the windshield wiper and the camera exposure pixel row in the current frame image. It should be understood that considering that the windshield wiper has a certain length and the acquisition field of view of the camera is limited, that is, the top of the windshield wiper is not necessarily within the acquisition field of view of the camera. Therefore, the top of the windshield wiper in the current frame image is not necessarily the actual top of the windshield wiper.

[0070] Since the sweeping speeds at different positions on the windshield wiper are different, that is, v2 at different positions on the windshield wiper is different, the determined relative distance change amounts for different positions on the windshield wiper are also different. Among them, in the windshield wiper, the position farther from the bottom of the windshield wiper (the fixed part of the windshield wiper on the vehicle) has a greater corresponding speed. When the exposure direction is the same as the sweeping direction, the relative distance change amount corresponding to this position is the smallest, and when the exposure direction is opposite to the sweeping direction, the relative distance change amount corresponding to this position is the largest.

[0071] In this embodiment, if the minimum difference between the exposure position and the position of the windshield wiper in the current frame scene image is less than or equal to the target relative distance change amount, then at the next moment or during the period from the current moment to the next moment, the camera will capture the windshield wiper during the process of acquiring the current frame image, and thus it is determined that there is a position of the windshield wiper at the next moment that is located in the exposure pixel row of the camera at the next moment.

[0072] It should be understood that in order to simplify the calculation, the position of the windshield wiper in the current frame image can also be defaulted to a horizontal position.

[0073] In this way, it is determined whether there is a position of the windshield wiper at the next moment in the current frame scene image that is located in the exposure pixel row of the camera at the next moment according to the above method.

[0074] Finally, if it is determined that there is a position of the windshield wiper at the next moment in the current frame scene image that is located in the exposure pixel row of the camera at the next moment, the sweeping state of the windshield wiper is adjusted to reduce the area of the windshield wiper in the current frame scene image.

[0075] The following describes the specific implementation manner of adjusting the sweeping state of the windshield wiper.

[0076] In one embodiment, adjusting the sweeping state of the windshield wiper may include: if it is determined that there is a position of the windshield wiper at the next moment in the current frame scene image that is located in the exposure pixel row of the camera at the next moment, then control the windshield wiper to stop sweeping until the positions of the windshield wiper are all within the exposed area of the current frame scene image, and then control the windshield wiper to sweep at the sweeping speed.

[0077] Figure 6It is a schematic diagram showing the position of another wiper in the current frame scene image and the exposure position according to an exemplary embodiment. In Figure 6 , it is assumed that the wiper moves up and down horizontally in the image. The rectangular frame 1 represents the camera exposure position, and the rectangular frame 2 represents the position of the wiper in the current frame scene image. As shown in the left figure in Figure 6 , assuming that the current camera exposure position and the position of the wiper in the current frame scene image are close, and at the next moment, the camera exposure position and the position of the wiper in the current frame scene image coincide, that is, the camera will capture the wiper at the next moment, then control the wiper to stop sweeping until the position of the wiper is within the exposed area of the current frame scene image, and then control the wiper to sweep in the sweeping direction. As shown in the right figure in Figure 6 , the rectangular frame 1 continues to move downwards, and the position of the rectangular frame 2 remains unchanged. In this way, after the rectangular frame 1 passes through the rectangular frame 2, the rectangular frame 2 is within the exposed area of the current frame scene image, and then control the wiper to sweep in its original sweeping direction. Among them, the exposed area of the current frame scene image refers to the area above the rectangular frame 1 in the current frame scene image, and the unexposed area refers to the area below the rectangular frame 1 in the current frame scene image. The area where the rectangular frame 1 is located is recorded as the current exposure position, or the current exposure pixel row.

[0078] Since the sweeping speed of the wiper is much lower than the exposure speed of the camera, therefore, whether the wiper sweeps upwards or downwards, it will not enter the unexposed area of the current frame scene image, and thus the camera will no longer capture the wiper. In this embodiment, controlling the wiper to stop sweeping can avoid the camera capturing the wiper, improve the accuracy of image acquisition, and thus improve the quality of the image captured by the camera. In addition, controlling the wiper to stop sweeping can also avoid the smear problem caused by the sweeping of the wiper, further improving the quality of the image captured by the camera.

[0079] In Figure 6 and subsequent Figure 7 , taking the position of the wiper in the current frame image as the horizontal position by default as an example for description.

[0080] In another embodiment, adjusting the sweeping state of the wiper may include: if it is determined that there is a position in the position of the wiper in the next moment in the current frame scene image that is located in the exposure pixel row of the camera in the next moment, then adjust the sweeping state of the wiper according to the sweeping direction of the wiper.

[0081] Optionally, if it exists and the sweeping direction of the wiper is opposite to the exposure direction of the camera, control the wiper to stop sweeping until the positions of the wiper are all within the exposed area of the current frame scene image, and then control the wiper to sweep in the sweeping direction; and if it exists and the sweeping direction of the wiper is the same as the exposure direction of the camera, control the wiper to sweep in a direction opposite to the sweeping direction for a preset duration, and then control the wiper to sweep in the sweeping direction.

[0082] Exemplarily, if the exposure direction of the camera is downward and the sweeping direction of the wiper is upward, it is determined that the sweeping direction of the wiper is opposite to the exposure direction of the camera. At this time, if it is determined that there is a position of the wiper in the next moment's position in the current frame scene image that is located in the position of the exposure pixel row of the camera in the next moment, control the wiper to stop sweeping until the positions of the wiper are all within the exposed area of the current frame scene image, and then control the wiper to sweep in the original sweeping direction. Refer to Figure 6 , in Figure 6 , when the positions of the wiper are all within the exposed area of the current frame scene image, control the wiper to sweep in its sweeping direction, that is, control the wiper to continue sweeping upward. In this way, during the process of collecting the current frame image, the camera will no longer capture the wiper, and thus an image with higher quality can be collected, improving the accuracy of image collection.

[0083] Another exemplarily, if the exposure direction of the camera is downward and the sweeping direction of the wiper is downward, it is determined that the sweeping direction of the wiper is the same as the exposure direction of the camera. At this time, if it is determined that there is a position of the wiper in the next moment's position in the current frame scene image that is located in the position of the exposure pixel row of the camera in the next moment, control the wiper to sweep in a direction opposite to the sweeping direction for a preset duration, that is, control the wiper to sweep upward for a preset duration, and then control the wiper to sweep downward. Among them, the preset duration can be set according to experience.

[0084] Figure 7 is a schematic diagram of the position of the wiper in the current frame scene image and the exposure position according to another exemplary embodiment. In Figure 7 , both the exposure direction of the camera and the sweeping direction of the wiper are downward. The rectangular frame 1 represents the camera exposure position, and the rectangular frame 2 represents the position of the wiper in the current frame scene image. As Figure 7 shown in the left figure in Figure 7 , at the current moment, the camera exposure position and the position of the wiper in the current frame scene image are close, so at the next moment, the camera exposure position and the position of the wiper in the current frame scene image coincide, that is, at the next moment, the camera will capture the wiper, and then control the wiper to sweep upward. AsFigure 7 As shown in the right figure in

[0085] Then, control the wiper to sweep downward. In this way, the probability that the wiper will be captured by the camera subsequently is further avoided, ensuring that the wiper sweeps only within the exposed area of the current frame of the scene image, avoiding the negative noise brought by the wiper during the image acquisition process, and further improving the accuracy of image acquisition.

[0086] In addition, in the present disclosure, the current frame image collected by the camera can be used for obstacle detection or stored in a driving recorder, and the present disclosure does not make specific limitations on this.

[0087] Based on the same inventive concept, the present disclosure also provides an obstacle detection method. Figure 8 It is a flowchart of an obstacle detection method shown according to an exemplary embodiment. As Figure 8 shown, the obstacle detection method may include the following steps.

[0088] In step S81, obtain the scene image collected by the camera.

[0089] Wherein, the scene image is obtained according to the image acquisition method provided by the present disclosure.

[0090] In step S82, input the scene image into a pre-trained detection model to obtain the detection result output by the detection model, and the detection result includes the position information of the obstacle.

[0091] Wherein, the pre-trained detection model can be a neural network model. When training the detection model, the training ground truth can be obtained through the labeled laser point cloud, and the detection model adopts CNN and transformer algorithms.

[0092] By adopting the above technical solution, the obstacle detection is carried out by using a relatively accurate scene image, which improves the accuracy of obstacle detection and further improves the safety of autonomous driving.

[0093] In the present disclosure, in addition to including the position information of the obstacle, the detection result may further include the category and moving speed of the obstacle. Among them, the obstacle category can be a vehicle, a pedestrian, a building, etc.

[0094] Based on the same inventive concept, the present disclosure also provides an image acquisition device. Figure 9 It is a block diagram of an image acquisition device shown according to an exemplary embodiment. As Figure 9 shown, the image acquisition device 900 may include:

[0095] A first determination module 901, configured to determine that the wiper of the vehicle has been started;

[0096] A second determination module 902, configured to determine a current exposure position of the camera in the current frame scene image and a current position of the windshield wiper in the current frame scene image when the camera disposed in the vehicle captures the current frame scene image;

[0097] An adjustment module 903, configured to adjust a sweeping state of the windshield wiper according to the current exposure position and the current position of the windshield wiper in the current frame scene image, so as to reduce an area of the windshield wiper in the current frame scene image.

[0098] Optionally, the exposure position is an exposure pixel row; the adjustment module 903 may include:

[0099] A first acquisition sub-module, configured to acquire a sweeping speed of the windshield wiper and an exposure speed of the camera;

[0100] A first determination sub-module, configured to determine whether there is a position of the windshield wiper at a next moment in the current frame scene image that is located in an exposure pixel row of the camera at the next moment according to the current exposure position, the exposure speed, the current position of the windshield wiper in the current frame scene image, and the sweeping speed;

[0101] A first adjustment sub-module, configured to adjust the sweeping state of the windshield wiper if there is such a position.

[0102] Optionally, the first adjustment sub-module is configured to: if there is such a position, control the windshield wiper to stop sweeping, and then control the windshield wiper to sweep at the sweeping speed until positions of the windshield wiper are all located in an exposed area of the current frame scene image.

[0103] Optionally, the first adjustment sub-module is configured to: if there is such a position, adjust the sweeping state of the windshield wiper according to a sweeping direction of the windshield wiper.

[0104] Optionally, the first adjustment sub-module is further configured to: if there is such a position and the sweeping direction of the windshield wiper is opposite to an exposure direction of the camera, control the windshield wiper to stop sweeping, and then control the windshield wiper to sweep at the sweeping direction until positions of the windshield wiper are all located in the exposed area of the current frame scene image; and

[0105] if there is such a position and the sweeping direction of the windshield wiper is the same as the exposure direction of the camera, control the windshield wiper to sweep in a direction opposite to the sweeping direction for a preset duration, and then control the windshield wiper to sweep at the sweeping direction.

[0106] Optionally, the first determination sub-module is configured to: respectively estimate the position of the wiper at the next moment in the current frame scene image and the exposure pixel row of the camera at the next moment in the current frame scene image according to the current exposure position, the current position of the wiper in the current frame scene image, the exposure speed, and the sweeping speed;

[0107] Determine whether there is a position in the position of the wiper at the next moment in the current frame scene image that is located in the exposure pixel row of the camera at the next moment according to the position of the wiper at the next moment in the current frame scene image and the exposure pixel row of the camera at the next moment in the current frame scene image.

[0108] Optionally, the first determination sub-module is configured to: determine a target relative distance change amount during the period from the current moment to the next moment according to the exposure speed and the sweeping speed;

[0109] If the minimum difference between the exposure position and the position of the wiper in the current frame scene image is less than or equal to the target relative distance change amount, it is determined that there is a position in the position of the wiper at the next moment in the current frame scene image that is located in the exposure pixel row of the camera at the next moment;

[0110] If the sweeping direction of the wiper is the same as the exposure direction of the camera, the target relative distance change amount is the minimum relative distance change amount; if the sweeping direction of the wiper is opposite to the exposure direction of the camera, the target relative distance change amount is the maximum relative distance change amount.

[0111] Optionally, the camera uses rolling shutter exposure.

[0112] Based on the same inventive concept, the present disclosure also provides an obstacle detection device. Figure 10 It is a block diagram of an obstacle detection device shown according to an exemplary embodiment. As Figure 10 shown, the obstacle detection device 1000 may include:

[0113] An acquisition module 1001, configured to acquire a scene image collected by a camera, where the scene image is obtained by the image acquisition method provided by the present disclosure;

[0114] An input module 1002, configured to input the scene image into a pre-trained detection model to obtain a detection result output by the detection model, where the detection result includes position information of an obstacle.

[0115] Regarding the device in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0116] The present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the image acquisition method provided by the present disclosure are implemented.

[0117] The present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the obstacle detection method provided by the present disclosure are implemented.

[0118] The present disclosure also provides a vehicle, including:

[0119] a windshield wiper and a camera disposed inside the vehicle;

[0120] a processor;

[0121] a memory for storing instructions executable by the processor;

[0122] wherein, when the processor is configured to execute the executable instructions, the steps of the image acquisition method provided by the present disclosure are implemented, or the steps of the obstacle detection method provided by the present disclosure are implemented.

[0123] Figure 11 is a block diagram of a vehicle shown according to an exemplary embodiment. For example, vehicle 600 may be a hybrid vehicle, or a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. Vehicle 600 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle. Vehicle 600 may include a windshield wiper and a camera disposed inside vehicle 600.

[0124] Referring to Figure 11 , vehicle 600 may include various subsystems. For example, the infotainment system 610, the perception system 620, the decision control system 630, the drive system 640, and the computing platform 650. Among them, vehicle 600 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of vehicle 600 may be interconnected by wired or wireless means.

[0125] In some embodiments, the infotainment system 610 may include a communication system, an entertainment system, and a navigation system, etc.

[0126] The perception system 620 may include several sensors for sensing information about the environment around vehicle 600. For example, the perception system 620 may include a global positioning system (the global positioning system may be a GPS system, or a Beidou system, or other positioning systems), an inertial measurement unit (IMU), lidar, millimeter wave radar, ultrasonic radar, and a camera device.

[0127] The decision-making control system 630 may include a computing system, a vehicle controller, a steering system, an accelerator, and a braking system.

[0128] The drive system 640 may include components that provide powered movement for the vehicle 600. In one embodiment, the drive system 640 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of an internal combustion engine, an electric motor, and an air compression engine. The engine is capable of converting the energy provided by the energy source into mechanical energy.

[0129] Some or all functions of the vehicle 600 are controlled by the computing platform 650. The computing platform 650 may include at least one processor 651 and a memory 652. The processor 651 may execute instructions 653 stored in the memory 652.

[0130] The processor 651 may be any conventional processor, such as a commercially available CPU. The processor may also include, for example, a Graphic Process Unit (GPU), a Field Programmable Gate Array (FPGA), a System on Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof.

[0131] The memory 652 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0132] In addition to the instructions 653, the memory 652 may also store data, such as road maps, route information, data on the position, direction, speed, etc. of the vehicle. The data stored in the memory 652 can be used by the computing platform 650.

[0133] In an embodiment of the present disclosure, the processor 651 may execute the instructions 653 to complete the steps of the above-described image acquisition method or the steps of the obstacle detection method.

[0134] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program that can be executed by a programmable device. The computer program has a code portion for executing the above-described image acquisition method or obstacle detection method when executed by the programmable device.

[0135] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the present disclosure. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field of the present disclosure that are not disclosed herein. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0136] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An image acquisition method, characterized in that, Including: Determine that the windshield wiper of the vehicle has been activated; When the camera installed in the vehicle captures the current frame scene image, determine the current exposure position of the camera in the current frame scene image and the current position of the windshield wiper in the current frame scene image; Adjust the sweeping state of the windshield wiper according to the current exposure position and the current position of the windshield wiper in the current frame scene image, so as to reduce the area of the windshield wiper in the current frame scene image.

2. The method according to claim 1, characterized in that, The exposure position is the exposure pixel row; the adjusting the sweeping state of the windshield wiper according to the current exposure position and the current position of the windshield wiper in the current frame scene image includes: Obtain the sweeping speed of the windshield wiper and the exposure speed of the camera; According to the current exposure position, the exposure speed, the current position of the windshield wiper in the current frame scene image, and the sweeping speed, determine whether there is a position in the position of the windshield wiper at the next moment in the current frame scene image that is located in the exposure pixel row of the camera at the next moment; If so, adjust the sweeping state of the windshield wiper.

3. The method according to claim 2, wherein The adjusting the sweeping state of the windshield wiper if so includes: If so, control the windshield wiper to stop sweeping until the positions of the windshield wiper are all within the exposed area of the current frame scene image, and then control the windshield wiper to sweep at the sweeping speed.

4. The method according to claim 2, wherein The adjusting the sweeping state of the windshield wiper if so includes: If so, adjust the sweeping state of the windshield wiper according to the sweeping direction of the windshield wiper.

5. The method according to claim 4, wherein The adjusting the sweeping state of the windshield wiper according to the sweeping direction of the windshield wiper if so includes: If so and the sweeping direction of the windshield wiper is opposite to the exposure direction of the camera, control the windshield wiper to stop sweeping until the positions of the windshield wiper are all within the exposed area of the current frame scene image, and then control the windshield wiper to sweep in the sweeping direction; and If so and the sweeping direction of the windshield wiper is the same as the exposure direction of the camera, control the windshield wiper to sweep in a direction opposite to the sweeping direction for a preset duration, and then control the windshield wiper to sweep in the sweeping direction.

6. The method according to claim 2, wherein The determining whether there is a position in the position of the windshield wiper at the next moment in the current frame scene image that is located in the exposure pixel row of the camera at the next moment according to the current exposure position, the exposure speed, the current position of the windshield wiper in the current frame scene image, and the sweeping speed includes: Estimate the position of the windshield wiper at the next moment in the current frame scene image and the exposure pixel row of the camera at the next moment in the current frame scene image respectively according to the current exposure position, the current position of the windshield wiper in the current frame scene image, the exposure speed, and the sweeping speed. Based on the position of the wiper in the next moment of the current frame scene image and the exposure pixel row of the camera in the next moment of the current frame scene image, determine whether there is a position of the wiper in the next moment of the current frame scene image that is located in the exposure pixel row of the camera in the next moment.

7. The method according to claim 2, wherein The determining whether there is a position of the wiper in the next moment of the current frame scene image that is located in the exposure pixel row of the camera in the next moment according to the current exposure position, the exposure speed, the current position of the wiper in the current frame scene image, and the sweeping speed includes: Determine the target relative distance change amount during the period from the current moment to the next moment according to the exposure speed and the sweeping speed; If the minimum difference between the exposure position and the position of the wiper in the current frame scene image is less than or equal to the target relative distance change amount, determine that there is a position of the wiper in the next moment of the current frame scene image that is located in the exposure pixel row of the camera in the next moment; If the sweeping direction of the wiper is the same as the exposure direction of the camera, the target relative distance change amount is the minimum relative distance change amount; if the sweeping direction of the wiper is opposite to the exposure direction of the camera, the target relative distance change amount is the maximum relative distance change amount.

8. The method according to any one of claims 1-7, characterized in that, The camera uses rolling shutter exposure.

9. An obstacle detection method, characterized in that, The method includes: Obtain a scene image collected by the camera, where the scene image is obtained according to the image collection method described in any one of claims 1-8; Input the scene image into a pre-trained detection model to obtain a detection result output by the detection model, where the detection result includes the position information of the obstacle.

10. An image acquisition device, characterized in that, The image collection device includes: A first determination module configured to determine that the wiper of the vehicle has been started; A second determination module configured to determine the current exposure position of the camera in the current frame scene image and the current position of the wiper in the current frame scene image when the camera disposed in the vehicle collects the current frame scene image; An adjustment module configured to adjust the sweeping state of the wiper according to the current exposure position and the current position of the wiper in the current frame scene image to reduce the area of the wiper in the current frame scene image.

11. An obstacle detection device, characterized in that, The obstacle detection device includes: An acquisition module configured to acquire a scene image collected by the camera, where the scene image is obtained according to the image collection method described in any one of claims 1-8; An input module configured to input the scene image into a pre-trained detection model to obtain a detection result output by the detection model, where the detection result includes the position information of the obstacle.

12. A vehicle, characterized in that, Includes: A wiper, a camera disposed in the vehicle; A processor; A memory for storing instructions executable by the processor; Wherein, when the processor is configured to execute the executable instructions, it implements the steps of the image collection method described in any one of claims 1-8, or implements the steps of the obstacle detection method described in claim 9.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instruction is executed by a processor, it implements the steps of the image acquisition method described in any one of claims 1-8, or implements the steps of the obstacle detection method described in claim 9.

Images