Computing device and method for removing raindrops from video images

Abstract

The present invention provides a method for removing raindrops from a video image. The method comprises the following steps: using a plurality of raindrop images marked in a plurality of rainy scene images to train a raindrop image recognition model; using the raindrop image recognition model to recognize a plurality of raindrop images in a plurality of scene images in a video sequence; and in response to a specific raindrop image in a current scene image among the plurality of scene images satisfying a predetermined condition, replacing the specific raindrop image with an image area corresponding to a specific scene image in the plurality of images preceding the current scene image to generate an output scene image.

Description

Technical Field

[0001] The present invention relates to object recognition, and more particularly to a computing device and a method for removing raindrops from video images. Background Art

[0002] With the advancement of science and technology, the technology of autonomous driving is booming. Autonomous driving software often needs to judge the road conditions based on the video images taken by the camera equipped with the vehicle. However, when the weather is bad, the video images taken by the camera equipped with the vehicle are very easily affected by raindrops, which will reduce the accuracy of the back-end vehicle recognition and road condition recognition, thereby affecting the decision-making of autonomous driving. Summary of the invention

[0003] In view of this, the present invention provides a computing device and a method for removing raindrops from a video image to solve the above-mentioned problem.

[0004] The present invention provides a method for removing raindrops from a video image. The method comprises the following steps: using a plurality of raindrop images marked in a plurality of rainy scene images to train a raindrop image recognition model; using the raindrop image recognition model to recognize a plurality of raindrop images in a plurality of scene images in a video sequence; and in response to a specific raindrop image in a current scene image among the plurality of scene images satisfying a predetermined condition, replacing the specific raindrop image with an image area corresponding to a specific scene image in the plurality of images preceding the current scene image to generate an output scene image.

[0005] In some embodiments, the method further includes: recording the corresponding position and size of each identified raindrop image in the corresponding scene image; and calculating the intersection-and-union ratio of the specific raindrop image in the current scene image and the specific raindrop image in the specific scene image. In addition, the predetermined condition is that the intersection-and-union ratio of the specific raindrop image in the current scene image and the specific raindrop image in the specific scene image is less than a predetermined ratio threshold.

[0006] In some embodiments, the specific raindrop image is selected from a region of interest of the current scene image. The plurality of raindrop images in the current scene image has a first number, and the method further comprises: in response to the number of the plurality of raindrop images that have been successfully replaced reaching a predetermined proportion of the first number, stopping replacing the remaining plurality of raindrop images in the current scene image.

[0007] In some embodiments, the method also includes: when the intersection-and-union ratio of the specific raindrop image in the current scene image and the specific raindrop image in multiple previous scene images in the video sequence is less than a predetermined ratio threshold, selecting the previous scene image with the smallest intersection-and-union ratio as the previous scene image corresponding to the specific raindrop image in a restorable candidate list.

[0008] In some embodiments, the method also includes: when the intersection-and-union ratios of the specific raindrop image in the multiple previous scene images are all less than a predetermined ratio threshold, selecting the previous scene image whose time point is closer to the current scene image as the previous scene image corresponding to the specific raindrop image in a restorable candidate list.

[0009] The present invention also provides a computing device, comprising: a non-volatile memory for storing a raindrop removal program; and a computing unit for executing the raindrop removal program to perform the following steps: using a plurality of raindrop images marked in a plurality of rainy scene images to train a raindrop image recognition model; using the raindrop image recognition model to recognize a plurality of raindrop images in a plurality of scene images in a video sequence; and in response to a specific raindrop image in a current scene image in the plurality of scene images satisfying a predetermined condition, replacing the specific raindrop image with an image area corresponding to a specific scene image in the plurality of images preceding the current scene image to generate an output scene image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram showing a computing device according to an embodiment of the present invention.

[0011] Figure 2 FIG. 4 is a schematic diagram showing a training image for marking raindrops in a rainy scene image according to an embodiment of the present invention.

[0012] Figure 3 FIG. 4 is a schematic diagram showing a training object recognition model according to an embodiment of the present invention.

[0013] Figure 4A FIG. 4 is a schematic diagram of using a raindrop image recognition model to perform raindrop recognition on an input scene image according to an embodiment of the present invention.

[0014] Figure 4B FIG. 4 is a schematic diagram of a raindrop removal process according to an embodiment of the present invention.

[0015] 5A and 5C are schematic diagrams showing current scene images according to an embodiment of the present invention.

[0016] 5B and 5D are schematic diagrams showing output scene images according to an embodiment of the present invention.

[0017] Figure 6 FIG. 4 is a flow chart showing a method for removing raindrops from a video image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following description is a preferred implementation of the invention, which is intended to describe the basic spirit of the invention, but is not intended to limit the invention. The actual content of the invention must refer to the scope of the following claims.

[0019] It must be understood that the words "comprise", "include" and the like used in this specification are used to indicate the existence of specific technical features, values, method steps, operation processing, components and / or components, but do not exclude the addition of more technical features, values, method steps, operation processing, components, components, or any combination of the above.

[0020] The words "first", "second", "third", etc. used in the claims are used to modify the components in the claims and are not used to indicate a priority order, a prior relationship, or that one component precedes another, or a temporal order in performing method steps. They are only used to distinguish components with the same name.

[0021] Figure 1 is a block diagram showing a computing device according to an embodiment of the present invention.

[0022] In one embodiment, the computing device 100 may be implemented by a personal computer, a car computer, a server, or a portable device, and may be installed in a vehicle. The computing device 100 includes a computing unit 110 , an image capture device 120 , a storage unit 130 , and a display 150 .

[0023] The computing unit 110 can be implemented in a variety of ways, such as with a dedicated hardware circuit or general-purpose hardware (e.g., a single processor, a multiprocessor with parallel processing capabilities, a graphics processor, or other processors with computing capabilities), and provides the functions described below when executing program codes or software related to various models and processes of the present invention. The image capture device 120 is, for example, a camera, and the image capture device 120 is used to capture a scene image within a predetermined field of view in front of the vehicle, such as capturing a vehicle window. In some embodiments, the computing device 100 may include a plurality of image capture devices 120, and each image capture device 120 may be disposed at a different predetermined position on the vehicle to obtain scene images of different predetermined perspectives around the vehicle.

[0024] The storage unit 130 includes a volatile memory 131 and a non-volatile memory 132. The non-volatile memory 132 is used to store a raindrop image database 136, as well as data and various program codes required in the object recognition process, such as a raindrop removal program 133, a raindrop image recognition model 135, etc. The non-volatile memory 132 can be, for example, a hard disk drive, a solid-state disk, a flash memory, or a read-only memory, but the present invention is not limited thereto. The volatile memory 131 can be a random access memory, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), but the present invention is not limited thereto. The volatile memory 131 can, for example, temporarily store intermediate data and images in the object recognition process.

[0025] In one embodiment, the non-volatile memory 132 stores a raindrop removal program 133 , and the computing unit 110 reads the raindrop removal program 133 from the non-volatile memory 132 to the volatile memory 131 for execution.

[0026] The display unit 150 may be a display panel (e.g., a thin-film liquid crystal display panel, an organic light-emitting diode panel, or other panels with display capabilities) for displaying a user interface provided by an application. In some embodiments, the display unit 150 may be integrated with a touch device to form a touch panel, and a user may perform touch operations on the user interface displayed on the display unit 150.

[0027] In one embodiment, the non-volatile memory 132 further includes a raindrop image database 136 and a raindrop image recognition model 135. For example, the raindrop image database 136 stores a plurality of raindrop images marked in a plurality of rainy scene images. Each rainy scene image may be, for example, a rainy street view image acquired in advance or taken in real time, and may or may not include vehicles on the street, that is, each rainy scene image includes a plurality of raindrop images, and the user may mark (select) the position and size of each raindrop image in the rainy scene image on the user interface of the raindrop removal program 133.

[0028] Figure 2 FIG. 4 is a schematic diagram showing a training image for marking raindrops in a rainy scene image according to an embodiment of the present invention.

[0029] like Figure 2As shown, the rainy scene image 200 is, for example, pre-stored in the image database 135. For example, the pixel coordinates in the rainy scene image 200 are based on the upper left corner as the origin, and the X axis is positive to the right and the Y axis is positive to the bottom. The coordinates of the upper left corner, the upper right corner, the lower left corner, and the lower right corner of the rainy scene image 200 are (0,0), (1,0), (0,1), (1,1), respectively. The user can first mark the position and size of each raindrop training image 210 in the rainy scene image 200 on the user interface of the raindrop removal program 133, and the raindrop removal program 133 can represent the position and size of each raindrop training image 210 by the horizontal / vertical ratio of each raindrop image 210 in the rainy scene image 200, wherein the horizontal / vertical ratio is a value between 0 and 1.

[0030] After the marking is completed, the raindrop removal program 133 can convert the position and size of each raindrop training image 210 in the rainy scene image 200 into a text file, such as Figure 2 In addition, the raindrop removal program 133 can further crop each raindrop training image 210 marked in the rainy scene image 200 to obtain a separate raindrop training image 210, and store each raindrop training image 210 and corresponding position / size information (e.g., a separate text file) in the raindrop image database 136.

[0031] Next, the user may input each raindrop training image 210 and the corresponding position / size information stored in the raindrop image database 136 into an object recognition model 300 for deep learning. For example, the raindrop image 1.png and the corresponding position / size information 1.txt are the first data, the raindrop image 1.png and the corresponding position / size information 1.txt are the second data, and so on. The object recognition model 300 may be, for example, a convolutional neural network (CNN), a deep neural network, etc., but the present invention is not limited thereto. Those skilled in the art may also use other known object recognition technologies.

[0032] For example, if Figure 3As shown, each raindrop training image 210 (for example, a png file) and the corresponding position / size information (for example, a txt file) in the raindrop image database 136 are input into each bias neuron x1~xr of the input layer 310 in the object recognition model 300, and each bias neuron has a corresponding weight W1~Wr. After the hidden layer 320 or the fully connected layer of the object recognition model 300 performs bias operation and activation function processing on the input layer data, the recognition result can be obtained in the output layer 330.

[0033] After a sufficient amount of training data is input, the object recognition model 300 has been trained to recognize raindrop images. Therefore, the object recognition model 300 can be called a raindrop image recognition model 135 and can be used to recognize raindrop images in each scene image in the video sequence input to the object recognition model 300.

[0034] Figure 4A FIG. 4 is a schematic diagram of using a raindrop image recognition model to perform raindrop recognition on an input scene image according to an embodiment of the present invention. Figure 4B FIG. 4 is a schematic diagram of a raindrop removal process according to an embodiment of the present invention.

[0035] like Figure 4A As shown, it is assumed that the image capture device 120 can capture 5 scene images per second, for example, scene images 401 to 405, and subsequent scene images ( Figure 4A For ease of explanation, only one raindrop image 420 is shown in the scene images 401-405, and the raindrop image 420 moves downward over time in the scene images 401-405, but in actual scene images 401-405, multiple raindrop images may be included. The scene images 401-405 are input into the raindrop image recognition model 400 in the order of shooting as the "current scene image", and the raindrop image recognition model 400 can sequentially recognize the raindrop image 421 in the scene images 401-405. The so-called current scene image represents the image being recognized. For example, if the scene image 403 is currently being recognized, the scene image 403 is the "current scene image", and the previously recognized scene images 401 and 402 are called "previous scene images".

[0036] After the raindrop image recognition model 400 has recognized all the raindrop images in the current scene image, the number and position of each raindrop image 421 in the current scene image can be known, and each raindrop image 421 can be removed according to a predetermined order or randomly selected. It should be noted that the position and size of the original raindrop image 420 in each scene image are close to the position and size of the recognized raindrop image 421. In some embodiments, the raindrop image recognition model 400 can set a stop condition to stop continuing to recognize and replace the raindrop image 421 in each scene image, wherein the above stop condition can be, for example: (1) the raindrop image recognition model 400 has recognized a predetermined number of raindrop images 421 in the current scene image; or (2) the raindrop image recognition model 400 has recognized all the raindrop images 421 (for example, a first number N) in the current scene image, and has successfully removed a predetermined proportion (for example, 0.5) of the first number of raindrop images 421, that is, N / 2 raindrop images 421 have been successfully removed.

[0037] For example, if the scene image 405 is the current scene image, when the raindrop image recognition model 400 has recognized the raindrop image 421 from the scene image 405, the position and size of the raindrop image 421 will be recorded, for example, similar to Figure 2 The position and size information is shown in the lower part. The raindrop removal process 133 then determines whether the identified raindrop image 421 can be replaced by a corresponding area in the previous scene image (ie, the scene images 401-404).

[0038] In detail, the raindrop removal program 133 can, for example, calculate the degree of overlap between the raindrop image in the current scene image (e.g., scene image 405) and the raindrop image 421 in any of the previous scene images (e.g., scene images 401-404), wherein the degree of overlap can be represented by, for example, intersection over union (IoU), which is the ratio of the intersection area to the union area. For example, for ease of explanation, it is assumed that the size of the raindrop image 421 in the current scene image and the previous scene image is A, and the overlap area of ​​the two accounts for 2 / 3 of the area of ​​the raindrop image 421 (i.e., ), indicating that the union area is 4 / 3 of the area of ​​the raindrop image (i.e. ), at this time, the corresponding intersection and union ratio of the raindrop image 421 in the current scene image and the previous scene image is

[0039] However, even if it is the same raindrop, the size of the raindrop image in the scene image at different time points may be different because the vehicle is photographed while traveling. However, through the above method, the raindrop removal program 133 can still calculate the intersection-and-union ratio of the raindrop image 421 in the current scene image and the previous scene image.

[0040] In this embodiment, the raindrop removal program 133 may also set a predetermined ratio threshold (e.g., 0.5) and determine whether the intersection-over-union ratio of the raindrop image 421 is less than the predetermined ratio threshold. When the intersection-over-union ratio is less than the predetermined ratio threshold, it means that the overlap degree of the raindrop image 421 in the current scene image and the previous scene image is small, so the raindrop removal program 133 may determine that the raindrop image 421 is restorable and put the raindrop image 421 into the candidate list of restorable images. In other words, the raindrop removal program 133 may replace the non-overlapping part of the raindrop image in the current scene image and the raindrop image in the previous scene image with the pixels of the corresponding area of ​​the non-overlapping part in the previous scene image. Therefore, all or part of the raindrop image in the current scene image may be replaced by the pixels of the corresponding area in the previous scene image.

[0041] In some embodiments, the raindrop removal program 133 first calculates the intersection-and-union ratio of the raindrop image in the current scene image (e.g., scene image 405, t=n) and the raindrop image 421 in the previous scene image (e.g., scene image 404, t=n-1). If the raindrop removal program 133 determines that the above-mentioned intersection-and-union ratio is less than a predetermined ratio threshold, the raindrop removal program 133 places the raindrop image in the current scene image into the restoreable candidate list and records the corresponding information of the previous scene image 404. If the raindrop removal program 133 determines that the above-mentioned intersection-and-union ratio is greater than or equal to the predetermined ratio threshold, the raindrop removal program 133 will further calculate the intersection-and-union ratio of the raindrop image in the current scene image (e.g., scene image 405, t=n) and the raindrop image 421 in the previous scene image (e.g., scene image 403, t=n-2). Similarly, if the raindrop removal program 133 determines that the above-mentioned intersection-and-union ratio is less than the predetermined ratio threshold, the raindrop removal program 133 places the raindrop image in the current scene image into the restoreable candidate list and records the information of the corresponding previous scene image 403. If the raindrop removal program 133 determines that the above-mentioned intersection-and-union ratio is greater than or equal to the predetermined ratio threshold, the raindrop removal program 133 will calculate the intersection-and-union ratio of the raindrop image in the current scene image (for example, scene image 405, t=n) and the raindrop image 421 in the previous scene image (for example, scene image 402, t=n-3).

[0042] The above process of determining whether the raindrop image in the current scene image can be placed in the candidate list of restoration can be repeatedly executed, and the raindrop removal program 133 can compare a predetermined number (e.g., 4) of previous scene images at most, but the present invention is not limited thereto. Therefore, the raindrop removal program 133 can determine the movement trajectory of the raindrop image 421 based on the intersection-and-union ratio between the raindrop image 421 in the current scene image and the raindrop images 421 in each previous scene image.

[0043] In other words, if the intersection-and-union ratio of the raindrop image 421 in the current scene image (e.g., scene image 405) and the raindrop image 421 in any of the previous scene images is greater than or equal to a predetermined ratio threshold, the raindrop removal program 133 will not place this raindrop image in the current scene image into the restoreable candidate list.

[0044] In some embodiments, if the intersection-and-union ratio of the raindrop image 421 in the current scene image (e.g., scene image 405) and the raindrop images 421 in multiple previous scene images is less than a predetermined ratio threshold, the raindrop removal program 133 will preferentially select a previous scene image whose time point is closer to the current scene image as the previous scene image corresponding to the raindrop image 421 in the restorable candidate list.

[0045] In other embodiments, when the corresponding IoU ratios of the raindrop images 421 in multiple previous images are less than a predetermined ratio threshold, the raindrop removal program 133 will first compare the multiple IoU ratios and select the previous scene image with the smallest IoU ratio (i.e., the one with the least overlap) as the previous scene image corresponding to the raindrop image 421 in the candidate list of restoration. If the raindrop images in multiple previous scene images have the smallest IoU ratio at the same time, the raindrop removal program 133 will preferentially select the previous scene image with a time point closer to the current scene image as the previous scene image corresponding to the raindrop image 421 in the candidate list of restoration.

[0046] Specifically, when the raindrop image 421 in the current scene image is recognized by the raindrop image recognition model 400 and the raindrop removal program 133 places the raindrop image in the restoreable candidate list, it means that the raindrop removal program 133 can perform raindrop removal processing on the raindrop image 421. Figure 4B As shown, the raindrop image 421 of the current scene image (e.g., scene image 405) does not overlap with the raindrop image 421 of the previous scene image (e.g., scene image 403), so the corresponding intersection-over-union ratio is 0. This means that the pixels of the raindrop image 421 of the current scene image (e.g., scene image 405) in the corresponding image area 422 of the previous scene image (e.g., scene image 403) can completely replace the raindrop image 421 in the current scene image to obtain the output scene image 405'. It should be noted that, for the convenience of calculation, the raindrop image recognized by the raindrop image recognition model 400 is generally represented by a rectangular image, and the corresponding image area of ​​the raindrop image in the previous scene image corresponds to the shape of the raindrop image. Through the scene images captured at different time points in the video sequence, the corresponding image area of ​​the previous scene image that is more suitable for replacing the raindrop image in the current scene image can be found. Therefore, the raindrop removal method of the video image in the present invention can achieve a better raindrop removal effect.

[0047] like Figure 5A and Figure 5C As shown, the current scene images 510 and 530 captured by the image capture device 120 contain many raindrop images, and after the raindrop image recognition model is used to recognize the raindrop images and the raindrop removal process is performed, the following can be obtained respectively: Figure 5B and Figure 5D Output scene images 511 and 531 are shown.

[0048] In some embodiments, the raindrop removal program 133 may set a region of interest (ROI) or a priority area, for example, a predetermined area around the center of the scene image may be set as the ROI. If the height of the scene image is H and the width is W, the ROI may be, for example, an area extending up and down H / 4 and left and right W / 4 from the center of the scene image. This ROI is also the area where the driver's attention is most focused, so it is necessary to give priority to removing raindrops from the scene image to improve the accuracy and reliability of the back-end image processing and the autonomous vehicle driving decision.

[0049] When the number of raindrop images in the candidate list of restoration corresponding to the current scene image is quite large, if the computing unit 110 has computing performance or power consumption considerations, the computing unit 110 does not necessarily need to perform raindrop removal processing on all raindrop images in the current scene image. Therefore, the raindrop removal program 133 may preferentially perform raindrop removal processing on raindrop images in the candidate list of restoration and located in the region of interest. If the number of raindrop images in the candidate list of restoration and located in the region of interest is still large, the raindrop removal program 133 may select raindrop images in the candidate list of restoration in a predetermined order (e.g., sequential scanning) or randomly in the region of interest for raindrop removal processing.

[0050] Similar to the aforementioned embodiment, in this embodiment, the raindrop image recognition model 400 may also set a stop condition to stop further recognition and replace the raindrop images 421 in the current scene image, wherein the above-mentioned stop condition may be, for example: (1) the raindrop image recognition model 400 has recognized a predetermined number of raindrop images 421 in the region of interest of the current scene image; (2) the raindrop image recognition model 400 has recognized all raindrop images 421 (for example, a first number N) in the region of interest of the current scene image, and has successfully removed a predetermined proportion (for example, 0.5) of the first number N of raindrop images 421, that is, N / 2 number of raindrop images 421 in the region of interest have been successfully removed.

[0051] In a specific embodiment, Figure 1The non-volatile memory 132 may only store the raindrop image recognition model 135. The raindrop image database 136 and the raindrop removal program 133 may be stored and executed by other computing devices, and then the generated raindrop image recognition model 135 is stored in the computing device 100 to perform the above operations, thereby reducing the required resources of the computing device 100.

[0052] Figure 6 is a flow chart showing a method for removing raindrops from a video image according to an embodiment of the present invention. Figure 1 and Figure 6 .

[0053] In step S610, a plurality of first raindrop images marked in the plurality of rainy scene images are used to train a raindrop image recognition model. For example, the user may mark the position and size of each raindrop image 210 in the rainy scene image 200 on the user interface of the raindrop removal program 133, and the raindrop removal program 133 may represent the position and size of each raindrop image 210 as the horizontal / vertical ratio of each raindrop image 210 in the rainy scene image 200, wherein the horizontal / vertical ratio is a value between 0 and 1.

[0054] In step S620, the raindrop image recognition model is used to recognize multiple raindrop images in multiple scene images in a video sequence, and the position and size of each raindrop image in the current scene image are recorded. For example, the image capture device 120 can be used to obtain the above video sequence, such as Figure 4A As shown, it is assumed that the image capture device 120 can capture 5 scene images per second, for example, scene images 401 - 405 , and the scene images 401 - 405 are input into the raindrop image recognition model 400 in the capturing order as the current scene image.

[0055] When the scene images 401 - 405 are input to the raindrop image recognition model 400 in the shooting order as the current scene image, the raindrop image recognition model 400 can sequentially recognize the raindrop images 421 in the scene images 401 - 405 and record the position and size of each raindrop image 421 .

[0056] In step S630, in response to a specific raindrop image in a current scene image among the plurality of scene images satisfying a predetermined condition, the specific raindrop image is replaced by an image area corresponding to a specific scene image of the plurality of images preceding the current scene image to generate an output scene image. For example, the raindrop removal program 133 may first analyze the movement trajectory of each raindrop image in the current scene image, for example, the overlap degree of the raindrop image in the current scene image (e.g., scene image 405) and the raindrop image 421 of any of the previous scene images (e.g., scene images 401-404) may be calculated, wherein the overlap degree may be represented by, for example, intersection over union (IoU). The raindrop removal program 133 may repeatedly execute the process of determining whether the raindrop image in the current scene image can be placed in the candidate list that can be restored, and the raindrop removal program 133 may compare a predetermined number (e.g., 4) of previous scene images at most, but the present invention is not limited thereto. The predetermined condition is, for example, that an intersection-over-union ratio of the specific raindrop image in the current scene image and the specific raindrop image in the specific scene image is less than a predetermined ratio threshold (for example, 0.5).

[0057] In addition, the raindrop removal program 133 performs raindrop removal processing on each raindrop image in the current scene image according to the moving trajectory of each raindrop image to obtain an output scene image after the raindrops are removed. For example, the raindrop removal program 133 replaces the raindrop image in the current scene image with the pixels in the corresponding area of ​​the previous scene image corresponding to the raindrop image in the restoreable list of the current scene image to obtain the output scene image after the raindrops are removed. The mechanism for selecting the raindrop image in the restoreable list can be adjusted according to different situations. For example, the raindrop images in the restoreable list in the entire current scene image can be selected in a predetermined order or randomly, or the raindrop images in the restoreable list in the area of ​​interest of the current scene image can be selected in a predetermined order or randomly.

[0058] In summary, the present invention provides a method for removing raindrops from video images using a computing device, wherein the corresponding image area of ​​a previous scene image that is more suitable for replacing the raindrop image in the current scene image is found through scene images captured at different time points in a video sequence. Therefore, the method for removing raindrops from video images in the present invention can achieve a better raindrop removal effect. After removing raindrops from the scene image, the accuracy and reliability of the back-end image processing and the driving decision of the automatic vehicle can be further improved.

[0059] The method of the present invention, or a specific form or part thereof, may be included in the form of program code on a physical medium, such as a floppy disk, a CD, a hard disk, or any other machine-readable (such as computer-readable) storage medium, wherein when the program code is loaded and executed by a machine, such as a computer, the machine becomes a device or system for participating in the present invention. The method, system and device of the present invention may also be transmitted in the form of program code through some transmission medium, such as wires or cables, optical fibers, or any transmission mode, wherein when the program code is received, loaded and executed by a machine, such as a computer, the machine becomes a device or system for participating in the present invention. When implemented on a general-purpose processor, the program code combines with the processor to provide a unique device that operates similarly to application-specific logic circuits.

[0060] Although the present invention is disclosed as above with reference to preferred embodiments, it is not intended to limit the scope of the present invention. Any person skilled in the art may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the appended claims.

[0061] [Explanation of symbols]

[0062] 100: Computing device

[0063] 100: Computing device

[0064] 110: Operation unit

[0065] 120: Image capture device

[0066] 130: Storage unit

[0067] 131: Volatile memory

[0068] 132: Non-volatile memory

[0069] 133:Raindrop removal procedure

[0070] 135:Raindrop Image Recognition Model

[0071] 136:Raindrop Image Database

[0072] 150: Display

[0073] 200: Rainy day scene image

[0074] 210: Raindrop training image

[0075] 300: Object Recognition Model

[0076] 310: Input layer

[0077] 320: Hidden layer

[0078] 330: Output layer

[0079] 400:Raindrop Image Recognition Model

[0080] 401-405: Scene Image

[0081] 420, 421: raindrop images

[0082] 422: Image area

[0083] 405': Output scene image

[0084] 510, 530: current scene image

[0085] 511, 531: output scene image

[0086] S610-S630~Steps

Claims

1. A method for removing raindrops from a video image, the method comprising: Using a plurality of raindrop training images marked in a plurality of rainy scene images to train a raindrop image recognition model; Using the raindrop image recognition model to recognize multiple raindrop images in multiple scene images in a video sequence; Record the corresponding position and size of each identified raindrop image in the corresponding scene image; calculating an intersection-over-union ratio of a specific raindrop image in a current scene image and the specific raindrop image in a plurality of previous scene images in the video sequence, wherein the specific raindrop image is selected from a region of interest in the current scene image; In response to an intersection-and-union ratio between the specific raindrop image in the current scene image and the specific raindrop image in the plurality of previous scene images being less than a predetermined ratio threshold, selecting a previous scene image with a minimum intersection-and-union ratio as a previous scene image corresponding to the specific raindrop image in a restorable candidate list; as well as The specific raindrop image is replaced by a corresponding image region of the previous scene image corresponding to the specific raindrop image in the restorable candidate list to generate an output scene image.

2. The method for removing raindrops from a video image as claimed in claim 1, wherein the plurality of raindrop images in the current scene image has a first quantity, and the method further comprises: In response to the number of the raindrop images that have been successfully replaced reaching a predetermined proportion of the first number, the replacement of the remaining raindrop images in the current scene image is stopped.

3. The method for removing raindrops from a video image according to claim 1, further comprising: When the specific raindrop image in multiple previous scene images simultaneously has the minimum intersection-over-union ratio, the previous scene image with a time point closer to the current scene image is selected as the previous scene image corresponding to the specific raindrop image in the restoreable candidate list.

4. A computing device, comprising: a non-volatile memory for storing a raindrop removal program; as well as A computing unit, configured to execute the raindrop removal process to perform the following steps: Using a plurality of raindrop training images marked in a plurality of rainy scene images to train a raindrop image recognition model; Using the raindrop image recognition model to recognize multiple raindrop images in multiple scene images in a video sequence; Record the corresponding position and size of each identified raindrop image in the corresponding scene image; calculating an intersection-over-union ratio of a specific raindrop image in a current scene image and the specific raindrop image in a plurality of previous scene images in the video sequence, wherein the specific raindrop image is selected from a region of interest in the current scene image; In response to an intersection-and-union ratio between the specific raindrop image in the current scene image and the specific raindrop image in the plurality of previous scene images being less than a predetermined ratio threshold, selecting a previous scene image with a minimum intersection-and-union ratio as a previous scene image corresponding to the specific raindrop image in a restorable candidate list; as well as The specific raindrop image is replaced by a corresponding image region of the previous scene image corresponding to the specific raindrop image in the restorable candidate list to generate an output scene image.

5. The computing device of claim 4 , wherein the plurality of raindrop images in the current scene image have a first quantity, and in response to the number of the plurality of raindrop images that have been successfully replaced reaching a predetermined proportion of the first quantity, the computing unit stops replacing the remaining plurality of raindrop images in the current scene image.

6. The computing device as described in claim 4, wherein when there are multiple previous scene images of the specific raindrop image that simultaneously have the minimum intersection-over-union ratio, the computing unit selects a previous scene image whose time point is closer to the current scene image as the previous scene image corresponding to the specific raindrop image in the restorable candidate list.

Images