Image-based processing method and device

Abstract

Provided is a single-image-based processing method. The method comprises receiving a single image, the single image showing at least rain or snow marks; inputting the single image into a first convolutional neural network to perform computer vision-based processing on the single image, the first convolutional neural network being obtained by training according to a first image and a second image, wherein the first image shows at least rain or snow marks, and the second image is obtained by performing stylization processing on the first image. Thus, for images including rain or snow marks, the robustness and accuracy of computer vision-based processing / perception are improved, while real-time processing / perception can be achieved.

Description

Technical Field

[0001] This invention relates to the field of image-based processing / perception, and more particularly to the application of artificial intelligence in the field of computer vision-based processing / perception. Background Technology

[0002] In current computer vision-based processing / perception, in order to achieve functions such as target detection, target tracking, image classification, target segmentation, and instance segmentation, artificial intelligence technology is needed to perform machine learning on the features of known images to obtain corresponding convolutional neural networks to perform computer vision-based processing / perception functions.

[0003] In bad weather, images are degraded, affecting the accuracy of image perception. In particular, on rainy days, not only do rain streaks degrade images, but heavy rain is often accompanied by fog, which further degrades image quality.

[0004] Currently, there are two methods for restoring the quality of images taken on rainy days: video-based and single-image-based. Video-based methods mainly rely on temporal information between video frames to remove rain streaks, while single-image-based methods use machine learning to obtain de-rained images.

[0005] Therefore, in computer vision-based processing / perception applications, it is necessary to first process the degraded image to restore its quality, such as performing deraining processing on an image degraded by rain. Then, the derained image is input into a trained convolutional neural network to perform computer vision-based processing / perception functions. Such methods are difficult to meet the requirements of real-time processing and cannot obtain satisfactory perception results. Summary of the Invention

[0006] An improved computer vision-based processing method is provided, which tends to learn the shape features of sample images, thereby increasing the robustness and accuracy of convolutional neural networks.

[0007] It has been recognized that current computer vision research and applications tend to train convolutional neural networks (CNNs) based on image texture features to perform computer vision-based processing / perception (e.g., object detection, object tracking, image classification, object segmentation, and instance segmentation), thus relatively neglecting shape features. The inventors recognize that during rain, rain streaks, as high-frequency noise, primarily degrade the texture features of an image. Therefore, when using CNNs trained with a focus on texture features to perform computer vision-based processing / perception on images showing rain streaks, satisfactory results are difficult to obtain due to the degradation of image texture features. In contrast, rain streaks have a relatively low degradation effect on image shape features, and in practice, when identifying an object, people often consider both shape and texture features. Therefore, ignoring shape features and focusing on texture features when training a CNN is clearly unreasonable.

[0008] According to various embodiments of the present disclosure, a convolutional neural network is trained based on a first image and a second image obtained by stylizing the first image. By stylizing the first image, the representation of the object by its texture features is reduced, while the representation of the object by its shape features is relatively highlighted. The convolutional neural network trained based on these features focuses on the shape features of the training sample images, reducing the importance of texture features. When using the convolutional neural network trained in this way to perform computer vision-based processing / perception on images taken on rainy days, since the convolutional neural network focuses on shape features with relatively low degradation, it can obtain better processing / perception results. Furthermore, it eliminates the need to first perform de-raining processing on the image; the image can be directly input into the convolutional neural network used to perform computer vision-based processing / perception functions, making real-time image-based processing / perception possible. In summary, the convolutional neural network designed in this way not only enhances robustness and accuracy for images taken on rainy days but also enables real-time processing / perception.

[0009] According to one aspect, a processing method based on a single image is provided. The method includes receiving a single image, the single image showing at least rain streaks or snow streaks; inputting the single image into a first convolutional neural network to perform computer vision-based processing on the single image, the first convolutional neural network being trained on a first image and a second image, wherein the first image shows at least rain streaks or snow streaks, and the second image is obtained by stylizing the first image.

[0010] According to another aspect, a processing apparatus based on a single image is provided. The apparatus includes a receiving unit that receives a single image, the single image showing at least rain streaks or snow streaks; and a processing unit that inputs the single image into a first convolutional neural network to perform computer vision-based processing on the single image, the first convolutional neural network being trained on a first image and a second image, wherein the first image shows at least rain streaks or snow streaks, and the second image is obtained by stylizing the first image.

[0011] According to another aspect, a method for training a convolutional neural network based on a single image is provided. The method includes receiving a first image, the first image showing at least rain streaks; performing stylization processing on the first image to obtain a second image; and training a first convolutional neural network for performing computer vision-based processing based on the first image and the second image.

[0012] According to another aspect, a convolutional neural network training device based on a single image is provided. The device includes a receiving unit that receives a first image, the first image showing at least rain streaks or snow streaks; a stylization processing unit that performs stylization processing on the first image to obtain a second image; and a training unit that trains a first convolutional neural network for performing computer vision-based processing based on the first image and the second image.

[0013] According to another aspect, a computer system is provided, including one or more processors; and one or more storage devices storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the methods described according to various embodiments of the present disclosure.

[0014] According to another aspect, a machine-readable medium is provided that stores computer-executable instructions, which, when executed, cause a computer or processor to perform the methods described according to various embodiments of the present disclosure. Attached Figure Description

[0015] In the accompanying drawings, embodiments are illustrated by way of example only and not by way of limitation, and similar reference numerals in the drawings refer to similar elements.

[0016] Figure 1 A training method for a convolutional neural network based on a single image according to an embodiment of the present invention is illustrated;

[0017] Figure 2A and 2B Examples of the original image and the stylized image are shown;

[0018] Figure 3A processing / perception method based on a single image according to an embodiment of the present invention is illustrated;

[0019] Figure 4 A training apparatus for a convolutional neural network based on a single image according to an embodiment of the present invention is shown;

[0020] Figure 5 A single-image-based processing / sensing device according to an embodiment of the present invention is shown;

[0021] Figure 6 A schematic diagram of a computer system according to an embodiment of the present invention is shown.

[0022] Various aspects and features of the embodiments of the present invention have been described with reference to the accompanying drawings. The drawings are merely illustrative and not restrictive. The dimensions, shapes, reference numerals, or appearances of the various elements in the drawings may vary without departing from the spirit of the invention; furthermore, not all parts of the headphones or devices of the embodiments of the present invention are indicated by reference numerals in the drawings, and only relevant components are indicated in some drawings. This does not limit the various parts to what is shown in the accompanying drawings. Detailed Implementation

[0023] Although the following reference images include rain streaks, it can be understood that the same technical solution is applicable to images that include snow streaks.

[0024] In computer vision-based processing / perception applications (e.g., object detection, object tracking, image classification, object segmentation, and instance segmentation), the selection of training features is crucial when training convolutional neural networks (CNNs). Current research focuses on training CNNs based on texture features. However, this disclosure recognizes that rain streaks degrade the texture features of images; therefore, using a texture-based CNN for processing / perception on such images is unlikely to yield satisfactory results. Therefore, this disclosure provides a method that focuses more on training CNNs based on shape features and uses such a trained CNN to perform computer vision-based processing / perception functions. Since rain streaks are recognized to have relatively low degradation on shape features, the CNN trained in this way still provides good robustness and accuracy for images showing rain streaks. Furthermore, in this case, de-raining processing of the image is not required before inputting the image into the CNN, thus enabling real-time computer vision-based processing / perception.

[0025] Figure 1 A training method 100 for a convolutional neural network based on a single image according to an embodiment of the present invention is shown.

[0026] like Figure 1 As shown, according to this training method, in step 110, a first image is received, which at least shows rain streaks and is expected to also show the target to be detected. In step 120, stylization processing is performed on the received first image to obtain a second image. Specifically, the style of the first image is changed by an algorithm, such as style transfer, so that the style of the current first image becomes a predetermined style. The style of the second image obtained is different from the style of the first image. Figure 2A and 2B Examples of the original image and the stylized image are shown respectively. It is conceivable that multiple stylization processes can be performed on the first image to obtain multiple stylized first images, in which case multiple second images can be obtained. After stylization, the shape features of the first image are preserved, while the texture features are appropriately destroyed.

[0027] In step 130, a first convolutional neural network is trained based on the first and second images to perform computer vision-based processing / perception. The resulting first convolutional neural network is more biased towards object detection based on shape features, and therefore can still robustly and accurately detect objects even when rain streaks destroy the texture features of the image.

[0028] Rainy days are often accompanied by fog, which can distort the shape and features of an image. Therefore, it is necessary to remove the effects of fog.

[0029] According to the training method 100, in step 115, a third image is received, which at least shows fog, especially fog caused by rain. In one embodiment, the third image may include the first image. In step 125, a second convolutional neural network for defogging is trained based on the third image through supervised learning.

[0030] In particular, the second convolutional neural network was trained based on the following physical model.

[0031] I(x)=(J(x)+R(x))t(x)+A(1-t(x))

[0032] Where I(x) is the third image, x represents a pixel, J(x) is the third image to be obtained after dehazing, R(x) is a model representing the effect of rain, A represents the global illumination level, and t(x) represents the atmospheric transparency.

[0033] In this model, atmospheric transparency t(x) represents the portion of light reaching the camera, which decreases with distance from the camera. Global illumination level represents the overall degree of fog in the image. Based on this physical model, the farther an object is from the camera, the smaller t(x) and the larger A(1-t(x)), making the object less visible. Furthermore, the greater the fog, the larger A and A(1-t(x)), and the less visible the object becomes. This designed physical model accurately reflects the effects of rain and fog, and the convolutional neural network trained on this model is more suitable for computer vision-based processing / perception of images taken on rainy days.

[0034] After obtaining the trained first and second convolutional neural networks, in step 135, the first and second convolutional neural networks are combined to obtain the final convolutional neural network for computer vision-based processing / perception of images taken on rainy days.

[0035] The resulting convolutional neural network, comprising the first and second convolutional neural networks, can conveniently perform computer vision-based processing on a single image. The user only needs to input an image, which can be automatically dehazed based on the second convolutional neural network. The dehazed result is then automatically input into the first convolutional neural network, achieving end-to-end processing.

[0036] Figure 3 A single-image-based processing / perception method 200 according to an embodiment of the present invention is shown, the method 200 particularly employing the above-mentioned reference. Figure 1 The training method described is used to train a convolutional neural network.

[0037] According to method 200, in step 210, a single image for computer vision-based processing / perception is received. For example, a single image for image classification is received. This single image at least shows rain streaks, and it is conceivable that it also shows the object to be classified, and / or fog.

[0038] In one embodiment, firstly in step 220, the received single image is input into the second convolutional neural network trained as described above to perform dehazing processing. Then, in step 230, the dehazed single image is input into the first convolutional neural network trained as described above for computer vision-based processing / perception.

[0039] If the fog is not shown in the single image, step 220 can be omitted, and the single image can be directly input into the first convolutional neural network.

[0040] In step 240, the results of the computer vision-based processing / perception performed on the single image are output. These results can be output to other components for further processing. For example, in the field of autonomous driving, if a car is identified in the image, the result can be output to a controller, which then outputs driving control signals to control the car's operation.

[0041] Figure 4 A training device 10 for a convolutional neural network based on a single image according to an embodiment of the present invention is shown. The training device 10 includes a receiving unit 11, a stylization unit 12, a training unit 13, and a combining unit 14.

[0042] The receiving unit 11 receives sample images for training the convolutional neural network, including a first image showing at least rain streaks. In one embodiment, the sample images also include a third image showing at least fog. The first image may include the third image, particularly an image of fog caused by rain taken on a rainy day. The receiving unit 11 inputs the received first image into the stylization processing unit 12 to perform stylization processing on the first image, thereby obtaining a second image. The second image is then input into the training unit 13 for training the first convolutional neural network CNN1.

[0043] The receiving unit 11 also inputs the received third image into the training unit 13 for training a second convolutional neural network CNN2 through supervised learning. This second convolutional neural network is used for dehazing, and is specifically trained based on the physical model described above. Other types or structures of convolutional neural networks are also expected, as long as they can achieve the dehazing function.

[0044] After the training unit 13 trains the first and second convolutional neural networks with sufficient samples, the combination unit 14 combines the first and second convolutional neural networks to form the final convolutional neural network.

[0045] Figure 5 A single-image-based processing / sensing device 20 according to an embodiment of the present invention is shown. The device 20 includes a receiving unit 21 and a processing unit 22.

[0046] The receiving unit 21 receives a single image, which at least shows rain streaks. The processing unit 22 inputs the received single image into the first convolutional neural network CNN1 trained as described above to perform computer vision-based processing on it.

[0047] In one embodiment, when dehazing is required on a single image, the single image is first input into the second convolutional neural network CNN2 trained as described above for dehazing, and then the dehazed single image is input into the first convolutional neural network CNN1 trained to perform computer vision-based processing / perception.

[0048] Figure 6 A schematic diagram of a computer system 30 according to an embodiment of the present invention is shown. Figure 6 A schematic diagram of a computer system 30 according to an embodiment of the present disclosure is shown. Figure 6 As shown, the computer system 30 may include at least one processor 31, a memory (e.g., non-volatile memory) 32, a memory 33, and a communication interface 34, and the at least one processor 31, memory 32, memory 33, and communication interface 34 are connected together via a bus 36. The at least one processor 31 executes at least one computer-readable instruction (i.e., the elements implemented in software above) stored or encoded in the memory.

[0049] In one embodiment, computer-executable instructions are stored in memory that, when executed, cause at least one processor 31 to perform the above-described combinations of the various embodiments of this disclosure. Figure 1-3 The description includes various operations and functions.

[0050] According to one embodiment, a computer program product, such as a machine-readable medium (e.g., a non-transitory machine-readable medium), is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software as described above), which, when executed by a machine, cause the machine to perform the above-described combinations of the various embodiments of this disclosure. Figure 1-3 The various operations and functions described. Specifically, a system or apparatus equipped with a readable storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer or processor of the system or apparatus to read and execute the instructions stored in the readable storage medium.

[0051] The exemplary embodiments of this disclosure cover both: creating / using computer programs / software of this disclosure from the outset, and converting existing programs / software to use computer programs / software of this disclosure by means of updates.

[0052] Computer programs for performing the methods according to the embodiments of this disclosure may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0053] Computer programs can also be provided on networks such as the World Wide Web and can be downloaded from such networks to microprocessor-based computers.

[0054] It must be noted that the embodiments of this disclosure are described with reference to different subjects. In particular, some embodiments are described with reference to method claims, while others are described with reference to device claims. However, those skilled in the art will understand from the above and below that, unless otherwise specified, any combination of features relating to different subjects, in addition to any combination of features belonging to one type of subject matter, is also considered to be disclosed in this application. Furthermore, it is possible to combine all features to provide a synergistic effect greater than the simple sum of the features.

[0055] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0056] The present disclosure has been described above with reference to specific embodiments. Those skilled in the art should understand that the technical solutions of the present disclosure can be implemented in various ways without departing from the spirit and essential characteristics of the present disclosure. The specific embodiments are merely illustrative and not restrictive. Furthermore, these embodiments can be arbitrarily combined to achieve the purposes of the present disclosure. The scope of protection of the present disclosure is defined by the appended claims.

[0057] The word "comprising" in the specification and claims does not exclude the presence of other elements or steps. The expressions "first," "second," "step," etc., and the order of the various steps shown in the figures do not limit their sequence or quantity. The functions of the various elements described in the specification or recorded in the claims can also be separated or combined, and implemented by multiple corresponding elements or a single element.

Claims

1. A method of processing based on a single image, comprising: receiving a single image, the single image showing at least rain or snow marks; inputting the single image into a first convolutional neural network to perform computer vision based processing on the single image, the first convolutional neural network being obtained by training from a first image and a second image, wherein the first image shows at least rain or snow marks and the second image is obtained by stylizing the first image, the stylizing being for reducing the degree of representation of texture features on objects in the first image and relatively highlighting the representation of shape features on the objects.

2. The treatment method of claim 1, wherein, the single image also shows fog, and the method further comprises: inputting the single image into a second convolutional neural network to perform defogging processing on the single image; and inputting the defogged single image into the first convolutional neural network.

3. The treatment method of claim 2, wherein, the second convolutional neural network is obtained by supervised learning from a third image, wherein the third image shows fog.

4. The treatment method of claim 3, wherein, the second convolutional neural network is obtained by training based on a model as follows: I(x) = (J(x) + R(x)) t(x) + A (1 - t(x)) where I(x) is the third image, x represents a pixel, J(x) is the defogged third image to be obtained, R(x) is a model representing the effect of rain or snow, A represents a global light level, and t(x) represents the degree of atmospheric transparency.

5. A processing device based on a single image, comprising: a receiving unit that receives a single image, the single image showing at least rain or snow marks; a processing unit that inputs the single image into a first convolutional neural network to perform computer vision based processing on the single image, the first convolutional neural network being obtained by training from a first image and a second image, wherein the first image shows at least rain or snow marks and the second image is obtained by stylizing the first image, the stylizing being for reducing the degree of representation of texture features on objects in the first image and relatively highlighting the representation of shape features on the objects.

6. The processing device of claim 5, wherein, the single image also shows fog, the processing unit inputs the single image into a second convolutional neural network to perform defogging processing on the single image, and inputs the defogged single image into the first convolutional neural network.

7. The processing device of claim 6, wherein, the second convolutional neural network is obtained by supervised learning from a third image, wherein the third image shows at least fog.

8. The processing device of claim 7, wherein, the second convolutional neural network is obtained by training based on a model as follows: I(x) = (J(x) + R(x)) t(x) + A (1 - t(x)) where I(x) is the third image, x represents a pixel, J(x) is the defogged third image to be obtained, R(x) is a model representing the effect of rain or snow, A represents a global light level, and t(x) represents the degree of atmospheric transparency.

9. A method of training a convolutional neural network based on a single image, comprising receiving a first image, the first image showing at least rain marks; performing stylization processing on the first image to obtain a second image, the stylization processing being configured to reduce a degree of representation of a texture feature to an object in the first image and relatively highlight a shape feature to the object; and training, according to the first image and the second image, a first convolutional neural network configured to perform a computer vision-based processing.

10. The convolutional neural network training method of claim 9, further comprising: receiving a third image, the third image showing at least fog; and training, according to the third image, a second convolutional neural network configured to de-fog by supervised learning.

11. The convolutional neural network training method of claim 10, further comprising: combining the first convolutional neural network and the second convolutional neural network.

12. A convolutional neural network training device based on a single image, comprising: a receiving unit configured to receive a first image, the first image showing at least rain or snow; a stylization processing unit configured to perform stylization processing on the first image to obtain a second image, the stylization processing being configured to reduce a degree of representation of a texture feature to an object in the first image and relatively highlight a shape feature to the object; and a training unit configured to train, according to the first image and the second image, a first convolutional neural network configured to perform a computer vision-based processing. the receiving unit is further configured to receive a third image, the third image showing at least fog; and the training unit is further configured to train, according to the third image, a second convolutional neural network configured to de-fog by supervised learning.

13. The convolutional neural network training device of claim 12, wherein, 14. A computer system, comprising: one or more processors; and one or more storage devices storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1-4 and 9-11.

15. A computer-readable medium storing computer-executable instructions that, when executed by a computer or processor, cause the computer or processor to perform the method of any one of claims 1-4 and 9-11. ​

Images