Image processing method and device, electronic equipment and storage medium
Through the training image dedusting model, the dust-containing image in the loader operating environment is removed, which solves the problem that the demasting algorithm in the prior art is difficult to be applicable to the loader operating scenarios, and realizes the loader operating environment The effective removal of medium dust improves the reliability and safety of remote control.
Patent Information
- Application Number
- CN202411412710.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-02-07
AI Technical Summary
The existing mist removal algorithm is difficult to be applicable to loader operating scenarios and cannot effectively remove interference from strong dust in loader operating environments, resulting in low image clarity and affecting the reliability and safety of remote control.
By obtaining dust-free images and pre-generated pure dust images in indoor working scenes, generating dust-containing images, building training sample sets, and training obtained image dust-removing model for dust removal operations on actual dust-removing images.
It realizes clear processing of actual dust-carrying images, improves the image dust removal effect in the loader operating environment, and ensures the reliability and safety of remote control.
Smart Images
Figure CN119399074A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, electronic equipment and storage medium. Background Art
[0002] Wheel loaders are widely used mobile transport equipment, usually used to transport materials in mines or construction site mixing plants. Usually, experienced operators are required to operate the loader on site under extreme working conditions (such as high dust and extreme temperature). With the development of industrial automation technology, it is urgent to propose a remote control method for wheel loaders, that is, the operator remotely operates the loader in the office to perform loading operations, thereby avoiding the above-mentioned dangerous working environment.
[0003] In the related technologies, there are not many dust perspective algorithms for specific scenarios of loaders. Most of them are concentrated on demisting algorithms in the field of autonomous driving to ensure the normal operation of the vehicle vision system in foggy scenes.
[0004] However, since the training data used by the dehazing algorithm is mostly based on meteorological data collected from outdoor scenes, the existing dehazing algorithms are difficult to apply to actual loader operation scenarios. Summary of the invention
[0005] The purpose of the present invention is to provide an image processing method, device, electronic device and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problem that the existing dehazing algorithm is not applicable to actual loader operation scenarios.
[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:
[0007] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:
[0008] Acquire a plurality of dust-free images in an indoor working scene, wherein the dust-free images are taken in the indoor working scene, and when the dust-free images are collected, the dust concentration in the indoor working scene is lower than a preset threshold;
[0009] Generate dust-containing images corresponding to the dust-free images according to the dust-free images and the pre-generated pure dust images, wherein the pure dust images are composed of dust elements, and the dust-containing images are based on the dust-free images as background and contain dust elements;
[0010] Constructing a training sample set according to each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images, and obtaining an image dust removal model through training based on the training sample set;
[0011] The actual dust-bearing image collected in the indoor working scene is input into the image dust removal model to obtain a target dust-free image corresponding to the actual dust-bearing image.
[0012] In a second aspect, an embodiment of the present application further provides an image processing device, the device comprising:
[0013] An acquisition module, used to acquire a plurality of dust-free images in an indoor working scene, wherein the dust-free images are obtained by taking pictures in the indoor working scene, and when the dust-free images are acquired, the dust concentration in the indoor working scene is lower than a preset threshold;
[0014] A generating module, configured to generate dust-bearing images corresponding to the dust-free images according to the dust-free images and the pre-generated pure dust images, wherein the pure dust images are composed of dust elements, and the dust-bearing images are based on the dust-free images as background and contain dust elements;
[0015] A construction module, used to construct a training sample set according to each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images;
[0016] A training module, used for training an image dust removal model based on the training sample set;
[0017] The dust removal module is used to input the actual dust-bearing image collected in the indoor working scene into the image dust removal model to obtain the target dust-free image corresponding to the actual dust-bearing image.
[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the image processing method provided in the first aspect.
[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image processing method provided in the first aspect is executed.
[0020] The beneficial effects of this application are:
[0021] The present application provides an image processing method, device, electronic device and storage medium. Based on multiple dust-free images and pre-generated pure dust images in indoor working scenes, dust-bearing images corresponding to each dust-free image are generated, and dust training data sets are constructed based on each dust-free image and the dust-bearing images corresponding to each dust-free image. Then, based on the dust training data set, an image dust removal model that can be specifically applied to indoor dust scenes is trained. When it is necessary to perform dust removal operation on the actual dust-bearing image collected in the indoor working scene, it is only necessary to input the actual dust-bearing image into the image dust removal model to obtain the target dust-free image corresponding to the actual dust-bearing image, thereby completing the clearing operation of the actual dust-bearing image and improving the dust removal effect in the indoor scene. At the same time, the operator can realize remote control of the loader based on the target dust-free image, thereby ensuring the reliability and safety of the remote control operation and solving the problem that the existing de-hazing algorithm cannot be applied to the actual loader working scene. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0023] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;
[0024] Figure 2 A flowchart of another image processing method provided in an embodiment of the present application;
[0025] Figure 3 A flowchart of another image processing method provided in an embodiment of the present application;
[0026] Figure 4 A flowchart of another image processing method provided in an embodiment of the present application;
[0027] Figure 5 A flowchart of another image processing method provided in an embodiment of the present application;
[0028] Figure 6 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;
[0029] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0030] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.
[0031] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0032] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0033] First, the prior art involved in this application is introduced.
[0034] It is understandable that remote control of loaders requires the operating system to provide clear and reliable real-time image data to assist operators in making accurate decisions and operations. One of the major difficulties is that loaders usually work in a strong dusty environment, and the image data transmitted remotely is mixed with a lot of dust, with extremely poor clarity. Moreover, these images with strong dust will greatly increase the difficulty of remote control and increase the risk of misoperation.
[0035] Therefore, there is an urgent need to provide an image dust removal model for indoor strong dust environments, so that the image dust removal model can be used to perform dust removal operations on dusty images collected in strong dust environments, and obtain target dust-free images corresponding to the dusty images, so that the operator can realize safe and reliable remote control operations based on the target dust-free images.
[0036] The existing technologies are analyzed from the algorithm level and hardware level, as follows:
[0037] (1) At the algorithm level, there are currently few dust removal models for indoor strong dust environments, and more focus is on dehazing algorithms in the field of autonomous driving to ensure the normal operation of the vehicle vision system in foggy scenes. With the rapid development of deep learning, related learning-based dehazing algorithms are becoming more and more abundant. For example, related algorithms include DehazeFormer, gUnet, etc., and these deep learning methods all need to be trained based on effective supervised data.
[0038] However, the above dehazing algorithm is difficult to apply to loader operation scenarios for the following two main reasons: 1. Meteorological data is more suitable for outdoor scenes, and the model trained based on meteorological data is difficult to be effective in warehouse scenes such as silos and cabins where loaders operate; 2. The training data generated based on the atmospheric scattering model can better simulate foggy scenes, and its noise distribution is very different from dust data. Therefore, the existing related dehazing models are also difficult to generalize to dust scenes.
[0039] (2) At the hardware level, there are also a large number of perspective cameras in the current market that are designed for scenes with strong smoke, strong dust, and night scenes. Specifically, the general principle is to equip conventional cameras with some infrared cameras or ultra-wide spectrum camera modules, which can remove optical interference, smoke and dust interference, etc. through a wider band.
[0040] However, the biggest problem with the above hardware level is the high cost. Ultra-wide spectrum cameras are relatively expensive, which poses a major cost challenge to the technical solution of remote-controlled loader.
[0041] Based on the above-mentioned problems, an embodiment of the present application provides an image processing method for indoor strong dust environments. Based on multiple dust-free images in indoor working scenes and pre-generated pure dust images, dusty images corresponding to each dust-free image are generated, and a dust training data set is constructed based on each dust-free image and the dusty images corresponding to each dust-free image. Then, based on the dust training data set, an image dust removal model is trained. When it is necessary to perform dust removal operation on the actual dusty image collected in the indoor working scene, it is only necessary to input the actual dusty image into the image dust removal model to obtain the target dust-free image corresponding to the actual dusty image, thereby achieving the clarification of the actual dusty image, so that the operator can realize remote control of the loader based on the target dust-free image, and ensure the reliability and safety of the remote control operation, thereby solving the problem that the existing dehazing algorithm cannot be applied to the actual loader working scene.
[0042] The implementation principles and corresponding beneficial effects of the image processing method steps provided in this application will be explained through multiple specific embodiments as follows.
[0043] In one embodiment, reference Figure 1 As shown, an image processing method is provided. Optionally, the execution subject of the method can be a terminal device with data processing function such as a personal computer, a laptop computer, a smart phone and a tablet computer.
[0044] It should be understood that in other embodiments, the order of some steps in the image processing method can be interchanged according to actual needs, or some steps can be omitted or deleted. Figure 2 As shown, the method includes:
[0045] S101: Acquire multiple dust-free images in an indoor working scene.
[0046] The dust-free image is captured in an indoor working scene, and the dust concentration in the indoor working scene is lower than a preset threshold when the dust-free image is captured.
[0047] Among them, indoor working scenes are indoor industrial production scenes with high dust concentration, such as a large amount of dust generated by production activities in a mine crushing station; for example, a large amount of dust is also generated when a loader loads and unloads materials indoors. In such a dusty indoor working environment, the strong dust can easily affect the line of sight, making it difficult for operators to accurately select the appropriate working point.
[0048] In one feasible method, for example, taking the indoor operation scene of a loader as an example, before the loader starts working (i.e., no dust is generated), a sprinkler truck is used to spray water around the loader in advance to eliminate the dust in the operation scene as much as possible; then, 1-2 industrial cameras installed on the loader are used to collect dust-free images at different angles. That is, the dust-free images collected at this time are absolutely clear and dust-free images of the indoor operation scene.
[0049] Optionally, the collected dust-free image may be further subjected to histogram equalization processing and / or neutral gray reduction processing to further improve the clarity of the dust-free image.
[0050] Among them, a large number of dust-free images collected above are used as target images for learning the image dust removal model.
[0051] S102 : generating dust-bearing images corresponding to the dust-free images according to the dust-free images and the pre-generated pure dust images.
[0052] Among them, the pure dust image is composed of dust elements, and the dust-containing image has a dust-free image as the background and contains dust elements.
[0053] Each pixel in the pure dust image is used to represent the brightness value of the dust element. When the pixel value of a pixel in the pure dust image is large, it can be determined that the brightness of the dust element represented by the pixel is high.
[0054] The pixel value of each pixel in the dust image is between the pixel value of the corresponding pixel in the dust-free image and the pixel value of the corresponding pixel in the pure dust image, that is, the dust image is used to simulate an image taken in a scene with a certain dust concentration.
[0055] In one feasible method, for example, the weight value of each pixel in the dust-free image is set to 0.4, and the weight value of each pixel in the dust-free image is set to 0.6. Then, according to the preset weight value, the pixel value of each pixel in the dust-free image is weighted with the pixel value of the same pixel in the pure dust image to generate a dusty image corresponding to the dust-free image, that is, the dusty image is synthesized based on the dust-free image and the pre-generated pure dust image.
[0056] S103: construct a training sample set according to each dust-free image and the dust-bearing image corresponding to each dust-free image, and obtain an image dust removal model through training based on the training sample set.
[0057] In one feasible method, for example, dust-free images and dust-containing images corresponding to the dust-free images are used as a group of training samples, and multiple groups of training samples can be constructed using the dust-free images and dust-containing images corresponding to the dust-free images obtained as mentioned above; then, the multiple groups of training samples are input into the selected initial training model, and the initial training model is iteratively trained for multiple times to obtain an image dust removal model.
[0058] The image dust removal model trained by this scheme has good transferability, that is, it can be deployed in any required working environment, such as the control system on the loader, the control system on the mine car, etc.
[0059] S104: Input the collected actual dusty image in the indoor working scene into the image dust removal model to obtain a target dust-free image corresponding to the actual dusty image.
[0060] In one feasible method, for example, the image dust removal model obtained through the above training can be deployed on the control system of the loader, and during the operation of the loader, the actual dusty images in the indoor operation scene are collected by the industrial camera on the loader, and the actual dusty images are input into the image dust removal model to obtain the target dust-free image corresponding to the actual dusty image output by the image dust removal model, thereby achieving the clarification of the actual dusty image.
[0061] Therefore, by using the image dust removal model provided in this application, an effective powder layer perspective effect can be achieved on the actual dust-bearing images collected in indoor working scenes, thereby assisting the operator in realizing remote control of the loader, ensuring the reliability and safety of remote control operations, and solving the problem that the existing dehazing algorithm cannot be applied to actual loader operating scenes.
[0062] At the same time, the image processing method proposed in this application only needs to adapt 1-2 industrial cameras, and through a specific image dust removal model for indoor work scenes, it is possible to clarify dusty images without installing additional infrared cameras or ultra-wide spectrum camera modules. The dust removal effect of the image dust removal model provided by this application is also better than that of open source defogging algorithms.
[0063] In summary, the embodiment of the present application provides an image processing method for indoor strong dust environment, which generates dusty images corresponding to each dust-free image based on multiple dust-free images in indoor working scenes and pre-generated pure dust images, and constructs a dust training data set based on each dust-free image and the dusty image corresponding to each dust-free image; then, based on the dust training data set, an image dust removal model is trained to obtain a dust removal operation. When it is necessary to perform dust removal operation on the actual dusty image collected in the indoor working scene, it is only necessary to input the actual dusty image into the image dust removal model to obtain the target dust-free image corresponding to the actual dusty image, thereby completing the clarification operation of the actual dusty image, so that the operator can realize remote control of the loader based on the target dust-free image, thereby ensuring the reliability and safety of the remote control operation, and solving the problem that the existing dehazing algorithm cannot be applied to the actual loader working scene.
[0064] Optionally, the above-mentioned prior art embodiments have clearly pointed out that the current defogging algorithm cannot be applied to the actual loader working environment because the indoor working scene does not conform to the atmospheric scattering model. Therefore, it is necessary to construct reasonable dust training data in the loader dust environment. Therefore, the following embodiments will specifically explain how to construct dust training data for indoor working scenes.
[0065] Optionally, refer to Figure 2 As shown, the above step S102 includes:
[0066] S201 : Generate a depth image of the dust-free image according to the dust-free image.
[0067] In one achievable manner, for example, the depth value of each pixel in the dust-free image may be calculated according to the triangulation principle, and a depth image of the dust-free image may be generated based on the depth value of each pixel in the dust-free image.
[0068] S202 : Generate an image with dust corresponding to the dust-free image according to the dust-free image, the depth image and the pure dust image.
[0069] Optionally, in order to construct more reasonable dust training data, that is, a large number of dust-free image-dust image pairs, this requires that in the dust-free image-dust image pairs, except for the presence or absence of dust in the scene, other elements are consistent. However, due to the lighting in the real world, indoor work scenes are constantly changing. At different times, it is impossible to collect image pairs in which only the dust changes while other elements do not change, resulting in it being difficult to collect dust-free image-dust image pairs in the real world. Therefore, this scheme proposes that dust training data (i.e., dust-free image-dust image pairs) can be constructed manually, and the dust training data needs to be as close to the actual dust distribution as possible.
[0070] Optionally, considering that in indoor dusty operation scenes, the sight of points close to the camera is clearer, while the sight of points far from the camera is more blurred. Therefore, the depth information of the dust-free image is the key point to simulate the real dusty image. Therefore, the core of this solution is to simulate and generate dusty images similar to the real operation scenes for the dust-free image, the depth image of the dust-free image and the pure dust image, thereby improving the authenticity of the dusty image finally generated.
[0071] How to determine the depth image of the dust-free image will be explained in detail through the following embodiments.
[0072] Optionally, the above step S201 includes:
[0073] The dust-free image is input into a pre-trained depth estimation model to obtain a depth image of the dust-free image output by the depth estimation model.
[0074] The depth estimation model may be a network model trained based on multiple sets of pre-constructed original image-depth image pairs.
[0075] In this embodiment, in order to improve the accuracy of the depth estimation result of the original image, the depth estimation model used is a large model depth-anything, which has a strong generalization ability and can generate a depth image of the original image by directly inputting the original image. Therefore, the dust-free image obtained above can be input into the pre-trained depth estimation model to obtain the depth image of the dust-free image output by the depth estimation model, that is, the extraction operation of the depth information of the dust-free image is completed.
[0076] Optionally, the above step S202 includes:
[0077] According to the depth image, the distribution information of the pure dust image in the dust-free image is determined, and based on the distribution information, an image with dust corresponding to the dust-free image is generated.
[0078] The pixel value of each pixel in the depth image represents the depth information of each pixel in the dust-free image, that is, the larger the pixel value of a pixel in the depth image is, the farther the corresponding pixel in the dust-free image is from the camera.
[0079] In this scheme, considering that in indoor dusty working scenes, the position points close to the camera have clearer vision, while the position points far away from the camera have more blurred vision. Therefore, based on the depth image of the dust-free image, the proportion of pure dust image in the dust-free image can be determined, thereby improving the rationality of the dusty image finally generated.
[0080] Optionally, refer to Figure 3 As shown, the above steps determine the distribution information of the pure dust image in the dust-free image according to the depth image, and generate the dust-bearing image corresponding to the dust-free image based on the distribution information, including:
[0081] S301 : determining a distribution ratio of each pixel in a pure dust image in a dust-free image according to a dust scattering coefficient in an indoor operation scene and a depth image of a dust-free image.
[0082] S302: Generate a first fused image according to the distribution ratio and the pure dust image.
[0083] S303: Generate a second fused image according to the distribution ratio and the dust-free image.
[0084] S304: Superimpose the first fused image and the second fused image to generate an image with dust corresponding to the image without dust.
[0085] In this example, for example, the dust scattering coefficient in the indoor working scene is recorded as β, and the depth image of the dust-free image is recorded as D. Then, according to the dust scattering coefficient β in the indoor working scene and the depth image D of the dust-free image, the distribution ratio of each pixel in the pure dust image in the dust-free image is obtained, that is, e -β(1-D) Therefore, the distribution ratio e can be obtained from the above calculation -β(1-D) It can be obtained that when the depth value in the depth image D is larger, the distribution ratio of each pixel in the pure dust image in the dust-free image is also larger, that is, e -β(1-D) *B has a larger proportion; conversely, if the depth value in the depth image D is smaller, the distribution ratio of each pixel in the pure dust image in the dust-free image is also smaller, that is, (1-e -β(1-D) )*The larger the A is, the greater the proportion of dust-free images. In other words, the distribution ratio e -β(1-D)It means that as the depth value in the depth image D increases, the distribution ratio of each pixel in the pure dust image B in the dust-free image A also increases.
[0086] Then, according to the distribution ratio and the pure dust image B, a first fused image is generated, and the generated first fused image is recorded as e -β(1-D) *B; and generate a second fused image according to the distribution ratio and the dust-free image A, and the generated second fused image is recorded as (1-e -β(1-D) )*A; Finally, the first fused image is superimposed with the second fused image to generate a dust-bearing image M corresponding to the dust-free image, and the dust-bearing image corresponding to the dust-free image is recorded as M=e -β(1-D) *B+(1-e -β(1-D) )*A. Therefore, the formula M=e -β(1-D) *B+(1-e -β(1-D) )*A, the dust-free image A, the depth image D and the pure dust image B are fused to generate a dust-bearing image corresponding to the dust-free image. In this way, the rationality of the fused dust-bearing image can be ensured.
[0087] Among them, A is a dust-free image and B is a pure dust image.
[0088] The following example will explain in detail how to obtain a pure dust image.
[0089] Optionally, a pure dust image is obtained by the following steps:
[0090] An image of a target indoor scene is collected to obtain a pure dust image, wherein the target indoor scene includes a white background and dust with a concentration greater than a preset threshold.
[0091] In this embodiment, for example, the pure dust image does not contain any other elements (such as pedestrians, materials, etc.) except the dust element. However, in actual indoor operation scenes, it is difficult to capture an ideal image of pure dust. Therefore, a pure white background can be produced, and a strong dust scene with a concentration greater than a preset threshold can be created by some means, and a single pure dust image can be captured by the camera unit, and it is ensured that the pure dust image does not contain any background, so as to ensure that the pure dust image is composed of dust elements.
[0092] The following example will explain in detail how to obtain an image dust removal model based on the construction of dust training data.
[0093] Optionally, refer to Figure 4 As shown, the above step S103 includes:
[0094] S401 , taking each dust-free image and the dust-bearing image corresponding to each dust-free image as a sample image pair in a training sample set.
[0095] In one feasible method, for example, before training, a large number of dust training samples need to be constructed, that is, the dust-free images obtained above and the dust-containing images corresponding to each dust-free image are used as training sample sets, that is, the training sample set includes: multiple sample image pairs, each sample image pair includes: a dust-free image and an image with dust corresponding to the dust-free image, that is, dust-free image A-dust-containing image M.
[0096] S402 : Inputting the dust-carrying image in each sample image pair of the training sample set into the initial dust removal model to obtain a predicted dust-free image output by the initial dust removal model.
[0097] Exemplarily, for example, the initial dust removal model may be an existing mainstream dehazing model, such as dehazenet, dehazeFormer, gUnet, and the like.
[0098] S403. Determine the loss information of the initial dust removal model according to the difference between the predicted dust-free image and the dust-free image in the sample image pair, and iteratively correct the initial dust removal model according to the loss information until the corrected initial dust removal model meets the preset convergence condition, thereby obtaining the image dust removal model.
[0099] In one feasible manner, for example, the dust-bearing image M in each sample image pair of the training sample set is input into the initial dust removal model to obtain a predicted dust-free image A' output by the initial dust removal model; then, according to a preset loss function (such as L2 loss and similarity loss), the loss values of the predicted dust-free image A' and the dust-free image A in the sample image pair are calculated, that is, the initial dust removal model is supervised and trained using the dust-free image A; and the initial dust removal model is iteratively corrected according to the loss value calculated above. After the initial dust removal model is iteratively trained for multiple times, if the loss value of the initial dust removal model obtained by the current iterative training is converged, it means that the accuracy of the new model obtained by the training at this time cannot be significantly improved, and the iteration can be stopped. At this time, the initial dust removal model that meets the preset convergence conditions is used as the image dust removal model; if the initial dust removal model obtained by the current iterative training does not meet the preset convergence conditions, the iterative training process continues.
[0100] Optionally, refer to Figure 5 As shown, the method also includes:
[0101] S501 : performing target detection on a target dust-free image corresponding to an actual dust-bearing image to obtain at least one target object in the target dust-free image.
[0102] The target object may be a moving or non-moving object in an indoor working scene, such as a pedestrian, a pile of materials, etc.
[0103] S502: Perform enhancement processing on each target object in the target dust-free image to obtain an enhanced image.
[0104] Optionally, in this scheme, in order to ensure the efficiency and safety of remote operations, it is proposed that a target detection algorithm, such as the Canny operator, can be used to perform target detection on the target dust-free image output by the image dust removal model to obtain at least one target object in the target dust-free image, such as a pedestrian, a material pile, etc.; and an image enhancement algorithm is used to enhance each target object in the target dust-free image to obtain an enhanced image, that is, to enhance the display of key objects in the target dust-free image, so as to prompt the operator to pay attention to pedestrians during remote operations to avoid accidental injuries to pedestrians, etc., thereby further ensuring the efficiency and safety of remote operations.
[0105] Based on the same inventive concept, an image processing device corresponding to the image processing method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned image processing method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0106] Optionally, refer to Figure 6 As shown, the embodiment of the present application also provides an image processing device, which includes:
[0107] An acquisition module 601 is used to acquire a plurality of dust-free images in an indoor working scene, wherein the dust-free images are obtained by taking pictures in the indoor working scene, and when the dust-free images are acquired, the dust concentration in the indoor working scene is lower than a preset threshold value;
[0108] A generating module 602 is used to generate dust-containing images corresponding to each of the dust-free images according to each of the dust-free images and the pre-generated pure dust images, wherein the pure dust images are composed of dust elements, and the dust-containing images have the dust-free images as background and contain dust elements;
[0109] A construction module 603 is used to construct a training sample set according to each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images;
[0110] A training module 604 is used to train an image dust removal model based on the training sample set;
[0111] The dust removal module 605 is used to input the collected actual dust-bearing image in the indoor working scene into the image dust removal model to obtain a target dust-free image corresponding to the actual dust-bearing image.
[0112] Optionally, the generating module 602 is specifically configured to:
[0113] generating a depth image of the dust-free image according to the dust-free image;
[0114] A dust-containing image corresponding to the dust-free image is generated according to the dust-free image, the depth image and the pure dust image.
[0115] Optionally, the generating module 602 is specifically configured to:
[0116] The dust-free image is input into a pre-trained depth estimation model to obtain a depth image of the dust-free image output by the depth estimation model.
[0117] Optionally, the generating module 602 is specifically configured to:
[0118] According to the depth image, distribution information of the pure dust image in the dust-free image is determined, and based on the distribution information, an image with dust corresponding to the dust-free image is generated.
[0119] Optionally, the generating module 602 is specifically configured to:
[0120] Determining a distribution ratio of each pixel in the pure dust image in the dust-free image according to the dust scattering coefficient in the indoor working scene and the depth image of the dust-free image;
[0121] generating a first fused image according to the distribution ratio and the pure dust image;
[0122] generating a second fused image according to the distribution ratio and the dust-free image;
[0123] The first fused image and the second fused image are superimposed to generate an image with dust corresponding to the dust-free image.
[0124] Optionally, the acquisition module 601 is further used for:
[0125] An image of a target indoor scene is collected to obtain the pure dust image, wherein the target indoor scene includes a white background and dust with a concentration greater than a preset threshold.
[0126] Optionally, the construction module 603 is specifically configured to use each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images as a sample image pair in the training sample set;
[0127] The training module 604 is specifically used for:
[0128] Inputting the dust-carrying image in each sample image pair of the training sample set into the initial dust removal model to obtain a predicted dust-free image output by the initial dust removal model;
[0129] According to the difference between the predicted dust-free image and the dust-free image in the sample image pair, the loss information of the initial dust removal model is determined, and the initial dust removal model is iteratively corrected according to the loss information until the corrected initial dust removal model meets the preset convergence condition, so as to obtain the image dust removal model.
[0130] Optionally, the device further comprises:
[0131] A detection module, configured to perform target detection on a target dust-free image corresponding to the actual dust-bearing image, and obtain at least one target object in the target dust-free image;
[0132] The processing module is used to perform enhancement processing on each target object in the target dust-free image to obtain an enhanced image.
[0133] Optionally, the acquisition module 601 is specifically configured to:
[0134] Acquire multiple original dust-free images of indoor working scenes;
[0135] The original dust-free images are respectively subjected to a clearing process to generate a plurality of dust-free images; wherein the clearing process includes: a histogram equalization process and / or a neutral gray reduction process.
[0136] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.
[0137] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
[0138] Figure 7 This is a structural diagram of an electronic device provided in an embodiment of the present application. The electronic device includes: a processor 701, a storage medium 702 and a bus 703. The storage medium 702 stores machine-readable instructions executable by the processor 701. When the electronic device is running, the processor 701 communicates with the storage medium 702 through the bus 703. The processor 701 executes the machine-readable instructions to perform the following steps:
[0139] Acquire a plurality of dust-free images in an indoor working scene, wherein the dust-free images are taken in the indoor working scene, and when the dust-free images are collected, the dust concentration in the indoor working scene is lower than a preset threshold;
[0140] Generate dust-containing images corresponding to the dust-free images according to the dust-free images and the pre-generated pure dust images, wherein the pure dust images are composed of dust elements, and the dust-containing images are based on the dust-free images as background and contain dust elements;
[0141] Constructing a training sample set according to each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images, and obtaining an image dust removal model through training based on the training sample set;
[0142] The actual dust-bearing image collected in the indoor working scene is input into the image dust removal model to obtain a target dust-free image corresponding to the actual dust-bearing image.
[0143] Optionally, the processor 701 generates dust-bearing images corresponding to the dust-free images according to the dust-free images and the pre-generated pure dust images, specifically for:
[0144] generating a depth image of the dust-free image according to the dust-free image;
[0145] A dust-containing image corresponding to the dust-free image is generated according to the dust-free image, the depth image and the pure dust image.
[0146] Optionally, the processor 701, when executing the step of generating a depth image of the dust-free image according to the dust-free image, is specifically configured to:
[0147] The dust-free image is input into a pre-trained depth estimation model to obtain a depth image of the dust-free image output by the depth estimation model.
[0148] Optionally, the processor 701, when executing the step of generating the dust-bearing image corresponding to the dust-free image according to the dust-free image, the depth image and the pure dust image, is specifically configured to:
[0149] According to the depth image, distribution information of the pure dust image in the dust-free image is determined, and based on the distribution information, an image with dust corresponding to the dust-free image is generated.
[0150] Optionally, the processor 701, when executing the step of determining distribution information of the pure dust image in the dust-free image according to the depth image, and generating the dust-bearing image corresponding to the dust-free image based on the distribution information, is specifically configured to:
[0151] Determining a distribution ratio of each pixel in the pure dust image in the dust-free image according to the dust scattering coefficient in the indoor working scene and the depth image of the dust-free image;
[0152] generating a first fused image according to the distribution ratio and the pure dust image;
[0153] generating a second fused image according to the distribution ratio and the dust-free image;
[0154] The first fused image and the second fused image are superimposed to generate an image with dust corresponding to the dust-free image.
[0155] Optionally, the processor 701 is further configured to execute:
[0156] An image of a target indoor scene is collected to obtain the pure dust image, wherein the target indoor scene includes a white background and dust with a concentration greater than a preset threshold.
[0157] Optionally, the processor 701 executes the step of constructing a training sample set according to each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images, and training an image dust removal model based on the training sample set, specifically for:
[0158] taking each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images as a sample image pair in the training sample set;
[0159] Inputting the dust-carrying image in each sample image pair of the training sample set into the initial dust removal model to obtain a predicted dust-free image output by the initial dust removal model;
[0160] According to the difference between the predicted dust-free image and the dust-free image in the sample image pair, the loss information of the initial dust removal model is determined, and the initial dust removal model is iteratively corrected according to the loss information until the corrected initial dust removal model meets the preset convergence condition, so as to obtain the image dust removal model.
[0161] Optionally, the processor 701 is further configured to execute:
[0162] Performing target detection on a target dust-free image corresponding to the actual dust-bearing image to obtain at least one target object in the target dust-free image;
[0163] Each target object in the target dust-free image is enhanced to obtain an enhanced image.
[0164] Optionally, the processor 701, when executing the step of acquiring a plurality of dust-free images in an indoor operation scene, is specifically configured to:
[0165] Acquire multiple original dust-free images of indoor working scenes;
[0166] The original dust-free images are respectively subjected to a clearing process to generate a plurality of dust-free images; wherein the clearing process includes: a histogram equalization process and / or a neutral gray reduction process.
[0167] Optionally, the present invention further provides a program product, such as a computer-readable storage medium, comprising a program, which is used to execute when executed by a processor, and the processor performs the following steps:
[0168] Acquire multiple dust-free images in indoor working scenes;
[0169] Generate dust-containing images corresponding to the dust-free images according to the dust-free images and the pre-generated pure dust images, wherein the pure dust images are composed of dust elements, and the dust-containing images are based on the dust-free images as background and contain dust elements;
[0170] Constructing a training sample set according to each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images, and obtaining an image dust removal model through training based on the training sample set;
[0171] The actual dust-bearing image collected in the indoor working scene is input into the image dust removal model to obtain a target dust-free image corresponding to the actual dust-bearing image.
[0172] Optionally, the processor, when executing the step of generating the dust-bearing image corresponding to each of the dust-free images according to each of the dust-free images and the pre-generated pure dust image, is specifically configured to:
[0173] generating a depth image of the dust-free image according to the dust-free image;
[0174] A dust-containing image corresponding to the dust-free image is generated according to the dust-free image, the depth image and the pure dust image.
[0175] Optionally, when the processor generates a depth image of the dust-free image according to the dust-free image, the processor is specifically configured to:
[0176] The dust-free image is input into a pre-trained depth estimation model to obtain a depth image of the dust-free image output by the depth estimation model.
[0177] Optionally, when the processor executes the step of generating the dust-bearing image corresponding to the dust-free image according to the dust-free image, the depth image and the pure dust image, the processor is specifically configured to:
[0178] According to the depth image, distribution information of the pure dust image in the dust-free image is determined, and based on the distribution information, an image with dust corresponding to the dust-free image is generated.
[0179] Optionally, the processor, when executing the step of determining distribution information of the pure dust image in the dust-free image according to the depth image, and generating the dust-bearing image corresponding to the dust-free image based on the distribution information, is specifically configured to:
[0180] Determining a distribution ratio of each pixel in the pure dust image in the dust-free image according to the dust scattering coefficient in the indoor working scene and the depth image of the dust-free image;
[0181] generating a first fused image according to the distribution ratio and the pure dust image;
[0182] generating a second fused image according to the distribution ratio and the dust-free image;
[0183] The first fused image and the second fused image are superimposed to generate an image with dust corresponding to the dust-free image.
[0184] Optionally, the processor is further configured to execute:
[0185] An image of a target indoor scene is collected to obtain the pure dust image, wherein the target indoor scene includes a white background and dust with a concentration greater than a preset threshold.
[0186] Optionally, the processor executes the step of constructing a training sample set according to each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images, and training an image dust removal model based on the training sample set, specifically for:
[0187] taking each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images as a sample image pair in the training sample set;
[0188] Inputting the dust-carrying image in each sample image pair of the training sample set into the initial dust removal model to obtain a predicted dust-free image output by the initial dust removal model;
[0189] According to the difference between the predicted dust-free image and the dust-free image in the sample image pair, the loss information of the initial dust removal model is determined, and the initial dust removal model is iteratively corrected according to the loss information until the corrected initial dust removal model meets the preset convergence condition, so as to obtain the image dust removal model.
[0190] Optionally, the processor is further configured to execute:
[0191] Performing target detection on a target dust-free image corresponding to the actual dust-bearing image to obtain at least one target object in the target dust-free image;
[0192] Each target object in the target dust-free image is enhanced to obtain an enhanced image.
[0193] Optionally, the processor, when executing the step of acquiring a plurality of dust-free images in an indoor operation scene, is specifically configured to:
[0194] Acquire multiple original dust-free images of indoor working scenes;
[0195] The original dust-free images are respectively subjected to a clearing process to generate a plurality of dust-free images; wherein the clearing process includes: a histogram equalization process and / or a neutral gray reduction process.
[0196] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0197] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0198] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0199] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.
Claims
1. An image processing method, characterized in that: The method comprises: Acquire a plurality of dust-free images in an indoor working scene, wherein the dust-free images are taken in the indoor working scene, and when the dust-free images are collected, the dust concentration in the indoor working scene is lower than a preset threshold; Generate dust-containing images corresponding to the dust-free images according to the dust-free images and the pre-generated pure dust images, wherein the pure dust images are composed of dust elements, and the dust-containing images are based on the dust-free images as background and contain dust elements; Constructing a training sample set according to each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images, and obtaining an image dust removal model through training based on the training sample set; The actual dust-bearing image collected in the indoor working scene is input into the image dust removal model to obtain a target dust-free image corresponding to the actual dust-bearing image.
2. The method according to claim 1, characterized in that The step of generating dust-bearing images corresponding to the dust-free images according to the dust-free images and the pre-generated pure dust images includes: generating a depth image of the dust-free image according to the dust-free image; A dust-containing image corresponding to the dust-free image is generated according to the dust-free image, the depth image and the pure dust image.
3. The method according to claim 2, characterized in that Generating a depth image of the dust-free image according to the dust-free image includes: The dust-free image is input into a pre-trained depth estimation model to obtain a depth image of the dust-free image output by the depth estimation model.
4. The method according to claim 2, characterized in that: The step of generating the dust-bearing image corresponding to the dust-free image according to the dust-free image, the depth image and the pure dust image includes: According to the depth image, distribution information of the pure dust image in the dust-free image is determined, and based on the distribution information, an image with dust corresponding to the dust-free image is generated.
5. The method according to claim 4, characterized in that The step of determining distribution information of the pure dust image in the dust-free image according to the depth image, and generating the dust-bearing image corresponding to the dust-free image based on the distribution information, includes: Determining a distribution ratio of each pixel in the pure dust image in the dust-free image according to the dust scattering coefficient in the indoor working scene and the depth image of the dust-free image; generating a first fused image according to the distribution ratio and the pure dust image; generating a second fused image according to the distribution ratio and the dust-free image; The first fused image and the second fused image are superimposed to generate an image with dust corresponding to the dust-free image.
6. The method according to claim 1, characterized in that The pure dust image is obtained by the following steps: An image of a target indoor scene is collected to obtain the pure dust image, wherein the target indoor scene includes a white background and dust with a concentration greater than a preset threshold.
7. The method according to claim 1, characterized in that The step of constructing a training sample set according to each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images, and obtaining an image dust removal model based on the training sample set includes: taking each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images as a sample image pair in the training sample set; Inputting the dust-carrying image in each sample image pair of the training sample set into the initial dust removal model to obtain a predicted dust-free image output by the initial dust removal model; According to the difference between the predicted dust-free image and the dust-free image in the sample image pair, the loss information of the initial dust removal model is determined, and the initial dust removal model is iteratively corrected according to the loss information until the corrected initial dust removal model meets the preset convergence condition, so as to obtain the image dust removal model.
8. The method according to any one of claims 1 to 7, characterized in that: Also includes: Performing target detection on a target dust-free image corresponding to the actual dust-bearing image to obtain at least one target object in the target dust-free image; Each target object in the target dust-free image is enhanced to obtain an enhanced image.
9. The method according to claim 1, characterized in that: The step of acquiring a plurality of dust-free images in an indoor operation scene includes: Acquire multiple original dust-free images of indoor working scenes; The original dust-free images are respectively subjected to a clearing process to generate a plurality of dust-free images; wherein the clearing process includes: a histogram equalization process and / or a neutral gray reduction process.
10. An image processing device, characterized in that: The device comprises: An acquisition module, used to acquire multiple dust-free images in indoor working scenes; A generating module, configured to generate dust-bearing images corresponding to the dust-free images according to the dust-free images and the pre-generated pure dust images, wherein the pure dust images are composed of dust elements, and the dust-bearing images are based on the dust-free images as background and contain dust elements; A construction module, used to construct a training sample set according to each of the dust-free images and the dust-bearing images corresponding to each of the dust-free images; A training module, used for training an image dust removal model based on the training sample set; The dust removal module is used to input the actual dust-bearing image collected in the indoor working scene into the image dust removal model to obtain the target dust-free image corresponding to the actual dust-bearing image.
11. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of any method as described in claims 1-9.
12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is executed.