Image restoration processing method and device, electronic equipment and storage medium
Through the image repair processing method, the position and posture information of the coal machine drum in the working surface scene is extracted, and the obstructed image is automatically repaired, solving the problem of inaccurate state perception of the coal machine drum and improving safety and efficiency.
Patent Information
- Application Number
- CN202411681194.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-04-25
AI Technical Summary
In the state perception of coal machine drums, due to the complexity and variability of the working face scene, coal machine drums are often blocked by obstacles such as top beams, guard plates, floating coal, etc., resulting in inaccurate state perception, affecting normal operation and safety.
An image repair processing method is provided. By obtaining the image to be repaired, the positional relationship and attitude information of the target object and the imaging device are extracted, the pre-trained repair model is called, and the image and control signal data are input in parallel, the repaired image is output, the occlusion is eliminated, and the complete state of the target object is restored.
It realizes automatic repair of the obscured image of the target object and restores its complete state, improves the accuracy of the state perception of the coal machine drum, and enhances the safety and efficiency of underground operations.
Smart Images
Figure CN119887583A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of fully mechanized mining working faces, and in particular to an image restoration processing method, device, electronic equipment and storage medium. Background Art
[0002] In modern mining, the complexity and danger of underground working faces have put forward extremely high requirements for safe production. In order to effectively monitor and manage the underground working environment, in recent years, intelligent video monitoring systems for working faces based on computer vision perception technology have been widely used. By integrating advanced image processing, pattern recognition and machine learning algorithms, the system realizes the automatic detection and identification of key parameters related to equipment, surrounding rock and environmental status in the mining area. At the same time, combined with intelligent monitoring technology, the system can timely detect and warn potential abnormal conditions, thereby significantly improving the technical means of mining area video monitoring and providing strong support for visual intelligent management and control of working faces.
[0003] However, although the existing intelligent video monitoring system has achieved remarkable results in underground working face monitoring, it still faces some challenges in practical application. In particular, in terms of the state perception of the coal machine drum, due to the complexity and variability of the working face scene, the coal machine drum is often blocked by obstacles such as the top beam, guard plate, floating coal, etc., resulting in limited accurate perception of its state. This not only affects the normal operation monitoring of the coal machine drum, but also brings potential risks to the safety and efficiency of underground operations.
[0004] Therefore, finding an image restoration method that can automatically repair the occluded target object and restore its complete state has become a current research hotspot. Summary of the invention
[0005] The present invention provides an image restoration processing method, device, electronic device and storage medium, which can automatically restore an image to be restored where a target object is blocked, thereby restoring its complete state and obtaining a restored image.
[0006] The present invention provides an image restoration processing method, the method comprising: acquiring an image to be restored, wherein the image to be restored is an image in which a target object in the image to be restored is blocked by an occluder; based on the image to be restored, obtaining control signal data corresponding to the image to be restored, wherein the control signal data is used to characterize the positional relationship between the target object in the image to be restored and a camera device that collects the image to be restored, as well as posture information of the target object; calling a pre-trained restoration model, and inputting the image to be restored and the control signal data corresponding to the image to be restored into the restoration model in parallel, to obtain a restored image output by the restoration model that matches the image to be restored, wherein the restored image is an image containing the target object with the occluder removed.
[0007] According to an image restoration processing method provided by the present invention, the control signal data corresponding to the image to be restored is obtained based on the image to be restored, which specifically includes: calling a pre-trained detection model, wherein the detection model is used to output the pixel size of the target object in the image to be restored, and the position information of the target object in the image to be restored; inputting the image to be restored into the detection model to obtain the pixel size of the target object in the image to be restored output by the detection model, and the position information of the target object in the image to be restored; acquiring the actual size information of the target object, the heading angle posture of the target object, and the posture information of the camera device relative to the target object; based on the pixel size of the target object in the image to be restored, the position information of the target object in the image to be restored, the actual size information of the target object, the heading angle posture of the target object, and the posture information of the camera device relative to the target object, the control signal data corresponding to the image to be restored is obtained.
[0008] According to an image restoration processing method provided by the present invention, the control signal data corresponding to the image to be restored is obtained based on the pixel size of the target object in the image to be restored, the position information of the target object in the image to be restored, the actual size information of the target object, the heading angle posture of the target object, and the posture information of the camera device relative to the target object. The method specifically includes: splicing the pixel size of the target object in the image to be restored, the position information of the target object in the image to be restored, the actual size information of the target object, the heading angle posture of the target object, and the posture information of the camera device relative to the target object to obtain a feature vector; and normalizing the feature vector to obtain the control signal data corresponding to the image to be restored.
[0009] According to an image restoration processing method provided by the present invention, the restoration model is trained in the following manner: a training sample data set is obtained, wherein the training sample data set includes multiple training sample data, and the training sample data includes a training image without occlusion, a training image with occlusion matching the training image without occlusion, and control signal training sample data matching the training image with occlusion, wherein the target object in the training image without occlusion is not occluded, and the target object in the training image with occlusion is occluded; the restoration model is trained based on the training sample data set to obtain the trained restoration model.
[0010] According to an image restoration processing method provided by the present invention, an occluded training image that matches the unoccluded training image is obtained in the following manner: an occluded reference image is obtained, and semantic segmentation is performed on the occluded reference image to extract an occluded area image; the occluded area image is enhanced to obtain an enhanced occluded area image; the enhanced occluded area image is matched and merged with the unoccluded training image to obtain an occluded training image that matches the unoccluded training image.
[0011] According to an image restoration processing method provided by the present invention, an occluded training image that matches the unoccluded training image is obtained in the following manner: calling a pre-trained Control-Net image generation model, wherein the Control-Net image generation model is used to generate an occluded training image; obtaining an occluded reference image, and processing the occluded reference image based on an edge detection operator to obtain an edge binary image; inputting the unoccluded training image and the edge binary image into the Control-Net image generation model to obtain an occluded training image output by the Control-Net image generation model that matches the unoccluded training image.
[0012] The present invention also provides an image restoration processing device, which includes: an acquisition module, used to acquire an image to be restored, wherein the image to be restored is an image in which a target object in the image to be restored is obscured by an occluder; a processing module, used to obtain control signal data corresponding to the image to be restored based on the image to be restored, wherein the control signal data is used to characterize the positional relationship between the target object in the image to be restored and a camera device that collects the image to be restored, as well as posture information of the target object; a restoration module, used to call a pre-trained restoration model, and input the image to be restored and the control signal data corresponding to the image to be restored into the restoration model in parallel, to obtain a restored image output by the restoration model that matches the image to be restored, wherein the restored image is an image containing the target object with the occluder removed.
[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned image restoration processing methods is implemented.
[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image restoration processing method described in any one of the above is implemented.
[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image restoration processing method described above is implemented.
[0016] The image restoration processing method, device, electronic device and storage medium provided by the present invention obtain an image to be restored, wherein the image to be restored is an image in which a target object in the image to be restored is blocked by an occluder; based on the image to be restored, control signal data corresponding to the image to be restored is obtained; a pre-trained restoration model is called, and the image to be restored and the control signal data corresponding to the image to be restored are input into the restoration model in parallel, so that a restored image output by the restoration model and matching the image to be restored can be automatically obtained. The image to be restored in which the target object is blocked can be automatically restored, so as to restore its complete state and obtain a restored image. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 It is a flow chart of the image restoration processing method provided by the present invention.
[0019] Figure 2 It is a flow chart of obtaining control signal data corresponding to the image to be repaired based on the image to be repaired provided by the present invention.
[0020] Figure 3 This is one of the flow charts of obtaining an occluded training image that matches an unoccluded training image provided by the present invention.
[0021] Figure 4 This is the second flow chart of obtaining an occluded training image that matches an unoccluded training image provided by the present invention.
[0022] Figure 5 It is a structural schematic diagram of the image restoration processing device provided by the present invention.
[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] The image restoration processing method provided by the present invention can accurately restore the image of the obscured area of the drum target, and effectively solve the problem that the drum target object in the working face scene is often obscured by the top beam, guard plate, floating coal, etc. in the image, resulting in limited state perception.
[0026] Figure 1 It is a flow chart of the image restoration processing method provided by the present invention.
[0027] The following will be combined Figure 1 The process of the image restoration processing method provided by the present invention is described.
[0028] In an exemplary embodiment of the present invention, Figure 1 It can be seen that the image restoration processing method may include step 110 to step 130, and each step will be introduced below.
[0029] In step 110, an image to be repaired is obtained, wherein the image to be repaired is an image in which a target object in the image to be repaired is blocked by an occluding object.
[0030] In one embodiment, an image to be repaired may be obtained, wherein the image to be repaired may be an image in which a target object in the image to be repaired is blocked by an obstruction. In one example, the image to be repaired may be an image collected from a mine shaft and includes an image blocked by a roller, and it can be considered that the target object may be a key equipment such as a roller.
[0031] In step 120, based on the image to be repaired, control signal data corresponding to the image to be repaired is obtained, wherein the control signal data is used to characterize the positional relationship between the target object in the image to be repaired and the camera device that captures the image to be repaired, as well as the posture information of the target object.
[0032] In step 130, a pre-trained restoration model is called, and the image to be restored and the control signal data corresponding to the image to be restored are input into the restoration model in parallel, so as to obtain a restored image output by the restoration model that matches the image to be restored, wherein the restored image is an image containing the target object with occluders removed.
[0033] In the process of repairing to obtain an unobstructed repaired image, the occluded area of the occluded image to be repaired is actually the projection of the target object on the current image plane, and the projection result is closely related to the relative position relationship between the camera and the target object and the posture of the target object itself. Therefore, in the process of obtaining a repaired image that matches the image to be repaired based on the repair model, the above factors need to be considered.
[0034] Therefore, in one embodiment, control signal data corresponding to the image to be repaired can be obtained based on the image to be repaired, wherein the control signal data can be used to characterize the positional relationship between the target object in the image to be repaired and the camera device that captures the image to be repaired, as well as the posture information of the target object. Further, a pre-trained repair model is called, and the image to be repaired and the control signal data corresponding to the image to be repaired are input into the repair model in parallel, so that a repaired image output by the repair model that matches the image to be repaired can be obtained. Since the influence of the control signal data is comprehensively considered in this embodiment, it can be ensured that the repaired image matching the image to be repaired based on the output of the repair model is more real and accurate.
[0035] The image restoration processing method, device, electronic device and storage medium provided by the present invention obtain an image to be restored, wherein the image to be restored is an image in which a target object in the image to be restored is blocked by an occluder; based on the image to be restored, control signal data corresponding to the image to be restored is obtained; a pre-trained restoration model is called, and the image to be restored and the control signal data corresponding to the image to be restored are input into the restoration model in parallel, so that a restored image output by the restoration model and matching the image to be restored can be automatically obtained. The image to be restored in which the target object is blocked can be automatically restored, so as to restore its complete state and obtain a restored image.
[0036] Figure 2 It is a flow chart of obtaining control signal data corresponding to the image to be repaired based on the image to be repaired provided by the present invention.
[0037] The following will be combined Figure 2 The process of obtaining control signal data corresponding to the image to be repaired based on the image to be repaired provided by the present invention is described.
[0038] In an exemplary embodiment of the present invention, Figure 2 It can be known that, based on the image to be restored, obtaining the control signal data corresponding to the image to be restored may include steps 210 to 240, and each step will be introduced below.
[0039] In step 210, a pre-trained detection model is called, wherein the detection model is used to output the pixel size of the target object in the image to be repaired and the position information of the target object in the image to be repaired.
[0040] In step 220, the image to be repaired is input into the detection model to obtain the pixel size of the target object in the image to be repaired and the position information of the target object in the image to be repaired output by the detection model.
[0041] In one embodiment, a pre-trained detection model can be called, and the image to be repaired can be input into the detection model, so as to obtain the pixel size of the target object in the image to be repaired (for example, the pixel size of the drum target in the image) and the position information of the target object in the image to be repaired (for example, the position information of the drum target in the image) output by the detection model.
[0042] It should be noted that the detection model can be obtained by pre-training the training sample data set, and the detection model can be used to output the pixel size of the target object in the image to be repaired, as well as the position information of the target object in the image to be repaired. In the process of pre-training the training sample data set to obtain the detection model, a plurality of training sample data images of target objects blocked by occluders can be obtained; further, part of the training sample data images are annotated to obtain the annotated information of the drum target, and then the annotated information and the corresponding training sample data images are used to train the detection model.
[0043] In step 230, the actual size information of the target object, the heading angle posture of the target object, and the posture information of the camera device relative to the target object are obtained.
[0044] In step 240, control signal data corresponding to the image to be repaired is obtained based on the pixel size of the target object in the image to be repaired, the position information of the target object in the image to be repaired, the actual size information of the target object, the heading angle attitude of the target object, and the attitude information of the camera device relative to the target object.
[0045] In another embodiment, the actual size information of the target object, the heading angle attitude of the target object, and the attitude information of the camera device relative to the target object can be obtained. For the sake of convenience, the target object can be described by taking a roller as an example. In the application process, the actual size information of the target object (such as the actual size of the roller), the heading angle attitude of the target object (such as the heading angle attitude data of the roller), and the attitude information of the camera device relative to the target object (such as the camera attitude data) can be obtained. Further, based on the pixel size of the target object in the image to be repaired, the position information of the target object in the image to be repaired, the actual size information of the target object, the heading angle attitude of the target object, and the attitude information of the camera device relative to the target object, the control signal data corresponding to the image to be repaired is obtained. In this embodiment, based on the control signal data, the projection result (the blocked area) is comprehensively considered to be closely related to the relative positional relationship between the camera and the target object and the attitude of the target object itself, thereby laying a foundation for outputting a more real and accurate repaired image that matches the image to be repaired based on the repair model.
[0046] In another exemplary embodiment of the present invention, continuing with the above Figure 2 The above embodiment is used as an example to illustrate that based on the pixel size of the target object in the image to be repaired, the position information of the target object in the image to be repaired, the actual size information of the target object, the heading angle attitude of the target object, and the attitude information of the camera device relative to the target object, the control signal data corresponding to the image to be repaired is obtained, which can be implemented in the following manner: The pixel size of the target object in the image to be restored, the position information of the target object in the image to be restored, the actual size information of the target object, the heading angle posture of the target object, and the posture information of the camera device relative to the target object are spliced to obtain a feature vector; The feature vector is normalized to obtain control signal data corresponding to the image to be repaired.
[0047] In one embodiment, continuing with the example of the target object being a roller, the pixel size of the roller target in the image, the position information of the roller target in the image, the actual size of the roller, the roller heading angle posture data, and the camera posture data can be spliced to form a feature vector. Furthermore, each dimension data of the feature vector is normalized separately to finally obtain control signal data.
[0048] In another exemplary embodiment of the present invention, the above-mentioned embodiment is used as an example to illustrate that the repair model can be trained in the following manner: Acquire a training sample data set, wherein the training sample data set includes a plurality of training sample data, and the training sample data includes a training image under no occlusion, a training image under occlusion matching the training image under no occlusion, and control signal training sample data matching the training image under occlusion, wherein the target object in the training image under no occlusion is not occluded, and the target object in the training image under occlusion is occluded; The repair model is trained based on the training sample data set to obtain the trained repair model.
[0049] In one embodiment, a large number of training sample data sets can be obtained, and the repair model can be trained based on the training sample data sets, so that a trained repair model can be obtained. The training sample data set may include multiple training sample data. The training sample data may include unoccluded training images, occluded training images that match the unoccluded training images, and control signal training sample data that match the occluded training images, wherein the target object in the unoccluded training images is not occluded, and the target object in the occluded training images is occluded. In this embodiment, in the process of training the repair model, the influence of the control signal training sample data is comprehensively considered, so as to ensure that the large repair model obtained by training has higher repair accuracy.
[0050] Figure 3 This is one of the flow charts of obtaining an occluded training image that matches an unoccluded training image provided by the present invention.
[0051] The following will be combined Figure 3The process of obtaining an occluded training image that matches an unoccluded training image provided by the present invention is described.
[0052] In an exemplary embodiment of the present invention, Figure 3 It can be seen that obtaining the occluded training image that matches the unoccluded training image may include steps 310 to 330, and each step will be described below.
[0053] In step 310, a reference image with occlusion is obtained, and semantic segmentation is performed on the reference image with occlusion to extract an image of the occlusion area; In step 320, the occluder region image is enhanced to obtain an enhanced occluder region image; In step 330, the enhanced occlusion region image is matched and merged with the unoccluded training image to obtain an occluded training image that matches the unoccluded training image.
[0054] It should be noted that in order to ensure that the restoration model trained based on the training sample data set has high accuracy, it is necessary to ensure the rationality and accuracy of the training sample data set. In one example, a reference image with occlusion can be obtained, that is, an image in the occlusion image set is obtained, and the reference image with occlusion is processed by semantic segmentation, so that an image of the occlusion area can be extracted, that is, an image of the occlusion area is extracted from the semantic segmentation result.
[0055] Furthermore, the occlusion area image is enhanced to obtain an enhanced occlusion area image, and the enhanced occlusion area image is matched and merged with the unoccluded training image, so as to obtain an occluded training image that matches the unoccluded training image.
[0056] It should be noted that the unobstructed training images are video frame images obtained based on actual video capture in mines, which improves the practical applicability of the unobstructed training images. Since the occluded training images are obtained based on the unobstructed training images, the practical applicability of the occluded training images is also improved, as well as the accuracy of the occluded training images compared with the unobstructed training images, thereby laying the foundation for obtaining a more accurate restoration model based on the training sample data set.
[0057] Figure 4 This is the second flow chart of obtaining an occluded training image that matches an unoccluded training image provided by the present invention.
[0058] The following will be combined Figure 4 Another process of obtaining an occluded training image that matches an unoccluded training image is described.
[0059] In an exemplary embodiment of the present invention, Figure 4 It can be seen that obtaining the occluded training image that matches the unoccluded training image may include steps 410 to 430, and each step will be described below.
[0060] In step 410, a pre-trained Control-Net image generation model is called, wherein the Control-Net image generation model is used to generate training images with occlusion.
[0061] In step 420, a reference image with occlusion is obtained, and the reference image with occlusion is processed based on an edge detection operator to obtain an edge binary image.
[0062] In step 430, the unoccluded training image and the edge binary map are input into the Control-Net image generation model to obtain the occluded training image output by the Control-Net image generation model that matches the unoccluded training image.
[0063] In one embodiment, a pre-trained Control-Net image generation model may be called, wherein the Control-Net image generation model may be used to generate training images with occlusion. It should be noted that the Control-Net image generation model is trained based on a training sample data set.
[0064] Furthermore, a reference image with occlusion can be obtained, and the reference image with occlusion can be processed based on the edge detection operator to obtain an edge binary map. Then, the training image without occlusion and the edge binary map are input into the Control-Net image generation model to obtain an occlusion training image output by the Control-Net image generation model that matches the training image without occlusion.
[0065] In this embodiment, the unobstructed training images are video frame images obtained based on actual video capture in a mine, thereby improving the practical applicability of the unobstructed training images. Since the occluded training images are obtained based on the unobstructed training images, the practical applicability of the occluded training images and the accuracy of the unobstructed training images are also improved, thereby laying a foundation for obtaining a more accurate restoration model based on the training sample data set.
[0066] According to the foregoing description, the image restoration processing method, device, electronic device and storage medium provided by the present invention obtain an image to be restored, wherein the image to be restored is an image in which the target object in the image to be restored is blocked by an occluder; based on the image to be restored, control signal data corresponding to the image to be restored is obtained; a pre-trained restoration model is called, and the image to be restored and the control signal data corresponding to the image to be restored are input into the restoration model in parallel, so that a restored image output by the restoration model that matches the image to be restored can be automatically obtained. The image to be restored in which the target object is blocked can be automatically restored, thereby restoring its complete state and obtaining a restored image.
[0067] The image restoration processing device provided by the present invention is described below. The image restoration processing device described below and the image restoration processing method described above can be referred to each other.
[0068] Figure 5 It is a structural schematic diagram of the image restoration processing device provided by the present invention.
[0069] The following will be combined Figure 5 The structure of the image restoration processing device is described.
[0070] In an exemplary embodiment of the present invention, Figure 5 It can be known that the image restoration processing device may include an acquisition module 510, a processing module 520, and a restoration module 530, and each module will be introduced below.
[0071] The acquisition module 510 may be configured to acquire an image to be restored, wherein the image to be restored is an image in which a target object in the image to be restored is blocked by an occluding object; The processing module 520 may be configured to obtain control signal data corresponding to the image to be restored based on the image to be restored, wherein the control signal data is used to represent the positional relationship between the target object in the image to be restored and the camera device that captures the image to be restored, and the posture information of the target object; The repair module 530 can be configured to call a pre-trained repair model, and input the image to be repaired and the control signal data corresponding to the image to be repaired into the repair model in parallel, so as to obtain a repaired image output by the repair model that matches the image to be repaired, wherein the repaired image is an image containing the target object with occluders removed.
[0072] In an exemplary embodiment of the present invention, the processing module 520 may obtain control signal data corresponding to the image to be restored based on the image to be restored in the following manner: Calling a pre-trained detection model, wherein the detection model is used to output the pixel size of the target object in the image to be repaired, and the position information of the target object in the image to be repaired; Inputting the image to be repaired into the detection model, obtaining the pixel size of the target object in the image to be repaired output by the detection model, and the position information of the target object in the image to be repaired; Acquire actual size information of the target object, heading angle posture of the target object, and posture information of the camera device relative to the target object; Based on the pixel size of the target object in the image to be repaired, the position information of the target object in the image to be repaired, the actual size information of the target object, the heading angle attitude of the target object, and the attitude information of the camera device relative to the target object, control signal data corresponding to the image to be repaired is obtained.
[0073] In an exemplary embodiment of the present invention, the processing module 520 may obtain control signal data corresponding to the image to be restored based on the pixel size of the target object in the image to be restored, the position information of the target object in the image to be restored, the actual size information of the target object, the heading angle attitude of the target object, and the attitude information of the camera device relative to the target object in the following manner: The pixel size of the target object in the image to be restored, the position information of the target object in the image to be restored, the actual size information of the target object, the heading angle posture of the target object, and the posture information of the camera device relative to the target object are stitched together to obtain a feature vector; The feature vector is normalized to obtain control signal data corresponding to the image to be repaired.
[0074] In an exemplary embodiment of the present invention, the repair module 530 may implement training to obtain the repair model in the following manner: Acquire a training sample data set, wherein the training sample data set includes a plurality of training sample data, and the training sample data includes a training image under no occlusion, a training image under occlusion matching the training image under no occlusion, and control signal training sample data matching the training image under occlusion, wherein the target object in the training image under no occlusion is not occluded, and the target object in the training image under occlusion is occluded; The repair model is trained based on the training sample data set to obtain the trained repair model.
[0075] In an exemplary embodiment of the present invention, the restoration module 530 may obtain the occluded training image that matches the unoccluded training image in the following manner: Obtaining a reference image with occlusion, and performing semantic segmentation processing on the reference image with occlusion to extract an image of the occlusion area; Performing enhancement processing on the occluder region image to obtain an enhanced occluder region image; The enhanced occlusion region image is matched and merged with the unoccluded training image to obtain an occluded training image that matches the unoccluded training image.
[0076] In an exemplary embodiment of the present invention, the restoration module 530 may obtain the occluded training image that matches the unoccluded training image in the following manner: Calling a pre-trained Control-Net image generation model, wherein the Control-Net image generation model is used to generate training images with occlusion; Acquire a reference image with occlusion, and process the reference image with occlusion based on an edge detection operator to obtain an edge binary image; The unobstructed training image and the edge binary map are input into the Control-Net image generation model to obtain the occluded training image output by the Control-Net image generation model and matching the unobstructed training image.
[0077] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the image restoration processing method, which includes: obtaining an image to be restored, wherein the image to be restored is an image in which a target object in the image to be restored is blocked by an occluder; based on the image to be restored, obtaining control signal data corresponding to the image to be restored, wherein the control signal data is used to characterize the positional relationship between the target object in the image to be restored and the camera device that collects the image to be restored, and the posture information of the target object; calling a pre-trained restoration model, and inputting the image to be restored and the control signal data corresponding to the image to be restored into the restoration model in parallel, to obtain a restored image output by the restoration model that matches the image to be restored, wherein the restored image is an image containing the target object after removing the occluder.
[0078] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0079] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image restoration processing method provided by the above-mentioned methods, which includes: obtaining an image to be restored, wherein the image to be restored is an image in which a target object in the image to be restored is obscured by an occluder; based on the image to be restored, obtaining control signal data corresponding to the image to be restored, wherein the control signal data is used to characterize the positional relationship between the target object in the image to be restored and a camera device that collects the image to be restored, as well as posture information of the target object; calling a pre-trained restoration model, and inputting the image to be restored and the control signal data corresponding to the image to be restored into the restoration model in parallel, to obtain a restored image output by the restoration model that matches the image to be restored, wherein the restored image is an image containing the target object with the occluder removed.
[0080] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image restoration processing method provided by the above-mentioned methods, the method comprising: obtaining an image to be restored, wherein the image to be restored is an image in which a target object in the image to be restored is obscured by an occluder; based on the image to be restored, obtaining control signal data corresponding to the image to be restored, wherein the control signal data is used to characterize the positional relationship between the target object in the image to be restored and a camera device that collects the image to be restored, and posture information of the target object; calling a pre-trained restoration model, and inputting the image to be restored and the control signal data corresponding to the image to be restored into the restoration model in parallel, to obtain a restored image output by the restoration model that matches the image to be restored, wherein the restored image is an image containing the target object with the occluder removed.
[0081] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An image restoration method, characterized in that: The method comprises: Acquire an image to be repaired, wherein the image to be repaired is an image in which a target object in the image to be repaired is blocked by an occluding object; Based on the image to be repaired, obtaining control signal data corresponding to the image to be repaired, wherein the control signal data is used to represent the positional relationship between the target object in the image to be repaired and the camera device that captures the image to be repaired, as well as the posture information of the target object; A pre-trained restoration model is called, and the image to be restored and the control signal data corresponding to the image to be restored are input into the restoration model in parallel, so as to obtain a restored image output by the restoration model that matches the image to be restored, wherein the restored image is an image containing the target object with occluders removed.
2. The image restoration method according to claim 1, characterized in that: The obtaining, based on the image to be repaired, control signal data corresponding to the image to be repaired specifically includes: Calling a pre-trained detection model, wherein the detection model is used to output the pixel size of the target object in the image to be repaired, and the position information of the target object in the image to be repaired; Inputting the image to be repaired into the detection model, obtaining the pixel size of the target object in the image to be repaired output by the detection model, and the position information of the target object in the image to be repaired; Acquire actual size information of the target object, heading angle posture of the target object, and posture information of the camera device relative to the target object; Based on the pixel size of the target object in the image to be repaired, the position information of the target object in the image to be repaired, the actual size information of the target object, the heading angle attitude of the target object, and the attitude information of the camera device relative to the target object, control signal data corresponding to the image to be repaired is obtained.
3. The image restoration method according to claim 2, characterized in that: The obtaining of control signal data corresponding to the image to be restored based on the pixel size of the target object in the image to be restored, the position information of the target object in the image to be restored, the actual size information of the target object, the heading angle attitude of the target object, and the attitude information of the camera device relative to the target object specifically includes: The pixel size of the target object in the image to be restored, the position information of the target object in the image to be restored, the actual size information of the target object, the heading angle posture of the target object, and the posture information of the camera device relative to the target object are stitched together to obtain a feature vector; The feature vector is normalized to obtain control signal data corresponding to the image to be repaired.
4. The image restoration method according to any one of claims 1 to 3, characterized in that: The repair model is trained in the following way: Acquire a training sample data set, wherein the training sample data set includes a plurality of training sample data, and the training sample data includes a training image under no occlusion, a training image under occlusion matching the training image under no occlusion, and control signal training sample data matching the training image under occlusion, wherein the target object in the training image under no occlusion is not occluded, and the target object in the training image under occlusion is occluded; The repair model is trained based on the training sample data set to obtain the trained repair model.
5. The image restoration method according to claim 4, characterized in that: The occluded training image that matches the unoccluded training image is obtained in the following manner: Obtaining a reference image with occlusion, and performing semantic segmentation processing on the reference image with occlusion to extract an image of the occlusion area; Performing enhancement processing on the occluder region image to obtain an enhanced occluder region image; The enhanced occlusion region image is matched and merged with the unoccluded training image to obtain an occluded training image that matches the unoccluded training image.
6. The image restoration method according to claim 4, characterized in that: The occluded training image that matches the unoccluded training image is obtained in the following manner: Calling a pre-trained Control-Net image generation model, wherein the Control-Net image generation model is used to generate training images with occlusion; Acquire a reference image with occlusion, and process the reference image with occlusion based on an edge detection operator to obtain an edge binary image; The unobstructed training image and the edge binary map are input into the Control-Net image generation model to obtain the occluded training image output by the Control-Net image generation model and matching the unobstructed training image.
7. An image restoration processing device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be repaired, wherein the image to be repaired is an image in which a target object in the image to be repaired is blocked by an occluding object; A processing module, configured to obtain control signal data corresponding to the image to be restored based on the image to be restored, wherein the control signal data is used to represent the positional relationship between the target object in the image to be restored and the camera device that captures the image to be restored, as well as the posture information of the target object; The repair module is used to call a pre-trained repair model, and input the image to be repaired and the control signal data corresponding to the image to be repaired into the repair model in parallel, so as to obtain a repaired image output by the repair model that matches the image to be repaired, wherein the repaired image is an image containing the target object with occluders removed.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image restoration method according to any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image restoration method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image restoration method according to any one of claims 1 to 6 is implemented.