Image inpainting method and device, nonvolatile storage medium and electronic equipment
By performing multiple block processing and motion blur removal on the image, combined with a convolutional neural network model, the problem of loss of non-key frame image features in image acquisition devices was solved, and the complete recovery of key image information was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU BRONCUS MEDICAL CO LTD
- Filing Date
- 2022-12-22
- Publication Date
- 2026-06-23
AI Technical Summary
In existing technologies, when image acquisition devices acquire images, non-keyframe images are prone to losing image features, resulting in the inability to obtain key information.
By performing multiple block processing on the image to be processed, using global and local motion blur removal, and combining a convolutional neural network model, image features are extracted and interacted to finally determine the repaired image.
It achieves efficient restoration of motion-blurred images, avoids loss of image features, and ensures the integrity of key information.
Smart Images
Figure CN116091344B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing, and more specifically, to an image restoration method, apparatus, non-volatile storage medium, and electronic device. Background Technology
[0002] Currently, image acquisition devices in related technologies, when continuously acquiring images, include both clear keyframe images and unclear non-keyframe images. Image features are easily lost in non-keyframe images, which negatively impacts the acquisition of crucial information from the image.
[0003] There is currently no effective solution to the above problems. Summary of the Invention
[0004] This application provides an image restoration method, apparatus, non-volatile storage medium, and electronic device to at least solve the technical problem of loss of key image information caused by the inability to extract image features from non-keyframe images in related technologies.
[0005] According to one aspect of the embodiments of this application, an image restoration method is provided, comprising: performing multiple block processing on an image to be processed to obtain multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale; performing global de-blurring on the image to be processed to obtain a first image feature; performing de-blurring on each block in each image block set, and determining a second image feature corresponding to each image block set after the de-blurring; determining a first target image based on the first image feature and the second image feature corresponding to each image block set, wherein the first target image is the restored image.
[0006] Optionally, the step of determining the first target image based on the first image features and the second image features corresponding to each image block set includes: performing feature interaction processing on the second image features corresponding to any two image block sets in the multiple image block sets to obtain a third image feature set, wherein the third image features in the third image feature set are image features obtained after performing feature interaction processing on two second image features; and determining the first target image based on the third image feature set and the first image features.
[0007] Optionally, the step of determining the first target image based on the third image feature set and the first image features includes: performing feature interaction processing on all third image features in the first image feature set and the third image feature set to obtain the second target image; and performing one-dimensional convolution processing on the second target image to obtain the first target image.
[0008] Optionally, the step of performing global motion demise on the image to be processed to obtain the first image feature includes: using a first neural network model to perform global motion demise on the image to be processed to obtain the first image feature, wherein the first neural network model includes a target convolutional neural network trained through a first training sample set, the first training sample set includes multiple sets of first type sample images, and second type sample images corresponding to each set of sample images in the multiple sets of first type sample images, wherein the sharpness of the second type sample images is greater than that of the first type sample images.
[0009] Optionally, the step of performing motion blur removal processing on each image block set includes: determining a second neural network model corresponding to the image block set; and performing motion blur removal processing on each image block in the image block set using the second neural network model.
[0010] Optionally, the second neural network model includes a target convolutional neural network, wherein the step of performing motion blur removal processing on each image block set includes: sequentially performing motion blur removal processing on each image block in the image block set through a target convolutional neural network.
[0011] Optionally, the second neural network model includes multiple target convolutional neural networks; the step of performing motion blur removal processing on each image block set using the second neural network model includes: determining the number of image blocks in the image block set; determining the second neural network model corresponding to the image block set based on the number of image blocks, wherein the second neural network model includes multiple target convolutional neural networks, and any one of the multiple target convolutional neural networks is matched with any image block in the image block set; and performing motion blur removal processing on each image block in the image block set using the multiple target convolutional neural networks contained in the determined second neural network model.
[0012] Optionally, the step of performing motion blur removal processing on each image block in the image block set using multiple target convolutional neural networks includes: determining the position information of each image block in the image to be processed; determining the target convolutional neural network corresponding to the image block based on the position information, wherein the target convolutional neural network is a convolutional neural network trained with a second training sample set, the second training sample set including multiple sets of first sample image blocks, and second sample image blocks corresponding to each set of first sample image blocks, the position information corresponding to the multiple sets of first sample image blocks is the same as the position information corresponding to each image block, and the sharpness of the second sample image block is greater than that of the first sample image block.
[0013] Optionally, after the step of performing motion blur removal processing on each image block in the image block set using the second neural network model, the image inpainting method further includes: determining a motion blur removal objective function corresponding to the motion blur removal image block set; and performing motion blur removal processing again on each image block after motion blur removal using the motion blur removal objective function.
[0014] Optionally, the step of determining the second image features corresponding to each set of image blocks includes: merging each image block after motion blur removal to obtain a third target image; and extracting the second image features from the third target image.
[0015] According to another aspect of the embodiments of this application, a bronchoscopy navigation method is also provided, comprising: acquiring a set of bronchoscopic images acquired by a bronchoscope; determining an image to be processed from the set of bronchoscopic images, wherein the image to be processed is a motion-blurred image; performing multiple block processing on the image to be processed to obtain multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale; performing global de-motion blurring processing on the image to be processed to obtain a first image feature; performing de-motion blurring processing on each image block set separately, and determining a second image feature corresponding to each image block set after the de-motion blurring processing; determining a first target image based on the first image feature and the second image feature corresponding to each image block set, wherein the first target image is a repaired image; replacing the image to be processed in the set of bronchoscopic images with the first target image, and registering the bronchoscopic images and a virtual bronchial tree model; and navigating the bronchoscope based on the registration result.
[0016] According to another aspect of the embodiments of this application, an image restoration apparatus is also provided, comprising: a segmentation module, configured to perform multiple block processing on an image to be processed to obtain multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale; a first processing module, configured to perform global de-blurring processing on the image to be processed to obtain a first image feature; a second processing module, configured to perform de-blurring processing on each block in each image block set, and determine a second image feature corresponding to each image block set after the de-blurring processing; and a restoration module, configured to determine a first target image based on the first image feature and the second image feature corresponding to each image block set, wherein the first target image is the restored image.
[0017] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is running, it controls the device where the non-volatile storage medium is located to execute an image restoration method or a bronchoscopy navigation method.
[0018] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, and the program executes an image restoration method or a bronchoscopy navigation method when it runs.
[0019] In this embodiment, the image to be processed is divided into multiple blocks to obtain multiple image block sets, wherein each image block set corresponds to a different block scale; global de-blurring is performed on the image to be processed to obtain a first image feature; de-blurring is performed on each block in each image block set, and a second image feature corresponding to each image block set is determined after the de-blurring; based on the first image feature and the second image feature corresponding to each image block set, a first target image is determined, wherein the first target image is the repaired image. By performing de-blurring on both the overall image to be processed and the image blocks separately, the goal of not losing image features is achieved, thereby realizing the technical effect of efficiently repairing motion-blurred images and avoiding the loss of key image information, thus solving the technical problem of loss of key image information caused by the inability to extract image features from non-key frame images in related technologies. Attached Figure Description
[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0021] Figure 1 This is a schematic diagram of the structure of an optional computer terminal according to an embodiment of this application;
[0022] Figure 2 This is a schematic flowchart of an optional image restoration method according to an embodiment of this application;
[0023] Figure 3 This is a schematic flowchart of an optional image restoration process according to an embodiment of this application;
[0024] Figure 4 This is a schematic flowchart of an optional bronchoscopic navigation method according to an embodiment of this application;
[0025] Figure 5 This is a schematic diagram of an optional image restoration device according to an embodiment of this application. Detailed Implementation
[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.
[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0028] In related technologies, image acquisition devices capture images including keyframe images and non-keyframe images. Non-keyframe images, due to their lack of clarity, may lose image features, making it impossible to determine key information within the image. For example, in bronchoscopy, the limited field of view and the typically rapid operation of doctors often result in non-keyframe images in the acquired bronchoscopic images. However, related technologies lack methods for repairing non-keyframe images in bronchoscopy, thus hindering the accurate acquisition of information about the bronchial interior. To address this issue, this application provides a solution, detailed below.
[0029] According to an embodiment of this application, an embodiment of an image restoration method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0030] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an image restoration method is shown. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0031] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).
[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image restoration method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the image restoration method of the aforementioned application. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0033] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0034] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).
[0035] Under the above operating environment, this application provides an image restoration method, such as... Figure 2 As shown, the method includes the following steps:
[0036] Step S202: Perform multiple block processing on the image to be processed to obtain multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale;
[0037] In the technical solution provided in step S202, because the segmentation scale of the image to be processed is different, the final image features obtained are also different. Specifically, the more segments there are, the more detailed features are extracted, but the fewer shape features are extracted; conversely, the fewer segments there are, the more shape features are extracted, but the fewer detailed features are extracted. Therefore, in order to improve the image restoration effect, different segmentation systems can be used to process the image to be processed. For example, the image to be processed can be divided into three image segment sets: 2×2, 3×3, and 4×4.
[0038] The images to be processed can be images captured in various tunnels. For example, images of the human neck and corresponding parts of the human body, such as the trachea or bronchi and heart vessels in the chest cavity, captured by acquisition devices such as endoscopes. These images can be obtained from a server or directly from the acquisition device.
[0039] Step S204: Perform global motion demise blurring on the image to be processed to obtain the first image features;
[0040] In the technical solution provided in step S204, since dividing the image to be processed into image blocks and then deblurring each image block separately may result in the loss of some shape feature information of the image, the deblurring process can be performed on the image to be processed as a whole from a global perspective. Specifically, the step of performing global deblurring on the image to be processed to obtain the first image feature includes: using a first neural network model to perform global deblurring on the image to be processed to obtain the first image feature. The first neural network model includes a target convolutional neural network trained on a first training sample set. The first training sample set includes multiple sets of first-type sample images and second-type sample images corresponding to each set of first-type sample images, wherein the clarity of the second-type sample images is greater than that of the first-type sample images.
[0041] Specifically, when performing motion blur removal on the image to be processed, the entire image can be directly input into the convolutional neural network F1 (including the encoder and decoder), and the output image of the convolutional neural network is obtained: Y1 = F1(X1) + X1, where Y1 is the globally considered motion blur removal result, and X1 is the image to be processed. The convolutional neural network F1 can adopt the u-net network architecture. The training sample set used to train the convolutional neural network F1 contains several sets of blurred sample images and corresponding sharp sample images.
[0042] Step S206: Perform motion blur removal processing on each block in each image block set, and determine the second image features corresponding to each image block set after motion blur removal processing;
[0043] In the technical solution provided in step S206, the step of performing motion blur removal processing on each image block set includes: determining a second neural network model corresponding to the image block set; and performing motion blur removal processing on each image block in the image block set through the second neural network model.
[0044] Specifically, the second neural network model described above may include one or more target convolutional neural networks. When the second neural network model includes one target convolutional neural network, the step of performing motion blur removal processing on each image block set includes: sequentially performing motion blur removal processing on each image block in the image block set using a target convolutional neural network.
[0045] In a second neural network model containing multiple target convolutional neural networks, where each target convolutional neural network corresponds to an image block in an image block set, the step of performing motion blur removal processing on each image block set includes: determining the number of image blocks in the image block set; determining a second neural network model corresponding to the image block set based on the number of image blocks, wherein the second neural network model includes multiple target convolutional neural networks, and any target convolutional neural network in the multiple target convolutional neural networks matches any image block in the image block set; and performing motion blur removal processing on each image block in the image block set using the multiple target convolutional neural networks contained in the determined second neural network model.
[0046] As an optional implementation, to improve processing speed, the second neural network model can be configured with multiple parallel target convolutional neural networks to process image blocks simultaneously. Specifically, the step of performing motion blur removal processing on each image block in the image block set using multiple target convolutional neural networks includes: determining the position information of each image block in the image to be processed; determining the target convolutional neural network corresponding to the image block based on the position information, wherein the target convolutional neural network is a convolutional neural network trained using a second training sample set, the second training sample set including multiple sets of first sample image blocks, and second sample image blocks corresponding to each set of first sample image blocks, the position information corresponding to the multiple sets of first sample image blocks is the same as the position information corresponding to each image block, and the sharpness of the second sample image block is greater than that of the first sample image block.
[0047] It should be noted that the target convolutional neural network in the second neural network model and the first neural network model can use the same model architecture, only the training dataset is different.
[0048] Specifically, suppose the image block set includes X2 (corresponding to a block size of 2×2), X3 (corresponding to a block size of 3×3), and X4 (corresponding to a block size of 4×4). Taking X2 as an example, it contains block 1, block 2, block 3, and block 4. Processing X2 requires four convolutional networks (f2, f3, f4, and f5) to process each block separately. The resulting image block set can be defined as M2, which contains the image blocks after deblurring each block. Each convolutional network uses a training sample set containing both blurred and sharp image blocks at the corresponding location.
[0049] Optionally, the architecture of the convolutional neural network described above can be a fully convolutional neural network or a u-net architecture.
[0050] In some embodiments of this application, after performing motion blur removal processing on each image block in the image block set using a second neural network model, a motion blur removal objective function can be constructed, and the image block set can be further de-blurred using the motion blur removal objective function. Specifically, firstly, a motion blur removal objective function corresponding to the de-blurred image block set is determined; then, each image block after motion blur removal is de-blurred again using the motion blur removal objective function.
[0051] Specifically, after constructing the motion blur removal objective function, the Alternating Direction Method of Multipliers (ADMM) can be used to solve the objective function, thereby achieving further motion blur removal processing on the image blocks. The formula for the constructed motion blur removal objective function is as follows:
[0052]
[0053] In the above formula These are Lagrange parameters. The penalty parameter is decomposed into three subproblems through duality. M refers to M2, M3, and M4, and m is the corresponding block. For example, M2 has four blocks m. M refers to the blurred image before the ADMM algorithm, and Y represents the result of removing motion blur after optimization by the ADMM algorithm. For learnable fuzzy operators, such as 2 = X2 / M2, where D is the matrix learned during training, and Z = DM, indicating that the neighborhood transformation process is applied to DM.
[0054] Solving the above formula is an augmented Lagrange problem, which can be divided into the following three sub-problems:
[0055]
[0056] The results obtained by solving the above three subproblems using the alternating direction multiplier method are as follows:
[0057]
[0058] In the technical solution provided in step S206, the step of determining the second image feature corresponding to each image block set includes: merging each image block after motion blur removal to obtain a third target image; and extracting the second image feature from the third target image.
[0059] Step S208: Determine the first target image based on the first image features and the second image features corresponding to each image block set, wherein the first target image is the repaired image.
[0060] In the technical solution provided in step S208, the step of determining the first target image based on the first image features and the second image features corresponding to each image block set includes: performing feature interaction processing on the second image features corresponding to any two image block sets in the multiple image block sets to obtain a third image feature set, wherein the third image features in the third image feature set are image features obtained after performing feature interaction processing on multiple second image features; and determining the first target image based on the third image feature set and the first image features.
[0061] Before performing interactive processing on the second image features corresponding to any two image block sets from multiple image block sets, it is first necessary to concatenate the image blocks in each image block set and extract features. Specifically, the blocks in Y2, Y3, Y4 (where Y2, Y3, and Y4 are image block sets optimized by de-motion blurring and ADMM algorithm, respectively) are merged and input into the corresponding neural network model. Furthermore, the neural network models will interact with each other's features. For example, if their features are J1, J2, and J3, the feature interaction is J1 = J1 + J2, or J1 = concat(J1, J2), J1 = J1 J2. The final output results are P1, P2, and P3.
[0062] For example, the individual blocks in Y2 and their corresponding position information can be input into the corresponding third neural network model (i.e., Figure 3 In the Aggregation CNN101, Aggregation CNN102, and Aggregation CNN103 shown, the third neural network model stitches together the blocks according to the position information to obtain a complete de-motion blurred image, and the second neural network model extracts the features of the complete de-motion blurred image to obtain the extraction result J1; similarly, each block in Y3 and the corresponding position information of each block are input into the corresponding second neural network model, and after processing by the second neural network model, the extraction result J2 is obtained; each block in Y4 and the corresponding position information of each block are input into the corresponding second neural network model, and after processing by the second neural network model, the extraction result J3 is obtained.
[0063] Specifically, Figure 3The convolutional neural networks Aggregation CNN101, Aggregation CNN102, and Aggregation CNN103 shown can be used to concatenate image features obtained from image blocks according to the position information of the image blocks, thereby obtaining the second image features corresponding to each set of image blocks.
[0064] As an alternative implementation, the above-mentioned convolutional neural networks Aggregation CNN101, AggregationCNN102 and Aggregation CNN103 can also first stitch together the motion-blurred image blocks in the image block set according to the position information, and then extract image features from the stitched image.
[0065] In some embodiments of this application, the step of determining the first target image based on the third image feature set and the first image features includes: performing feature interaction processing on the first image features and all the third image features in the third image feature set to obtain the second target image; and performing one-dimensional convolution processing on the second target image to obtain the first target image.
[0066] Specifically, the second target image O = Y1 + P1 P2+P2 P3, where Y1 represents the first image feature. By performing a one-dimensional convolution on O, the obtained feature information can be integrated to obtain the first target image.
[0067] It is understood that the method provided in this application embodiment does not limit the number of image blocks in the image block set, and users can divide the image into blocks according to their own needs. Assuming that the number of image blocks in the image block set is N (N is a positive integer), the final second target image can be represented as O = Y1 + P1 P2+P2 P3+……+P N-1 P N .
[0068] Additionally, it should be noted that in this application, it is also possible to first perform motion blur processing on the image blocks by constructing a motion blur objective function, and then perform motion blur processing on the image blocks again by using a convolutional neural network.
[0069] like Figure 3As shown, in this application, when performing motion blur removal on the image to be processed, the image is divided according to different scales to obtain multiple image block sets. Motion blur removal is then performed simultaneously on both the image to be processed and the multiple image block sets. Specifically, when performing motion blur removal on the image block sets, two methods are employed: convolutional neural networks and the construction of a motion blur removal objective function. Afterwards, image features from the image block sets are extracted for preliminary feature interaction. The features obtained from this preliminary feature interaction are then further interacted with the globally de-blurred image to be processed, ultimately resulting in the de-blurred image.
[0070] Furthermore, by performing multiple block processing on the image to be processed, multiple image block sets are obtained, each with a different block scale. Global de-blurring is then performed on the image to be processed to obtain the first image feature. De-blurring is then performed on each block within each image block set, and a second image feature corresponding to each image block set is determined after the de-blurring. Based on the first image feature and the second image feature corresponding to each image block set, a first target image is determined, where the first target image is the restored image. By performing de-blurring on both the overall image to be processed and the image blocks separately, the goal of not losing image features is achieved, thus realizing the technical effect of efficiently restoring motion-blurred images. This solves the technical problem of losing key image information due to the inability to extract image features from non-keyframe images in related technologies.
[0071] This application provides a bronchoscopy navigation method. Figure 4 This is a flowchart illustrating the bronchoscopic navigation method, as shown below. Figure 4 As shown, the method includes:
[0072] Step S402: Obtain the set of bronchoscopic images acquired by the bronchoscope;
[0073] Step S404: Determine the image to be processed from the set of bronchoscopic images, wherein the image to be processed is a motion-blurred image;
[0074] Step S406: Perform multiple block processing on the image to be processed to obtain multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale;
[0075] Step S408: Perform global motion demise blurring on the image to be processed to obtain the first image features;
[0076] Step S410: Perform motion blur removal processing on each image block set, and determine the second image features corresponding to each image block set after motion blur removal processing;
[0077] Step S412: Determine the first target image based on the first image features and the second image features corresponding to each image block set, wherein the first target image is the repaired image;
[0078] Step S414: Replace the images to be processed in the bronchoscopic image set with the target images, and register the bronchoscopic images and the virtual bronchial tree model.
[0079] Step S416: Navigate the bronchoscope based on the registration results.
[0080] This application provides an image restoration device. Figure 5 This is a schematic diagram of the image restoration device, as shown below. Figure 5 As shown, the device includes: a segmentation module 50, used to perform multiple block processing on the image to be processed to obtain multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale; a first processing module 52, used to perform global de-blurring processing on the image to be processed to obtain a first image feature; a second processing module 54, used to perform de-blurring processing on each block in each image block set, and after the de-blurring processing, determine the second image feature corresponding to each image block set; and a repair module 56, used to determine a first target image based on the first image feature and the second image feature corresponding to each image block set, wherein the first target image is the repaired image.
[0081] In some embodiments of this application, the first processing module 52 performs global motion-de-blurring on the image to be processed to obtain first image features. The first processing module 52 performs global motion-de-blurring on the image to be processed using a first neural network model to obtain first image features. The first neural network model includes a target convolutional neural network trained on a first training sample set. The first training sample set includes multiple sets of first-type sample images and second-type sample images corresponding to each set of sample images in the multiple sets of first-type sample images. The second-type sample images have higher clarity than the first-type sample images.
[0082] In some embodiments of this application, the step of the second processing module 54 performing motion blur removal processing on each image block set includes: determining a second neural network model corresponding to the image block set; and performing motion blur removal processing on each image block in the image block set using the second neural network model.
[0083] In some embodiments of this application, the second neural network model includes a target convolutional neural network, wherein the step of the second processing module 54 performing motion blur removal processing on each image block set includes: sequentially performing motion blur removal processing on each image block in the image block set through a target convolutional neural network.
[0084] In some embodiments of this application, the second neural network model includes multiple target convolutional neural networks; the second processing module 54 performs motion blur removal processing on each image block set using the second neural network model, including: determining the number of image blocks in the image block set; determining the second neural network model corresponding to the image block set based on the number of image blocks, wherein the second neural network model includes multiple target convolutional neural networks, and any one of the multiple target convolutional neural networks matches any image block in the image block set; and performing motion blur removal processing on each image block in the image block set using the multiple target convolutional neural networks included in the determined second neural network model.
[0085] In some embodiments of this application, the second processing module 54 performs motion blur removal processing on each image block in the image block set using multiple target convolutional neural networks. This includes: determining the position information of each image block in the image to be processed; determining the target convolutional neural network corresponding to the image block based on the position information, wherein the target convolutional neural network is a convolutional neural network trained using a second training sample set. The second training sample set includes multiple sets of first sample image blocks and second sample image blocks corresponding to each set of first sample image blocks. The position information corresponding to the multiple sets of first sample image blocks is the same as the position information corresponding to each image block, and the clarity of the second sample image block is greater than that of the first sample image block.
[0086] In some embodiments of this application, after the second processing module 54 performs motion blur removal processing on each image block in the image block set using the second neural network model, the image inpainting method further includes: determining a motion blur removal objective function corresponding to the motion blur removal image block set; and performing motion blur removal processing again on each image block after motion blur removal using the motion blur removal objective function.
[0087] In some embodiments of this application, the step of the second processing module 54 in determining the second image features corresponding to each image block set includes: merging each image block after motion blur removal to obtain a third target image; and extracting the second image features from the third target image.
[0088] In some embodiments of this application, the step of the repair module 56 in determining the first target image based on the first image features and the second image features corresponding to each image block set includes: performing feature interaction processing on the second image features corresponding to any two image block sets in the plurality of image block sets to obtain a third image feature set, wherein the third image features in the third image feature set are image features obtained after performing feature interaction processing on two second image features; and determining the first target image based on the third image feature set and the first image features.
[0089] In some embodiments of this application, the step of the repair module 56 in determining the first target image based on the third image feature set and the first image features includes: performing feature interaction processing on the first image features and all the third image features in the third image feature set to obtain the second target image; and performing one-dimensional convolution processing on the second target image to obtain the first target image.
[0090] It should be noted that each module in the above-mentioned image restoration device can be a program module (for example, a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.
[0091] This application provides a non-volatile storage medium storing a program. During program execution, the device containing the non-volatile storage medium performs the following image restoration method: The image to be processed is divided into multiple blocks to obtain multiple image block sets, each image block set having a different block scale; global de-blurring is performed on the image to be processed to obtain a first image feature; de-blurring is performed on each block in each image block set, and after de-blurring, a second image feature corresponding to each image block set is determined; based on the first image feature and the second image feature corresponding to each image block set, a first target image is determined, wherein the first target image is the restored image.
[0092] In some embodiments of this application, the above program can also control the device containing the non-volatile storage medium to perform the following bronchoscopy navigation method during runtime: acquiring a set of bronchoscopic images collected by the bronchoscope; determining an image to be processed from the set of bronchoscopic images, wherein the image to be processed is a motion-blurred image; performing multiple block processing on the image to be processed to obtain multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale; performing global de-motion blurring on the image to be processed to obtain a first image feature; performing de-motion blurring on each image block set separately, and determining a second image feature corresponding to each image block set after de-motion blurring; determining a first target image based on the first image feature and the second image feature corresponding to each image block set, wherein the first target image is a repaired image; replacing the image to be processed in the set of bronchoscopic images with the first target image, and registering the bronchoscopic images and the virtual bronchial tree model; navigating the bronchoscope based on the registration result.
[0093] This application provides an electronic device, including a memory and a processor. The processor runs a program stored in the memory. When the program runs, it executes the following image restoration method: performing multiple block processing on the image to be processed to obtain multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale; performing global de-blurring on the image to be processed to obtain a first image feature; performing de-blurring on each block in each image block set, and determining a second image feature corresponding to each image block set after the de-blurring; determining a first target image based on the first image feature and the second image feature corresponding to each image block set, wherein the first target image is the restored image.
[0094] In some embodiments of this application, the program may also execute the following bronchoscopy navigation method during runtime: acquiring a set of bronchoscopic images collected by a bronchoscope; determining an image to be processed from the set of bronchoscopic images, wherein the image to be processed is a motion-blurred image; performing multiple block processing on the image to be processed to obtain multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale; performing global de-motion blurring on the image to be processed to obtain a first image feature; performing de-motion blurring on each image block set separately, and determining a second image feature corresponding to each image block set after de-motion blurring; determining a first target image based on the first image feature and the second image feature corresponding to each image block set, wherein the first target image is a repaired image; replacing the image to be processed in the set of bronchoscopic images with the first target image, and registering the bronchoscopic images and the virtual bronchial tree model; navigating the bronchoscope based on the registration result.
[0095] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0096] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be aggregated or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.
[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0098] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0099] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0100] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. An image restoration method, characterized in that, include: The image to be processed is divided into multiple blocks to obtain multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale; The image to be processed is subjected to global motion demise blurring to obtain the first image features; Each image block set is subjected to motion blur removal processing, and after motion blur removal processing, the second image feature corresponding to each image block set is determined. Determining the second image feature corresponding to each image block set includes: merging the motion blur removed blocks in the image block set to obtain a third target image; and extracting the second image feature from the third target image. Based on the first image features and the second image features corresponding to each image block set, a first target image is determined, wherein the first target image is the repaired image; The step of determining the first target image based on the first image features and the second image features corresponding to each image block set includes: The second image features corresponding to any two image block sets in the plurality of image block sets are subjected to feature interaction processing to obtain a third image feature set. The third image features in the third image feature set are image features obtained after feature interaction processing of multiple second image features. The feature interaction processing includes feature superposition, feature concatenation or feature multiplication. The first target image is determined based on the third image feature set and the first image features.
2. The image restoration method according to claim 1, characterized in that, The step of determining the first target image based on the third image feature set and the first image features includes: The second target image is obtained by performing feature interaction processing on all third image features in the first image feature set and the third image feature set. The second target image is subjected to one-dimensional convolution processing to obtain the first target image.
3. The image restoration method according to claim 1, characterized in that, The step of performing global motion-deblurring on the image to be processed to obtain the first image features includes: A first neural network model is used to perform global motion-de-blurring on the image to be processed to obtain the first image features. The first neural network model includes a target convolutional neural network trained with a first training sample set. The first training sample set includes multiple sets of first-type sample images and second-type sample images corresponding to each set of sample images in the multiple sets of first-type sample images. The second-type sample images have higher sharpness than the first-type sample images.
4. The image restoration method according to claim 1, characterized in that, The step of performing motion blur removal processing on each of the image block sets includes: Determine the second neural network model corresponding to the image block set; The second neural network model is used to perform motion blur removal on each image block in the image block set.
5. The image restoration method according to claim 4, characterized in that, The second neural network model includes a target convolutional neural network, wherein the step of performing motion blur removal processing on each set of image blocks includes: The target convolutional neural network is used to sequentially perform motion blur removal processing on each image block in the image block set.
6. The image restoration method according to claim 4, characterized in that, The second neural network model contains multiple target convolutional neural networks; The steps of performing motion blur removal processing on each image block set using the second neural network model include: Determine the number of image blocks in the image block set; A second neural network model corresponding to the set of image blocks is determined based on the number of image blocks, wherein the second neural network model includes multiple target convolutional neural networks, and any one of the multiple target convolutional neural networks is matched with any image block in the set of image blocks; The determined second neural network model contains multiple target convolutional neural networks, which are used to perform motion blur removal processing on each image block in the image block set.
7. The image restoration method according to claim 6, characterized in that, The step of performing motion blur removal processing on each image block in the image block set using multiple target convolutional neural networks contained in the determined second neural network model includes: Determine the position information of each image block in the image to be processed; The target convolutional neural network corresponding to the image block is determined based on the location information. The target convolutional neural network is a convolutional neural network trained with a second training sample set. The second training sample set includes multiple sets of first sample image blocks and a second sample image block corresponding to each set of first sample image blocks. The location information corresponding to the multiple sets of first sample image blocks is the same as the location information corresponding to each image block. The clarity of the second sample image block is greater than that of the first sample image block.
8. The image restoration method according to claim 4, characterized in that, After the step of performing motion blur removal processing on each image block in the image block set using the second neural network model, the image restoration method further includes: Determine the motion blur objective function corresponding to the set of image blocks after motion blur removal; The motion blur objective function is used to perform motion blur removal processing again on each image block after the motion blur removal process.
9. A bronchoscopic navigation method, characterized in that, include: Obtain a set of bronchoscopic images acquired by a bronchoscope; The image to be processed is determined from the set of bronchoscopic images, wherein the image to be processed is a motion-blurred image; The image to be processed is divided into multiple blocks to obtain multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale; The image to be processed is subjected to global motion demise blurring to obtain the first image features; Each image block set is subjected to motion blur removal processing, and after motion blur removal processing, the second image feature corresponding to each image block set is determined. Determining the second image feature corresponding to each image block set includes: merging the motion blur removed blocks in the image block set to obtain a third target image; and extracting the second image feature from the third target image. Based on the first image features and the second image features corresponding to each image block set, a first target image is determined, wherein the first target image is the repaired image; The image to be processed in the bronchoscopic image set is replaced with the first target image, and the bronchoscopic image and the virtual bronchial tree model are registered. The bronchoscope is navigated based on the registration results; The step of determining the first target image based on the first image feature and the second image feature corresponding to each image block set includes: performing feature interaction processing on the second image features corresponding to any two image block sets in the plurality of image block sets to obtain a third image feature set, wherein the third image feature in the third image feature set is an image feature obtained after performing feature interaction processing on a plurality of second image features, and the feature interaction processing includes feature superposition, feature concatenation or feature multiplication; The first target image is determined based on the third image feature set and the first image features.
10. An image restoration device, characterized in that, include: The partitioning module is used to perform multiple block-based processing on the image to be processed, resulting in multiple image block sets, wherein each image block set in the multiple image block sets corresponds to a different block scale; The first processing module is used to perform global motion blur removal processing on the image to be processed to obtain the first image features; The second processing module is used to perform motion blur removal processing on the blocks in each image block set, and after the motion blur removal processing, determine the second image feature corresponding to each image block set. Determining the second image feature corresponding to each image block set includes: merging the motion blur removed blocks in the image block set to obtain a third target image; and extracting the second image feature from the third target image. The repair module is used to determine a first target image based on the first image features and the second image features corresponding to each image block set, wherein the first target image is the repaired image; The repair module is further configured to perform feature interaction processing on the second image features corresponding to any two image block sets in the plurality of image block sets to obtain a third image feature set, wherein the third image features in the third image feature set are image features obtained after feature interaction processing on a plurality of second image features, and the feature interaction processing includes feature superposition, feature concatenation or feature multiplication; and determine the first target image based on the third image feature set and the first image features.
11. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device containing the non-volatile storage medium to perform the image restoration method of any one of claims 1 to 8, or the bronchoscopy navigation method of claim 9.
12. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the image restoration method of any one of claims 1 to 8, or the bronchoscopic navigation method of claim 9.