Image deblurring method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an image deblurring method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring a to-be-processed image; the jitter amount of the electronic equipment is obtained, and the jitter amount is used for representing the jitter degree of the electronic equipment in the time period when a camera of the electronic equipment collects the to-be-processed image; according to the jitter amount, a target deblurring strategy is determined, and the target deblurring strategy is matched with the blurring degree of the to-be-processed image; and performing deblurring processing on the to-be-processed image according to the target deblurring strategy to obtain a clear image. According to the image deblurring method provided by the invention, the image deblurring effect can be improved, so that the visual experience of a user is better met.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and more particularly, to an image deblurring method, apparatus, electronic device, and readable storage medium. Background Art

[0002] During the process of using the camera of an electronic device (such as a mobile phone) to capture an object to be captured, if the electronic device moves relative to the object to be captured (for example, the electronic device shakes or the object to be captured moves, etc.), the image captured by the camera is a blurred image. Currently, for a blurred image, the blurred image can be restored to a clear image (i.e., an unblurred image) through deblurring technology. In related technologies, the electronic device inputs the obtained blurred image into a deblurring network for deblurring processing, which easily causes the deblurring network to process the originally clear image content in the blurred image into blurred image content, or easily causes the deblurring network not to restore all the blurred image content in the blurred image to clear content, that is, the traditional technology has the problem of poor deblurring effect, resulting in a poor visual experience for users.

[0003] Therefore, how to improve the effect of image deblurring has become an urgent problem to be solved currently. Summary of the Invention

[0004] This application provides an image deblurring method, apparatus, electronic device, and readable storage medium, which can improve the image deblurring effect, thereby better meeting the visual experience of users.

[0005] In a first aspect, this application provides an image deblurring method, which is applied to an electronic device. The method includes: obtaining an image to be processed; obtaining the amount of jitter of the electronic device, where the amount of jitter is used to represent the degree of jitter of the electronic device during the period when the camera of the electronic device captures the image to be processed; determining a target deblurring strategy according to the amount of jitter, where the target deblurring strategy matches the degree of blurring of the image to be processed; and performing deblurring processing on the image to be processed according to the target deblurring strategy to obtain a clear image.

[0006] The image to be processed is an image captured by the camera of the electronic device for an object to be captured. For example, the image to be processed can be, but is not limited to, a locally blurred image or a globally blurred image. It can be understood that when the blur in an image is local blur, the image can be called a locally blurred image, that is, the pixel points in the local area of the locally blurred image are blurred. When the blur in an image is global blur, the image can be called a globally blurred image, that is, all the pixel points in the entire blurred image are blurred.

[0007] The matching of the target de-blurring strategy with the blurring degree of the image to be processed means that the image obtained after the target de-blurring strategy performs de-blurring on the image to be processed is a clear image, and this clear image can meet the user's de-blurring requirements.

[0008] For example, when the image to be processed is a globally blurred image, the target de-blurring strategy is a global de-blurring strategy. For example, when the image to be processed is a locally blurred image, the target de-blurring strategy is a local de-blurring strategy.

[0009] In the above technical solution, after the electronic device obtains the image to be processed, the electronic device first determines the strategy for performing de-blurring on the image to be processed (i.e., the target de-blurring strategy) according to the amount of jitter. Since the amount of jitter represents the degree of jitter of the electronic device during the period when the camera of the electronic device captures the image to be processed, therefore, the target de-blurring strategy determined based on the amount of jitter matches the blurring degree of the image to be processed. Then, the electronic device performs de-blurring processing on the image to be processed according to the target de-blurring strategy determined based on the amount of jitter, and a clear image can be obtained. This method avoids the problem in the traditional technology that the de-blurring effect is poor due to the mismatch between the de-blurring processing strategy and the blurring degree of the image to be processed when the electronic device performs de-blurring processing on the obtained image to be processed. Therefore, this method can improve the image de-blurring effect and thus better meet the user's visual experience.

[0010] In a possible implementation manner, the target de-blurring strategy includes a local de-blurring strategy or a global de-blurring strategy, and determining the target de-blurring strategy according to the amount of jitter includes: when the amount of jitter satisfies the first jitter degree, determining the target de-blurring strategy as a local de-blurring strategy; when the amount of jitter satisfies the second jitter degree, determining the target de-blurring strategy as a global de-blurring strategy, where the second jitter degree is greater than the first jitter degree.

[0011] The amount of jitter satisfying the first jitter degree means that the degree of jitter of the electronic device during the period when the camera captures the image to be processed is small. In this case, the image captured by the camera is a locally blurred image. The amount of jitter satisfying the second jitter degree means that the degree of jitter of the electronic device during the period when the camera captures the image to be processed is large. In this case, the image captured by the camera is a globally blurred image.

[0012] There is no specific limitation on the presentation form of the first jitter degree and the second jitter degree. For example, at least one of the first jitter degree and the second jitter degree can be a specific jitter degree. For example, at least one of the first jitter degree and the second jitter degree can be a jitter degree within a preset range, and no specific limitation is made on this.

[0013] In the above technical solution, when the electronic device determines that the jitter degree of the electronic device is relatively large during the period when the camera of the electronic device captures the image to be processed, it is determined that the defocusing strategy for the image to be processed is the global defocusing strategy. When the electronic device determines that the jitter degree of the electronic device is relatively small during the period when the camera of the electronic device captures the image to be processed, it is determined that the defocusing strategy for the image to be processed is the local defocusing strategy. In summary, this method can achieve the purpose of performing defocusing processing on local blurred images and global blurred images, thereby improving the defocusing effect of global blurred images and local blurred images, and better meeting the visual experience of users.

[0014] In another possible implementation manner, performing defocusing processing on the image to be processed according to the target defocusing strategy to obtain a clear image includes: using the target defocusing network to perform defocusing processing on the image to be processed to obtain a clear image, where the target defocusing network is a neural network model obtained by training an initial defocusing network using training samples, and the training samples include training images, label images, and motion mask images. The training images are blurred images, the label images are clear images corresponding to the training images, and the motion mask images are images including the characteristics of the blurred areas of the training images.

[0015] The structure of the initial defocusing network may be a decoder-decoder structure. For example, the initial defocusing network may be, but is not limited to, a Unet network.

[0016] The number of training samples may be one or more, and no specific limitation is made thereto. In the case where the number of training samples is multiple, no specific limitation is made on whether any two of the multiple training samples are the same.

[0017] In one example, when the training image is a local blurred image, that is, the local area in the training image is the blurred content, and the area other than the local area in the training image is the clear content. In this case, the motion mask image includes the characteristics of the local area in the training image. For example, the area with a pixel value of "1" in the motion mask image corresponds to the characteristics of the local area (i.e., the blurred area) in the training image, and the area with a pixel value of "0" in the motion mask image corresponds to the area other than the local area (i.e., the clear area) in the training image.

[0018] In one example, when the training image is a global blurred image, that is, all areas in the training image are blurred content. In this case, the motion mask image includes the characteristics of all areas in the training image. For example, the pixel values of all areas of the motion mask image are "1".

[0019] In the above technical solution, the training samples for the electronic device to train the initial deblurring network include not only training images and the label images corresponding to the training images, but also the motion mask images corresponding to the training images. Different from the traditional training method of only inputting the training images and the label images into the initial network model, when the target deblurring network used in this image deblurring method performs training on the initial deblurring network based on the training samples (i.e., training images, label images, and motion mask images), the deblurring accuracy of the trained target deblurring network can be improved. Thereafter, based on the trained target deblurring network, performing deblurring processing on the image to be processed can improve the image deblurring effect, thereby better meeting the user's visual experience.

[0020] In another possible implementation, the initial deblurring network includes an encoder and a decoder. Among them, the decoder includes a first network module and a second network module, and the target deblurring network is specifically a neural network model obtained by training the encoder, the adjusted first network module, and the second network module according to the first loss value. The first loss value is determined based on the difference between the predicted image and the training image. The predicted image is the image obtained by the second network module performing decoding processing on the second feature image, and the second feature image is the image obtained by the adjusted first network module performing feature extraction on the first feature image. The adjusted first network module adjusts the parameters of the first network module according to the second loss value. The first feature image is the image obtained by the encoder performing feature extraction on the training image, and the second loss value is determined based on the difference between the first feature image and the motion mask image.

[0021] In the above technical solution, after the first network module in the decoder obtains the first feature image obtained by the encoder performing feature extraction processing on the training image, it first adjusts the weights (i.e., parameters) of the first network module according to the difference between the first feature image and the pre-acquired motion mask image corresponding to the training image. Then, the second network module in the decoder obtains the predicted image based on the optimized first feature image (i.e., the second feature image) output by the adjusted first network module. Finally, according to the difference between the predicted image and the training image, the weights of the encoder, the adjusted first network module, and the second network module are adjusted to obtain the trained network (i.e., the target deblurring network). During this training process, since the motion mask image corresponding to the pre-acquired training image is used to optimize the feature image (i.e., the first feature image) extracted by the encoder, the optimized first feature image (i.e., the second feature image) can meet the deblurring requirements. Therefore, this method can improve the image deblurring effect, thereby better meeting the user's visual experience.

[0022] In another possible implementation, when the training image is a globally blurred image, the entire region of the training image is blurred content, the motion mask image includes the features of the entire region, and the target deblurring network is a global deblurring network; when the training image is a locally blurred image, a partial region of the training image is blurred content, the motion mask image includes the features of the partial region, and the target deblurring network is a local deblurring network.

[0023] In the above technical solution, since the blur types of the training samples (for example, global blur type or local blur type) are different, the corresponding motion mask images are also different. In this way, when training the initial deblurring network based on the corresponding training samples, the network can be guided to learn the blur processing ability of the corresponding type, so that the trained global deblurring network has strong global deblurring ability, and the trained local deblurring network has strong local deblurring ability. Then, based on the trained local deblurring network, the purpose of performing deblurring processing on the locally blurred image can be achieved, and based on the trained global deblurring network, the purpose of performing deblurring processing on the globally blurred image can be achieved, thereby improving the deblurring effect of the globally blurred image and the deblurring effect of the locally blurred image, and better meeting the user's visual experience.

[0024] In another possible implementation, obtaining the jitter amount of the electronic device includes: obtaining angular acceleration data, where the angular acceleration data is used to represent the pose data of the electronic device during the period when the camera captures the image to be processed; determining the jitter amount based on the angular acceleration data.

[0025] For example, a gyroscope sensor located in the electronic device can detect and record the pose data of the electronic device at different times. In this case, the above angular acceleration data can be obtained by the electronic device from the gyroscope sensor located in the electronic device.

[0026] In the above technical solution, the electronic device determines the jitter amount based on the pose data (i.e., angular acceleration data) of the electronic device during the period when the camera captures the image to be processed. This implementation process is relatively simple and can improve the efficiency of image deblurring.

[0027] In another possible implementation, the angular acceleration data includes multiple angular accelerations, where the multiple angular accelerations correspond to multiple target points in the image to be processed. Each angular acceleration is the pose data of the electronic device when the corresponding target point is captured by the camera. The multiple target points correspond to multiple first two-dimensional positions, and the position of each target point in the image to be processed is the corresponding first two-dimensional position. Moreover, based on the angular acceleration data, determining the jitter amount includes: obtaining multiple second two-dimensional positions based on the multiple angular accelerations and the multiple first two-dimensional positions; obtaining multiple blur values corresponding to the multiple target points according to the multiple second two-dimensional positions and the multiple first two-dimensional positions; and determining the jitter amount according to the multiple blur values.

[0028] In one example, the multiple target points correspond one-to-one with multiple image blocks included in the image to be processed, and each target point is a point in the corresponding image block. For example, each target point is the center point of the corresponding image block. For example, each target point is the point at the lower left corner of the corresponding image block.

[0029] In the above technical solution, the electronic device measures the blur degree of each image block through the pixel points in each image block, and then determines the jitter amount of the electronic device based on the multiple blur values corresponding to the multiple image blocks. This method can improve the accuracy of the obtained jitter amount of the electronic device. After that, the target deblurring strategy determined based on this jitter amount matches the image to be processed better, so as to improve the deblurring effect of the globally blurred image and the locally blurred image, and better meet the user's visual experience.

[0030] In another possible implementation, determining the jitter amount according to the multiple blur values includes: when the smallest blur value among the multiple blur values is less than a preset threshold, determining that the jitter amount satisfies a first jitter degree; when the smallest blur value among the multiple blur values is greater than or equal to the preset threshold, determining that the jitter amount satisfies a second jitter degree, where the second jitter degree is greater than the first jitter degree.

[0031] In the above technical solution, the electronic device determines the jitter degree of the electronic device by comparing the relationship between the multiple blur values and the preset threshold. This implementation process is relatively simple and can improve the efficiency of image deblurring.

[0032] In another possible implementation, before obtaining the image to be processed, the method further includes: displaying a shooting interface; obtaining the image to be processed, including: in response to a shooting operation on the shooting interface, obtaining the image to be processed through the camera; after performing deblurring processing on the image to be processed according to the target deblurring strategy to obtain a clear image, it further includes: displaying an interface including the clear image.

[0033] In the above technical solution, in response to a user's shooting operation, an electronic device acquires an image to be processed. Then, the electronic device processes the image to be processed using a target deblurring strategy. Finally, the electronic device presents an interface including a clear image to the user. The process of the electronic device performing deblurring processing on the image to be processed is executed in the background, that is, the user is not aware of this process, which can improve the user's visual experience.

[0034] In a second aspect, the present application provides an image deblurring device applied to an electronic device. The device includes a processing unit. The processing unit is configured to: acquire an image to be processed, where the image to be processed is an image obtained by the camera of the electronic device shooting an object to be photographed; acquire the jitter amount of the electronic device, where the jitter amount is used to represent the degree of jitter of the electronic device during the period when the camera of the electronic device acquires the image to be processed; determine a target deblurring strategy according to the jitter amount; and perform deblurring processing on the image to be processed according to the target deblurring strategy to obtain a clear image.

[0035] In a third aspect, the present application provides an electronic device including a unit for executing any of the methods in the first aspect. The device can be a terminal device or a chip inside the terminal device. The device can include an input unit and a processing unit.

[0036] When the device is a terminal device, the processing unit can be a processor, and the input unit can be a communication interface; the terminal device can further include a memory for storing computer program code. When the processor executes the computer program code stored in the memory, the terminal device is caused to execute any of the image deblurring methods in the first aspect.

[0037] When the device is a chip inside the terminal device, the processing unit can be a processing unit inside the chip, and the input unit can be an output interface, a pin, or a circuit, etc.; the chip can further include a memory, and the memory can be a memory inside the chip (for example, a register, a cache, etc.) or a memory located outside the chip (for example, a read-only memory, a random access memory, etc.); the memory is used to store computer program code. When the processor executes the computer program code stored in the memory, the chip is caused to execute any of the image deblurring methods in the first aspect.

[0038] In a possible implementation, the memory is used to store computer program code; a processor, the processor executes the computer program code stored in the memory. When the computer program code stored in the memory is executed, the processor is used to execute any of the image deblurring methods in the first aspect.

[0039] Fourthly, the present application provides a computer-readable storage medium storing computer program code, which, when run by an image deblurring device, causes the image deblurring device to execute any one of the image deblurring methods in the first aspect.

[0040] Fifthly, the present application provides a computer program product, which includes computer program code, which, when run by an image deblurring device, causes the image deblurring device to execute any one of the image deblurring methods in the first aspect.

[0041] It can be understood that the beneficial effects of the above second to fifth aspects can be referred to the relevant descriptions in the first aspect above, and will not be elaborated here.

[0042] It should be understood that the description of technical features, technical solutions, beneficial effects or similar languages in the present application does not imply that all features and advantages can be achieved in any single embodiment. On the contrary, it can be understood that the description of features or beneficial effects means that at least one embodiment includes specific technical features, technical solutions or beneficial effects. Therefore, the descriptions of technical features, technical solutions or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in this embodiment can be combined in any appropriate manner. Those skilled in the art will understand that an embodiment can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in specific embodiments that do not embody all embodiments. Description of the Drawings

[0043] Figure 1 is a schematic diagram of the user interface and the shooting result of an electronic device when the electronic device shoots an object to be photographed.

[0044] Figure 2 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application.

[0045] Figure 3 is a schematic diagram of an image deblurring method provided by an embodiment of the present application.

[0046] Figure 4 is Figure 3 a schematic diagram of a local blurred image and a clear image involved in the provided image deblurring method.

[0047] Figure 5 is Figure 3 a schematic diagram of a global blurred image and a clear image involved in the provided image deblurring method.

[0048] Figure 6 Yes Figure 3 Schematic diagram of multiple image blocks corresponding to a blurred image involved in the provided image deblurring method.

[0049] Figure 7 Schematic diagram of a system to which the training method provided by an embodiment of the present application is applicable.

[0050] Figure 8 Schematic block diagram of the hardware structure of a neural network processor provided by an embodiment of the present application.

[0051] Figure 9 Schematic diagram of the architecture of a deblurring network model provided by an embodiment of the present application.

[0052] Figure 10 During the training of the above Figure 9 Schematic diagram of a training sample for the deblurring network model shown.

[0053] Figure 11 During the training of the above Figure 9 Schematic diagram of another training sample for the deblurring network model shown.

[0054] Figure 12 Schematic diagram of a model training method provided by an embodiment of the present application.

[0055] Figure 13 Schematic diagram of an image deblurring method provided by an embodiment of the present application.

[0056] Figure 14 Schematic diagram of a user interface displayed on an electronic device when the electronic device executes the image deblurring method provided by an embodiment of the present application.

[0057] Figure 15 Schematic diagram of another user interface displayed on an electronic device when the electronic device executes the image deblurring method provided by an embodiment of the present application. Detailed implementation manners

[0058] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0059] With the development of mobile terminals and the maturity of image processing technology, people's requirements for terminal photography are also gradually increasing. In practical applications, when there is relative movement between the camera of an electronic device and the object to be photographed, it will cause the captured photo to be blurred, resulting in a poor visual experience for users. For example, during shooting, if the camera of the electronic device remains stationary while the object to be photographed is moving, in this case, the obtained image is a partially blurred image, that is, there is a blurred phenomenon in the area where the moving object to be photographed is located in the image. Another example is that during shooting, if the camera of the electronic device is moving, regardless of whether the object to be photographed is moving or not, in this case, the obtained image is a globally blurred image, that is, there is a blurred phenomenon in all areas of the image.

[0060] For example, taking the example of using the camera configured on a mobile phone to photograph an object to be photographed, please refer to Figure 1 as shown in Figure (1) therein. The objects to be photographed presented on the shooting interface S1 provided by the mobile phone include a person and a building. In response to the user triggering the shooting control 10 in the shooting interface S1, the mobile phone photographs the object to be photographed to obtain a captured image. During the shooting process, if the mobile phone shakes, in this case, regardless of whether the object to be photographed is moving or not, the captured image obtained by the mobile phone photographing the object to be photographed is a globally blurred image. For example, Figure 1 the captured image shown in the album interface S2 provided by the mobile phone shown in Figure (2) therein is a globally blurred image, that is, all areas of the image shown in the album interface S2 present a blurred state. During the shooting process, if the mobile phone is stationary (i.e., not shaking) and the object to be photographed (for example, a person is walking during the shooting) is moving, then the captured image obtained by the mobile phone photographing the object to be photographed is a partially blurred image. For example, Figure 1 the captured image shown in the album interface S3 provided by the mobile phone shown in Figure (3) therein is a partially blurred image. Since the object to be photographed, the person, is in a moving state during the shooting process, some areas of the captured image shown in the album interface S3 present blurred content (i.e., the person area presents blurred content), and the remaining areas of the captured image present clear content (i.e., the areas other than the person area).

[0061] In the related art, the electronic device directly inputs the obtained blurred image into a deblurring network for deblurring processing. This easily causes the deblurring network to process the originally clear image content in the blurred image into blurred image content, or easily causes the deblurring network not to restore all the blurred image content in the blurred image to a clear image. That is, the traditional technology has the problem of poor deblurring effect, resulting in a poor visual experience for users.

[0062] As described above, to solve the problem of poor image deblurring effect in the conventional technology, the following technical solution is adopted in the image deblurring method provided in this application: After the electronic device acquires the image to be processed, the electronic device first determines the deblurring strategy (i.e., the target deblurring strategy) to be executed on the image to be processed according to the amount of jitter. Since the amount of jitter represents the degree of jitter of the electronic device during the period when the camera of the electronic device acquires the image to be processed, therefore, the target deblurring strategy determined based on the amount of jitter matches the degree of blurring of the image to be processed. After that, the electronic device then performs deblurring processing on the image to be processed according to the target deblurring strategy determined based on the amount of jitter, and a clear image can be obtained. This method avoids the problem in the conventional technology that the deblurring processing strategy adopted by the electronic device directly performing deblurring processing on the acquired image to be processed does not match the degree of blurring of the image to be processed. Therefore, this method can improve the image deblurring effect and thus better meet the user's visual experience.

[0063] The image deblurring method provided in this application can be applied to an electronic device. For example, the electronic device can be but is not limited to a mobile phone, a smart screen, a tablet computer, a wearable electronic device, a vehicle-mounted electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, a vehicle-mounted device, etc. That is, the specific type of the electronic device is not limited in the embodiments of this application.

[0064] Next, the structure of the electronic device applicable to the image deblurring method provided in this application will be introduced with reference to the accompanying drawings.

[0065] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Refer to Figure 2, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0066] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than Figure 1 shown, or combine certain components, or split certain components, or have different component arrangements. Figure 1 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0067] The processor 110 is used to execute the image deblurring method provided by the embodiments of the present application. The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0068] Among them, the controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0069] A memory can also be set in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0070] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0071] It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only for illustrative purposes and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods in the above embodiments.

[0072] The display screen 194 is used to display user interfaces, images, videos, etc. For example, the display screen 194 can display the Figure 4 images shown below (e.g., clear images) and / or Figure 5 images shown below (e.g., clear images), and Figure 14 and / or Figure 15The user interface shown. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or L display screens 194, where L is an integer greater than 1.

[0073] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular acceleration of the electronic device 100 around three axes (i.e., the x-axis, y-axis, and z-axis) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and makes the lens offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used in scenarios such as navigation and motion-sensing games.

[0074] The touch sensor 180K, also known as the "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. Exemplarily, please refer to Figure 14 Figure (1) below. After the user triggers the camera control 10 in the shooting interface S4 of the mobile phone, the touch sensor 180K can sense the user's trigger operation. Exemplarily, please refer to Figure 15 Figure (2) below. After the user triggers the deblurring control 20 in the photo album interface S8 of the mobile phone, the touch sensor 180K can sense the user's trigger operation. The touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event (e.g., click type, press type, or slide type, etc.). In addition, a visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from the display screen 194.

[0075] Figure 3 It is a schematic diagram of an image deblurring method provided by an embodiment of the present application. This image deblurring method can be executed by the electronic device shown in the above text Figure 2 For example, the electronic device can be, but is not limited to, a mobile phone, a tablet, a computer, etc. As Figure 3 shown, this image deblurring method includes S310 to S380. Below, S310 to S380 will be introduced in detail.

[0076] S310: The first electronic device acquires a blurred image to be processed, where the blurred image to be processed is an image obtained by the camera of the second electronic device shooting an object to be photographed.

[0077] The blurred image to be processed can be a locally blurred image or a globally blurred image, and no specific limitation is made thereto. It can be understood that when the blur in an image is local blur, then this image can be called a locally blurred image, that is, the pixel points in the local area of the locally blurred image are blurred. When the blur in an image is global blur, then this image can be called a globally blurred image, that is, all the pixel points in all areas of the globally blurred image are blurred.

[0078] For example, when the blurred image to be processed is a locally blurred image, the blurred image to be processed can be Figure 4 the locally blurred image shown in (1) of

[0079] where the person area in this locally blurred image is the blurred content, and the area other than the person area in this locally blurred image is clear (i.e., not blurred) content. Figure 5 For example, when the blurred image to be processed is a globally blurred image, the blurred image to be processed can be

[0080] the globally blurred image shown in (1) of

[0081] where all areas in this globally blurred image are blurred content.

[0080] The first electronic device and the second electronic device can be the same electronic device or two different electronic devices, and no specific limitation is made thereto. For example, the first electronic device and the second electronic device can be the same mobile phone. Another example is that the first electronic device can be a computer and the second electronic device can be a mobile phone.

[0081] No specific limitation is made to the image acquisition method for the first electronic device to acquire the blurred image to be processed in S310 above, and it can be selected according to the actual situation.

[0082] As an example of the present application, the first electronic device and the second electronic device are different electronic devices. The blurred image to be processed is an image obtained by the second electronic device photographing an object to be photographed. The first electronic device executes S310, that is, the first electronic device obtains the blurred image to be processed, including: the first electronic device obtains a second blurred image from the second electronic device. In this implementation manner, before the first electronic device executes S310, the second electronic device may also photograph the object to be photographed to obtain the blurred image to be processed.

[0083] As another example of the present application, the blurred image to be processed is an image obtained by the first electronic device photographing an object to be photographed. The first electronic device executes S310, that is, the first electronic device obtains the blurred image to be processed, including: the first electronic device photographs the object to be photographed to obtain the blurred image to be processed.

[0084] S320: The first electronic device divides the blurred image to be processed into P image blocks including P target points, and obtains the first two-dimensional position of the i-th target point among the P target points, where the P image blocks and the P target points correspond one by one, the i-th target point is a point in the i-th image block, P is a positive integer greater than 1, and i = 1, 2,..., P.

[0085] The first two-dimensional position of the i-th target point among the P target points refers to the two-dimensional coordinate position of the i-th target point in the i-th image block among the P image blocks. It can be understood that the size of the first two-dimensional position of the i-th target point is determined in the image coordinate system. The i-th target point among the P target points, i = 1, 2,..., P, that is, the i-th target point refers to each target point among the P target points. For the convenience of description, in the following, the first two-dimensional position of the i-th target point will be denoted as (X i , Y i ).

[0086] There are no specific limitations on the number of the P image blocks, the number of the P target points, and the shape of the i-th image block, and they can be set according to the actual situation. For example, P is 2, 5, 10, or 12, etc. For example, the shape of each image block can be but is not limited to any one of the following shapes: rectangle, square, or triangle, etc.

[0087] In an exemplary case, the target point of the i-th image block is the center point of the i-th image block. For example, taking the blurred image to be processed as the Figure 6 shown image as an example, please refer to Figure 6 , the blurred image to be processed is divided into P = 18 image blocks by multiple dotted lines. The shape of each image block is a rectangle, and the center point of each image block is the center point of each image block.

[0088] In another example, the target point of the i-th image block can be any point other than the center point of the i-th image block. For example, the target point of the i-th image block can also be the point at the lower left corner or the upper right corner of the i-th image block.

[0089] S330: The first electronic device obtains P first three-dimensional positions of the second electronic device, and obtains the i-th second two-dimensional position based on the i-th first three-dimensional position and the first two-dimensional position of the i-th target point, so as to obtain P second two-dimensional positions, where the i-th first three-dimensional position is the pose parameter of the second electronic device when the camera of the second electronic device captures the i-th target point.

[0090] The i-th first three-dimensional position is the pose parameter of the second electronic device when the camera of the second electronic device captures the i-th target point. During the process of using the camera of the second electronic device to photograph the object to be photographed, the gyroscope sensor of the second electronic device will detect and record the angular acceleration of the second electronic device at each moment during the image acquisition period of the second electronic device (that is, including multiple moments), where the second electronic device will capture the information of the corresponding point in the object to be photographed at each moment. In this way, the second electronic device can obtain the pose parameter (i.e., angular acceleration) of the first electronic device at any moment from the gyroscope sensor. For the convenience of description, hereinafter, the i-th first three-dimensional position obtained by the second electronic device from the gyroscope sensor is denoted as (x i , y i , z i ).

[0091] As an example of the present application, the first electronic device obtaining the i-th second two-dimensional position based on the i-th first three-dimensional position and the first two-dimensional position of the i-th target point may include the following steps: The first electronic device may perform the following calculation on the abscissa X in the i-th first two-dimensional position to obtain the abscissa X i ' of the i-th second two-dimensional position: i '

[0092] X i ' = kR i k (-1) X i = H i X i (3.1)

[0093] In the above formula (3.1), R i represents the i-th rotation matrix. k represents the internal and external parameters of the camera of the first electronic device. H i (that is, kR i k (-1) ) represents the i-th transformation matrix.

[0094] Further, the first electronic device may perform the following calculation on the ordinate Y in the i-th first two-dimensional position to obtain the ordinate Y' in the i-th second two-dimensional position: i i '

[0095] Y i ' = kR i k (-1) Y i = H i Y i (3.2)

[0096] In the above formula (3.2), R i represents the i-th rotation matrix. For example, the rotation matrix is the Rodrigues matrix. k represents the internal and external parameters of the camera of the first electronic device. k (-1) represents the result of taking the inverse of k. H i (i.e., kR i k (-1) ) represents the i-th transformation matrix.

[0097] The first electronic device may calculate R in the above formula (3.1) through the following formula: i :

[0098]

[0099] In the above formula (3.3), I represents the identity matrix. (xi, yi, zi) T is the representation form of the rotation axis of (x i , y i , z i ). n i is the vector obtained after processing (xi, yi, zi). nx i represents the component of the ni vector on the x-axis, nyi represents the component of the ni vector on the y-axis, and n zi represents the component of the n i vector on the z-axis.

[0100] The first electronic device calculates (x i , y i , z i ) T :

[0101]

[0102] In the above formula (3.4), n i is the vector obtained after processing (x i , y i , z i ), and n i ​It can be represented by the following mathematical formula:

[0103]

[0104] In the above formula (3.5),

[0105] S340: The first electronic device determines the fuzzy value of the i-th target point according to the first two-dimensional position and the i-th second two-dimensional position of the i-th target point, so as to obtain P fuzzy results of P target points. The i-th fuzzy result includes the fuzzy value of the i-th target point (denoted as γ i ).

[0106] Optionally, the i-th fuzzy result may further include the fuzzy direction of the i-th target point (denoted as θ i ), where the fuzzy direction of the i-th target point represents the jitter direction of the second electronic device when the camera of the second electronic device captures the i-th target point.

[0107] As an example of the present application, the fuzzy result of the i-th target point obtained by the first electronic device according to the first two-dimensional position and the i-th second two-dimensional position of the i-th target point can be represented by the following formula:

[0108]

[0109] In the above formula (3.6), γ i represents the fuzzy value of the i-th target point. θ i represents the fuzzy direction of the i-th target point. (X i ', Y i ) represents the i-th second two-dimensional position. (X i , Y i ) represents the first two-dimensional position of the i-th target point in the fuzzy image to be processed.

[0110] In this way, after the first electronic device executes the above S340, the fuzzy values of P target points corresponding to P image blocks in the fuzzy image to be processed can be obtained. Among the fuzzy values of the P target points, the fuzzy value of each target point can measure the fuzzy value of the image block where each target point is located.

[0111] S350: The first electronic device determines the first fuzzy degree and the second fuzzy degree according to the P fuzzy values of P target points, where the first fuzzy degree is the minimum fuzzy degree among the P fuzzy degrees, and the second fuzzy degree is the maximum fuzzy degree among the P fuzzy degrees.

[0112] The first fuzzy degree is less than the preset fuzzy degree, and the second fuzzy degree is greater than or equal to the preset fuzzy degree.

[0113] The first blurring degree is the blurring degree of the first image block among the P image blocks, and the second blurring degree is the blurring degree of the second image block among the P image blocks.

[0114] For example, taking the blurred image to be processed as Figure 6 the shown partial blurred image as an example, the above-mentioned first image block and second image block can be referred to Figure 6 the first image block and the second image block shown in

[0115] As an example of the present application, when the first electronic device executes S350, it includes: the first electronic device determines the minimum blurring degree among the multiple blurring degrees as the second blurring degree; and, the first electronic device determines the maximum blurring degree among the multiple blurring degrees as the second blurring degree.

[0116] In this way, after the first electronic device executes the above S350, the minimum blurring value and the maximum blurring value among the blurring values of the P target points corresponding to the P image blocks in the blurred image to be processed can be obtained.

[0117] S360: The first electronic device determines whether the first blurring degree is less than a preset threshold.

[0118] As an example of the present application, after the first electronic device executes S360, the first electronic device determines that the first blurring degree is less than the preset threshold (denoted as γ th ), that is, the first electronic device believes that the degree of jitter of the second electronic device when the second electronic device captures the blurred image to be processed is very small (that is, the jitter degree is less than the preset jitter degree). In this case, the first electronic device will consider the captured blurred image to be processed as a partial blurred image. Thereafter, the first electronic device executes S370, that is, calls a local deblurring network to process the blurred image to be processed to obtain a clear image after deblurring the blurred image to be processed.

[0119] As another example of the present application, after the first electronic device executes S360, the first electronic device determines that the first blurring degree is greater than or equal to the preset threshold (γ th ), that is, the first electronic device believes that the degree of jitter of the second electronic device when the second electronic device captures the blurred image to be processed is relatively large (that is, the jitter degree is greater than or equal to the preset jitter degree). In this case, the first electronic device will consider the captured blurred image to be processed as a global blurred image. Thereafter, the first electronic device executes S380, that is, calls a global deblurring network to process the blurred image to be processed to obtain a clear image after deblurring the blurred image to be processed.

[0120] There is no specific limitation on the size of the preset threshold (γ th ). For example, the preset threshold can be but is not limited to 0.8 or 1.

[0121] In this way, the first electronic device executes S350 and S360, that is, determines whether the blurred image to be processed is a global blurred image or a local blurred image. After that, when it is determined that the blurred image to be processed is a global blurred image, subsequently, the first electronic device performs deblurring processing on the blurred image to be processed by using a global deblurring network to obtain a clear image. When it is determined that the blurred image to be processed is a local blurred image, subsequently, the first electronic device performs deblurring processing on the blurred image to be processed by using a local deblurring network to obtain a clear image.

[0122] S370: The first electronic device calls a local deblurring network to perform deblurring processing on the blurred image to be processed, and obtains a clear image.

[0123] The local deblurring network is used to deblur the local blurred area in the blurred image to be processed to obtain a clear image. Neither the structure of the local deblurring network nor the training method for obtaining the local deblurring network is specifically limited.

[0124] As an example of the present application, the local deblurring network may be the local deblurring network obtained by the model training method provided below in the present application. The training method for training the neural network model to obtain the local deblurring network may refer to the relevant description below. The structure of the local deblurring network may refer to the network shown below. Details are not described herein again. Figure 12 As another example of the present application, the local deblurring network may be a network already proposed in the related art, and the training method for obtaining the local deblurring network may be a training method already proposed in the related art. For example, the training method includes training an encoder-decoder by using a plurality of training samples to obtain a local deblurring network, where each training sample includes a blurred image and a label image (i.e., the clear image corresponding to the blurred image). The clear image in S370 above is the image obtained after performing local deblurring processing on the blurred image to be processed. Figure 12 For example, taking the local blurred image shown in (1) in Figure 9 as the blurred image to be processed in S370 as an example, the clear image obtained after the first electronic device executes S370 may refer to the clear image shown in (2) in

[0125] In this way, the first electronic device can process the blurred image to be processed by using a local deblurring network that matches the local blur type of the blurred image to be processed, which can improve the deblurring effect of the blurred image to be processed, so that the obtained clear image can better meet the user's needs.

[0126] For example, taking the blurred image to be processed in S370 as Figure 4 the local blurred image shown in (1) in Figure 4 as an example, the clear image obtained after the first electronic device executes S370 may refer to the clear image shown in (2) in

[0127] In this way, the first electronic device can process the blurred image to be processed by using a local deblurring network that matches the local blur type of the blurred image to be processed, which can improve the deblurring effect of the blurred image to be processed, so that the obtained clear image can better meet the user's needs.

[0128] S380: The first electronic device invokes the global deblurring network to deblur the to-be-processed blurred image and obtain a clear image.

[0129] The global deblurring network is used to deblur the global blurred area in the to-be-processed blurred image to obtain a clear image. Neither the structure of the global deblurring network nor the training method for obtaining the global deblurring network is specifically limited.

[0130] As an example of the present application, the global deblurring network may be the global deblurring network obtained by the model training method provided below in the present application. For the training method of training the neural network model to obtain the global deblurring network, reference may be made to the relevant description below. Figure 12 For the structure of the global deblurring network, reference may be made to the network shown below. Details are not described herein again. Figure 12 For the structure of the global deblurring network, reference may be made to the network shown below. Details are not described herein again. Figure 9 For the structure of the global deblurring network, reference may be made to the network shown below. Details are not described herein again.

[0131] As another example of the present application, the global deblurring network may be a network already proposed in the related art, and the training method for obtaining the global deblurring network may be a training method already proposed in the related art. For example, the training method includes training an encoder-decoder using a plurality of training data pairs to obtain the global deblurring network, where each training data pair includes a blurred image and a label image (i.e., the clear image corresponding to the blurred image).

[0132] The clear image in S380 above is the deblurred image obtained after performing global deblurring processing on the to-be-processed blurred image.

[0133] For example, taking the global blurred image shown in (1) in Figure 5 as the to-be-processed blurred image in S380, the clear image obtained after the first electronic device executes S380 may be referred to the clear image shown in (2) in Figure 5 In this way, the first electronic device can use the global deblurring network that matches the global blur type of the to-be-processed blurred image to process the to-be-processed blurred image, which can improve the deblurring effect of the to-be-processed blurred image and make the obtained clear image better meet the user's needs.

[0134] It should be understood that the above-described image deblurring method is only illustrative and does not constitute any limitation on the image deblurring method provided by the embodiments of the present application. For example, it is also possible to determine the jitter degree of the electronic device during the period when the camera of the electronic device captures the to-be-processed blurred image based on other existing methods.

[0135] It should be understood that the above Figure 3 shown image deblurring method is only illustrative and does not constitute any limitation on the image deblurring method provided by the embodiments of the present application. For example, it is also possible to determine the jitter degree of the electronic device during the period when the camera of the electronic device captures the to-be-processed blurred image based on other existing methods.

[0136] In the embodiments of the present application, after the electronic device obtains the blurred image to be processed, the electronic device first determines a deblurring strategy (i.e., the target deblurring strategy) to be executed on the blurred image to be processed according to the amount of jitter. Since the amount of jitter represents the degree of jitter of the electronic device during the period when the camera of the electronic device captures the blurred image to be processed, therefore, the target deblurring strategy determined based on the amount of jitter is matched with the degree of blurring of the blurred image to be processed. After that, the electronic device can obtain a clear image by performing deblurring processing on the blurred image to be processed according to the target deblurring strategy determined based on the amount of jitter. This method avoids the problem in the traditional technology that the deblurring effect is poor due to the mismatch between the deblurring processing strategy and the degree of blurring of the blurred image to be processed when the electronic device performs deblurring processing on the obtained blurred image to be processed. Therefore, this method can improve the image deblurring effect and thus better meet the user's visual experience.

[0137] As described above, the image deblurring method provided in the present application involves a global deblurring network and a local deblurring network. Therefore, the present application also provides a training method for training a neural network model to obtain the global deblurring network and the local deblurring network. Before introducing the training method provided in the present application, below, the system architecture and the hardware structure of the neural network processor applicable to the training method provided in the present application will be introduced with reference to the accompanying drawings.

[0138] See Figure 7 , Figure 7 is a schematic diagram of a system 700 applicable to the training method provided in the embodiments of the present application. It should be understood that the following description should not be construed as a limitation of any example of the present disclosure. As Figure 7In the illustrated system 700, labeled training data can be stored in database 730. Database 730 can be located in a server or data center, or can be provided as a service by a cloud computing service provider. In the context of the present disclosure, labeled training data refers to training data for learning the training weights of the deblurring neural network 701 (also referred to as the deblurring network 701 for simplicity). In one example, the labeled training data includes training images, label images, and motion mask images, where the training images are blurred images, the label images are the corresponding clear images of the training images, and the motion mask images are images including the features of the blurred regions of the training images. The labeled training data is different from the training data during application, and the training data during application can be unlabeled real-world data (e.g., real-world images captured by the application device 710, which will be discussed below) or unlabeled test data. The data during application does not include label images and motion mask images. As will be further discussed below, the training during application of the deblurring network 701 can be performed with a single input real-world image to obtain the training weights during application of the deblurring network 701, and the deblurring network 701 including the training weights during application can be used to predict the corresponding single deblurred output image based on the single input real-world image.

[0139] Database 730 can contain, for example, labeled training data that has been previously collected and is generally used for training models related to image tasks (e.g., image recognition). The input images of the labeled training data stored in database 730 can optionally be images collected from the application device 710 (which can be a user device) (e.g., with the user's consent). For example, images captured by the camera of the application device 710 and stored on the application device 710 can optionally be anonymized and uploaded to database 730 for storage as input images of the labeled training data. The labeled training data stored in database 730 can include the training images, label images, and motion mask images described above.

[0140] As will be further discussed below, the training device 720 can be used to train the deblurring network 701 based on the training data stored in database 730. Additionally, or alternatively, the training device 720 can use training data obtained from other sources (e.g., distributed storage (or cloud storage platform)) to train the deblurring network 701. Training the deblurring network 701 (i.e., the result of the training of the training device 720) has a set of training weights. According to the examples disclosed herein, the application device 710 can further train the trained deblurring network 701 for deblurring specific blurred real-world images. The training during application of the application device 710 can be performed with images (e.g., digital photos) captured by the camera (not shown in the figure) of the application device 710. The application device 710 may not have access to the training data stored in database 730.

[0141] In the examples disclosed herein, the trained deblurring network 701 may be implemented in the processing unit 711 of the application device 710. For example, the deblurring network 701 may be encoded and then stored as instructions in a memory (not shown in the figure) of the application device 710, and the processing unit 711 executes the stored instructions to implement the deblurring network 701. In some examples, the deblurring network 701 may be encoded and then stored as instructions in the memory of the processing unit 711 (e.g., the weights of the deblurring network 701 may be stored in the corresponding weight memory of the processing unit 711, and the processing unit 711 may be embodied as a neural network processor 800 as shown in Figure 8 ). In some examples, the deblurring network 701 may be implemented in an integrated circuit of the application device 710 (as software and / or hardware). Although Figure 7 an example where the training device 720 is separated from the application device 710 is shown, it should be understood that the present disclosure is not limited to this embodiment. In some examples, there may be no separate training device 720 and application device 710. That is, the training of the deblurring network 701 and the application of the deblurring network 701 can be performed on the same device (e.g., the application device 710).

[0142] The application device 710 may be a user device, such as a client terminal, a mobile terminal, a tablet computer, a laptop computer, an augmented reality (AR) device, a virtual reality (VR) device, or a vehicle-mounted terminal, etc. The application device 710 may also be a server, a cloud computing platform, etc., which can be accessed by a user device. In Figure 7 , the application device 710 includes an I / O interface 712 for data interaction with external devices. For example, the application device 710 may provide upload data (e.g., image data, such as photos and / or videos captured by the application device 710) to the database 730 through the I / O interface 712. Although Figure 7 an example where a user directly interacts with the application device 710 is shown, it should be understood that the present disclosure is not limited to this embodiment. In some examples, there may be a separation between the user device and the application device 710, the user interacts with the user device, and the user device in turn exchanges data with the application device 710 through the I / O interface 712.

[0143] In this example, the application device 710 includes a data memory 714, which can be a system memory (e.g., random access memory (RAM), read-only memory (ROM), etc.) or a mass storage device (e.g., solid state drive and hard disk drive, etc.). The data memory 714 can store data accessible by the processing unit 711. For example, the data memory 714 can be separate from the processing unit 711 and store the captured images and / or the restored images on the application device 710.

[0144] In some examples, the application device 710 can optionally call data, code, etc. from the external data storage system 750 for processing, or can store the data, instructions, etc. obtained through the corresponding processing in the data storage system 750.

[0145] It should be noted that Figure 7 is only a schematic diagram of the example system architecture 700 according to the embodiments of the present disclosure. Figure 7 The relationships and interactions among the devices, components, processing units, etc. shown are not intended to limit the present disclosure.

[0146] Figure 8 is a schematic block diagram of the hardware structure of the neural network processor provided by the embodiments of the present application.

[0147] The neural network processor 800 can be disposed on an integrated circuit (also referred to as a computer chip). The neural network processor 800 can be disposed in Figure 7 the application device 710 shown, perform calculations for the processing unit 711 and implement the deblurring network 701 (including performing the training during the application of the deblurring network). Additionally, or alternatively, the neural network processor 800 can be disposed in Figure 7 the training device 720 shown to perform the training of the deblurring network 701. All algorithms of the layers in the neural network (e.g., the layers of the deblurring network 701 discussed further below) can be implemented in the neural network processor 800.

[0148] The neural network processor 800 can be any processor capable of performing the calculations required in a neural network (e.g., calculations of a large number of exclusive OR operations). For example, the neural network processor 800 can be a neural processing unit (NPU), a tensor processing unit (TPU), a graphics processing unit (GPU), etc. The neural network processor 800 can be a coprocessor of an optional host central processing unit (CPU) 820. For example, the neural network processor 800 and the host CPU 820 can be installed on the same package. The host CPU 820 can be responsible for executing the core functions of the application device 710 (e.g., execution of the operating system (OS), management of communications, etc.). The host CPU 820 can manage the operations of the neural network processor 800, e.g., by assigning tasks to the neural network processor 800.

[0149] The neural network processor 800 includes an arithmetic circuit 803. A controller 804 of the neural network processor 800 controls the arithmetic circuit 803, so as to extract data (e.g., matrix data) from an input memory 801 and a weight memory 802 of the neural network processor 800, and perform data operations (e.g., addition and multiplication operations), for example.

[0150] In some examples, multiple processing units (also called process engines (PEs)) are included inside the arithmetic circuit 803. In some examples, the arithmetic circuit 803 is a two-dimensional systolic array. In other examples, the arithmetic circuit 803 can be a one-dimensional systolic array or other electronic circuits capable of implementing mathematical operations such as multiplication and addition. In some examples, the arithmetic circuit 803 is a general matrix processor.

[0151] In an example operation, the arithmetic circuit 803 obtains the weight data of the weight matrix B from the weight memory 802 and caches the weight data in each PE of the arithmetic circuit 803. The arithmetic circuit 803 obtains the input data of the input matrix A from the input memory 801 and performs matrix operations based on the input data of matrix A and the weight data of matrix B, and the partial or final matrix result obtained is stored in an accumulator 808 of the neural network processor 800.

[0152] In this example, the neural network processor 800 includes a vector calculation unit 807. The vector calculation unit 807 includes a plurality of arithmetic processing units. If necessary, the vector calculation unit 807 further processes the output from the arithmetic circuit 803 (which can be retrieved by the vector calculation unit 807 from the accumulator 808), such as vector multiplication, vector addition, exponential operation, logarithmic operation, or magnitude comparison. The vector calculation unit 807 can be mainly used for operations in the non-convolutional layer or fully connected layer of the neural network. For example, the vector calculation unit 807 can perform processing such as pooling or normalization on the operations. The vector calculation unit 807 can apply a non-linear function to the output of the arithmetic circuit 803, such as a vector of cumulative values, to generate activation values. The activation values can be used by the arithmetic circuit 803 as the activation input for the next layer of the neural network. In some examples, the vector calculation unit 807 generates normalized values, combined values, or both normalized values and combined values.

[0153] In this example, the neural network processor 800 includes a storage unit access controller 805 (also known as direct memory access control (DMAC)). The storage unit access controller 805 is used to access a memory external to the neural network processor 800 (e.g., the data memory 714 of the execution device 710) via the bus interface unit 810. The storage unit access controller 805 can access data from a memory external to the neural network processor 800 and directly transfer the data to one or more memories of the neural network processor 800. For example, the storage unit access controller 805 can directly transfer weight data to the weight memory 802, or directly transfer input data to the unified memory 806 and / or the input memory 801. The unified memory 806 is used to store input data and output data (e.g., the processed vectors from the vector calculation unit 807).

[0154] The bus interface unit 810 is also used for the interaction between the storage unit access controller 805 and the instruction fetch memory (also known as the instruction fetch cache) 809. The bus interface unit 810 is also used to enable the instruction fetch memory 809 to obtain instructions from a memory external to the neural network processor 800 (e.g., the data memory 714 of the application device 710). The instruction fetch memory 809 is used to store instructions for the controller 804.

[0155] Generally, the unified memory 806, the input memory 801, the weight memory 802, and the instruction fetch memory 809 are all memories of the neural network processor 800 (also known as on-chip memories). The data memory 714 is independent of the hardware architecture of the neural network processor 800.

[0156] In the image deblurring method provided by this application, there is a step where a first electronic device calls a deblurring network model (for example, a global deblurring network or a local deblurring network) to deblur a blurred image. Below, the deblurring network model provided by this application will be introduced in detail with reference to the accompanying drawings. It should be noted that the training of the deblurring network model and the application of the deblurring network model will be further introduced below.

[0157] Figure 9 FIG. is a schematic diagram of the architecture of a deblurring network model provided by an embodiment of this application. For simplicity, in the following, the neural network layers (or blocks) of the deblurring network will be simply referred to as layers. As Figure 9 shown, the deblurring network model is based on a fully convolutional neural network. The deblurring network model includes an encoder, a decoder, skip connections between layers of the same size (i.e., Figure 9 the dotted arrows shown in ), and a connection part, where the connection part includes a plurality of convolutional layers for connecting the encoder and the decoder together.

[0158] In an embodiment of this application, the input of the deblurring network model includes a training data set, and the output of the deblurring network model includes a predicted deblurred image (briefly denoted as I' b ), and the predicted deblurred image is also called a clear image.

[0159] The training data set includes N training samples, where each training sample includes a blurred image (briefly denoted as I b ), a label image, and a motion mask image (briefly denoted as M b ), that is, the training data set includes N blurred images, N label images, and N motion mask images, and N is a positive integer.

[0160] The above N blurred images correspond one-to-one with the N label images, and each label image represents the clear image (i.e., the non-blurred image) corresponding to the corresponding blurred image. In one example, the blurred image in the training data set can be represented as a two-dimensional matrix, and this two-dimensional matrix is the encoding of multiple channels (for example, the red - green - blue (RGB) channels) of multiple individual pixels of the input image.

[0161] For example, when performing local blur training to obtain a local deblurring network, the training image can be Figure 10 the local blurred image shown in (2) of, and the label image can be Figure 10 the clear image shown in (1) of. Another example is that when performing global blur training to obtain a global deblurring network, the training image can be Figure 11 the global blurred image shown in (2) of, and the label image can be Figure 11The clear image shown in (1) therein.

[0162] The above N motion mask images correspond one-to-one with the N blurred images, and each motion mask image (M b ) includes the region where the moving object is located in the corresponding blurred image (I b ). That is to say, the region of interest (ROI) in the blurred image included in each motion mask image (M b ) is the region where the moving object is located in the blurred image (I b ), and the remaining region included in each motion mask image (M b ) is the background region. In one example, the pixel values of the region where the moving object is located (i.e., the region of interest) in the corresponding blurred image (I b ) included in each motion mask image (M b ) can be set to "1", that is, the region of interest appears as white in the motion mask image, and the pixel values of the remaining region (i.e., the background region) of each motion mask image (M b ) can be set to "0", that is, the background region appears as black in the motion mask image (M b ).

[0163] For example, taking the partial blurred image shown in (2) therein as the blurred image in the training data, the moving object in this partial blurred image is a person. Therefore, the motion mask image corresponding to this partial blurred image can be the motion mask image shown in (3) therein. The white region with pixel value "1" in this motion mask image is the region where the moving object is located (i.e., the region where the person is located) in the blurred image shown in (2) therein, and the motion mask image includes a black region with pixel value "0". The corresponding Figure 10 region in the blurred region shown in (2) therein except for the region where the moving object is located. Another example is taking the global blurred image shown in (2) therein as the blurred image in the training data. The moving objects in this partial blurred image include a person and a building. The motion mask image corresponding to this global blurred image can be the motion mask image shown in (3) therein. The white region with pixel value "1" in this motion mask image is the region where the moving objects are located (i.e., the person region and the building region) in the blurred image shown in (2) therein, and the black region with pixel value "0" corresponds to the Figure 10 region in the blurred region shown in (2) therein except for the region where the moving objects are located. Figure 10 Figure 10 Figure 11 Figure 11 Figure 11 Figure 11 ​​​​​​

[0164] As described above, the input of the deblurring network model includes a training data set. Among them, the processing flow of each of the N training samples included in the training data set by the deblurring network model is the same. Therefore, in the following, the training of the deblurring network model using one training sample is taken as an example for description.

[0165] As Figure 9 shown, the encoder of the deblurring network model includes an input layer 910, convolutional layers 920 (i.e., convolutional layer 920a1 and convolutional layer 920a2), and downsampling layers 930 (i.e., downsampling layer 930a and downsampling layer 930b). Among them, a convolutional layer is connected before each downsampling layer. For example, convolutional layer 920a2 is connected before downsampling layer 930b, and convolutional layer 920a1 is connected before downsampling layer 930a. The size of the convolution kernel of convolutional layer 920a1 is larger than the size of the convolution kernel of convolutional layer 920a2, and the size of the convolution kernel of downsampling layer 930a is larger than the size of the convolution kernel of downsampling layer 930b. For example, the size of the convolution kernel of convolutional layer 920a1 is 3×3, and the size of the convolution kernel of convolutional layer 920a2 is 2×2.

[0166] The encoder is used to perform feature encoding on the blurred image (I b ) input to the deblurring network model to obtain the feature map of the blurred image (I b ). Specifically, the input layer 910 is used to receive the blurred image (I b ), the label image, and the motion mask image (M b ) input to the deblurring network model, and output the blurred image (I b ), the label image, and the motion mask image (M b ). Convolutional layer 920a1 obtains the blurred image (I b ), the label image, and the motion mask image (M b ) from the input layer 910, and performs a convolution operation on the blurred image (I b ) to obtain the feature b of the feature encoding of the blurred image (I Figure 1 (i.e., feature representation). The feature Figure 1 output by convolutional layer 920a1, the label image, and the motion mask image (M b ) serve as the input to downsampling layer 930a. Downsampling layer 930a performs downsampling processing (also known as pooling processing) on the feature Figure 1 to obtain the feature Figure 2 (feature representation). After that, downsampling layer 930a outputs the feature Figure 2 , the label image, and the motion mask image (M b ). Next, convolutional layer 920a2 performs convolution on the feature Figure 2Perform a convolution operation to obtain features Figure 3 (i.e., feature representation). After that, the convolutional layer 920a2 outputs features Figure 3 , a label image, and a motion mask image (M b ). The downsampling layer 930b performs downsampling on the features Figure 3 to obtain features Figure 4 (i.e., feature representation). After that, the downsampling layer 930b outputs features Figure 4 , a label image, and a motion mask image (M b ). The features Figure 4 , a label image, and a motion mask image (M b ) output by the downsampling layer 930b are processed by multiple convolutional layers included in the connection part and used as input data for the decoder.

[0167] As Figure 9 shown, the decoder of the deblurring network model includes a deblurring feature reconstruction layer 940 (i.e., a deblurring feature reconstruction layer 940a and a deblurring feature reconstruction layer 940b), an upsampling layer 950 (i.e., an upsampling layer 950a and an upsampling layer 950b), a convolutional layer 920b (i.e., a convolutional layer 920b1 and a convolutional layer 920b2), and an output layer 960. Among them, a deblurring feature reconstruction layer is connected before each upsampling layer. The size of the convolutional kernel of the deblurring feature reconstruction layer 940a is smaller than the size of the convolutional kernel of the deblurring feature reconstruction layer 940b. The size of the convolutional kernel of the upsampling layer 950a is smaller than the size of the convolutional kernel of the upsampling layer 950b.

[0168] The deblurring feature reconstruction layer 940 is used to optimize the feature map corresponding to the obtained blurred image (I b ) according to the motion mask image (M b ) to generate an optimized feature map (i.e., feature representation). Next, taking the deblurring feature reconstruction layer 940a as an example, the function of the deblurring feature reconstruction layer 940 provided in the embodiments of the present application will be described. As Figure 9 shown, the size of the input feature map (i.e., feature representation) of the deblurring feature reconstruction layer 940a is h×w×n, where h represents the length of the feature map, w represents the width of the feature map, and n represents the number of layers of the feature map. In the deblurring feature reconstruction layer 940a, the parameters of the deblurring feature reconstruction layer 940a are adjusted according to the difference between the motion mask image (M b ) and the input feature image, so that the difference between the output feature image of the adjusted deblurring feature reconstruction layer 940a and the motion mask image (M b ) is less than a preset difference. For example, taking the blurred image (I b ) as a local blurred image, please refer to the motion mask image (M b ) Figure 9In the motion mask image, the feature map corresponding to a feature layer obtained by the deblurring feature reconstruction layer 940a is as Figure 9 shown in the 1-layer feature image in

[0169] Skip connections (also known as short connections) can supplement the same-scale features extracted in the corresponding downsampling layer 930 in the upsampling layer 950 to restore the features lost during downsampling. Skip connections can be used in residual neural networks to facilitate faster learning of the weights of the deblurring network. For example, Figure 9 a shown skip connection supplements the same-scale features extracted in the downsampling layer 930a in the upsampling layer 950b, and another skip connection supplements the same-scale features extracted in the downsampling layer 930b in the upsampling layer 950a.

[0170] The upsampling layer 950 splices the feature map corresponding to the output of the corresponding downsampling layer and the feature map output by the previous layer (that is, fuses the deep features with the shallow features to make the information richer). After that, the spliced feature map is decoded to generate an output feature map (that is, a feature representation). For example, taking the upsampling layer 950a as an example, the upsampling layer 950a splices the feature map (that is, a feature representation) output by the deblurring feature reconstruction layer 940a and the feature map (that is, a feature representation) output by the downsampling layer 930b. After that, the spliced feature map (that is, a feature representation) is decoded to generate an output feature map (that is, a feature representation). For example, taking the upsampling layer 950b as an example, the upsampling layer 950b splices the feature map output by the deblurring feature reconstruction layer 940b and the feature map (that is, a feature representation) output by the downsampling layer 930a. After that, the spliced feature map (that is, a feature representation) is decoded to generate an output feature map (that is, a feature representation).

[0171] The convolutional layer 920 connected after the upsampling layer 950 is used to perform a convolutional operation on the feature map (that is, a feature representation) output by the upsampling layer 950b.

[0172] The output layer 960 performs a convolutional operation on the feature map (that is, a feature representation) output by the connected convolutional layer 920b to generate a predicted deblurred image (I' b ). After that, according to the difference between the predicted deblurred image (I' b ) and the label image, the parameters of the encoder, the adjusted deblurring feature reconstruction layer 940, the upsampling layer 950, and the output layer 960 are adjusted until the preset training conditions are met, and the model training is stopped, thereby obtaining a trained deblurring network.

[0173] It should be understood that the above Figure 9The architecture of the deblurring network shown is only schematic and does not impose any limitation on the architecture of the deblurring network applicable to the image deblurring method provided in this application. That is, Figure 9 The architecture of the deblurring network shown can be modified (for example, having a smaller or larger number of neural network layers). For example, the convolutional layer 920 in the encoder can further include a larger number of convolutional layers, the downsampling layer 930 can further include a larger number of downsampling layers, and the upsampling layer 950 can further include a larger number of downsampling layers.

[0174] Next, taking the architecture of the deblurring network shown above Figure 9 as an example, this application embodiment will introduce the training method for training the deblurring network to obtain a global deblurring network and the training method for training the deblurring network to obtain a local deblurring network in combination with Figure 12 It should be noted that the model structures of the global deblurring network and the local deblurring network in the embodiments of this application are the same. The difference lies in that the model parameters of the global deblurring network are different from those of the local deblurring network, and the training datasets for obtaining the global deblurring network and the training datasets for obtaining the local deblurring network are different.

[0175] Figure 12 is a schematic diagram of a model training method provided by an embodiment of this application. It should be understood that Figure 12 the model training method shown is only schematic and does not impose any limitation on the model training method provided in this application.

[0176] Figure 12 is a schematic diagram of a model training method provided by an embodiment of this application. This model training method can be executed by the training device 720 shown above Figure 7 As shown in Figure 12 , this model training method includes S1210 to S1250. Next, S1210 to S1250 will be introduced in detail.

[0177] S1210: Obtain a training dataset, where the training dataset includes N training samples. The i-th training sample includes the i-th blurred image, the i-th motion mask image, and the i-th label image. The i-th motion mask image includes the blurred region of the i-th training image, and the i-th label image is the clear image corresponding to the i-th blurred image, i = 1, 2,..., N, and N is a positive integer.

[0178] The blurred images in any two of the N training samples can be the same or different.

[0179] The i-th motion mask image includes the blurred region of the i-th training image, also known as the feature that the i-th motion mask image includes the blurred region of the i-th training image. In the embodiments of the present application, the i-th motion mask image includes the region where the moving object (the moving object appears as blurred content in the i-th blurred image) in the i-th blurred image is located. It can be understood that during the time period when the i-th blurred image is captured, the moving object moves, resulting in the corresponding region of the i-th blurred image being in a blurred state. In one example, the pixel values of the region where the moving object is located in the i-th blurred image in the i-th motion mask image can be set to "1", that is, this region appears white in the i-th motion mask image, and the pixel values of the remaining region of the i-th motion mask image can be set to "0", that is, this region appears black in the i-th motion mask image.

[0180] As an example of the present application, in the scenario where local deblurring training is performed on the Figure 9 shown deblurring network to obtain the local deblurring network, the i-th blurred image included in the i-th training sample among the N training samples is a local blurred image, the i-th motion mask image includes the blurred region (i.e., the region of the moving object) of this local blurred image, and the i-th label image is the clear image corresponding to this local blurred image.

[0181] For example, the training image in a training sample included in the training dataset can be Figure 10 the local blurred image shown in (2) of Figure 10 , and the human region (i.e., the moving region) in this local blurred image is blurred. The motion blurred image in this training sample can be Figure 10 the motion mask image shown in (3) of

[0182] As an example of the present application, in the scenario where global deblurring training is performed on the Figure 9 shown deblurring network to obtain the global deblurring network, the i-th blurred image included in the i-th training sample among the N training samples is a global blurred image, the i-th motion mask image includes the blurred region (i.e., the region of the moving object) of this global blurred image, and the i-th label image is the clear image corresponding to this global blurred image.

[0183] For example, the training image in a training sample included in the training dataset can be Figure 11The global blurred image shown in (2) therein, where the person area (i.e., the moving area) in the global blurred image is blurred. The motion blurred image in one training sample can be Figure 11 The motion mask image shown in (3) therein, where the pixel values of all areas (i.e., the blurred areas) in the motion mask image are "1". The label image in one training sample can be Figure 11 The clear image shown in (1) therein.

[0184] There is no specific limitation on the method for obtaining N training samples, which can be set according to the actual scenario.

[0185] As an example of the present application, a high-frame-rate camera can be used to shoot a video, and the blurred pictures and clear pictures in consecutive frames are found from the video as a set of data (i.e., the i-th training image and the i-th label image in the i-th training sample).

[0186] For example, the i-th training image and the i-th label image in the i-th training sample can be obtained through the following steps: Obtain N video streams, where the i-th video stream includes a series of consecutive frames; perform weighted averaging on the series of consecutive frames included in the i-th video stream to obtain the i-th training image; determine a clear image frame among the series of consecutive frames included in the i-th video stream as the i-th label image to obtain the i-th label image.

[0187] As another example of the present application, the i-th clear picture is blurred with a known or randomly generated motion blur kernel to generate a corresponding set of data (i.e., the i-th training image and the i-th label image in the i-th training sample).

[0188] In this way, after performing the above processing on the i-th video stream among the N video streams respectively, N training images and N label images in N training samples can be obtained.

[0189] As an example of the present application, the i-th motion mask image in the i-th training sample can be obtained through the following steps: including: obtaining the moving foreground area in the i-th training image using a background subtraction algorithm; extracting the mask area from the moving foreground area using a connected component extraction algorithm to obtain the i-th motion mask image. Optionally, after obtaining the i-th motion mask image, an image filter can also be used to smooth the i-th motion mask image, or a morphological filtering operation can be used to filter the i-th motion mask image to filter out smaller noise areas in the i-th motion mask image.

[0190] In this way, after performing the above processing on the i-th blurred image among the N training samples respectively, N motion mask images in N training samples can be obtained.

[0191] S1220: Input the N training samples in the training dataset into the encoder. The encoder performs blurred feature extraction on the N training images corresponding to the N training samples to obtain N feature images #1 corresponding to the N training images. Among them, the initial deblurring network includes an encoder.

[0192] As an example of this application, the structure of the initial deblurring network can refer to Figure 9 the network shown. For the structure of the initial deblurring network and the steps of the encoder for feature extraction of the i-th training image among the N training images, please refer to the relevant descriptions above and will not be elaborated here. It can be understood that the encoder is Figure 9 the encoder shown in

[0193] S1230: Input the N feature images #1 and the N motion mask images output by the encoder into the first network module of the decoder. Adjust the parameters of the first network module according to the loss value #1, so that the adjusted first network module outputs N feature images #2. Among them, the loss value #1 is determined based on the difference between the i-th feature image #1 and the i-th motion mask image. The initial deblurring network further includes a decoder.

[0194] In the above S1230, adjusting the parameters of the first network module using the loss value #1 means adjusting the weights (i.e., parameters) of the first network module using the loss value #1. As an example of this application, the structure of the initial deblurring network can refer to Figure 9 the network shown. The decoder is Figure 9 the decoder shown in Figure 9 the two defocused feature reconstruction layers 940 shown in

[0195] The loss value #1 is based on the difference between the i-th feature image #1 and the i-th motion mask image. Among them, the difference between the i-th feature image #1 and the i-th motion mask image can be calculated using a loss function to obtain the loss value #1.

[0196] During the process of training a neural network model, since the output of the neural network model is as close as possible to the value that is truly desired to be predicted, the weight vectors of each layer of the neural network can be updated by comparing the predicted value of the current network with the truly desired target value and then based on the difference between the two. (Of course, there is usually an initialization process before the first update, that is, parameters are preconfigured for each layer in the neural network model.) For example, if the predicted value of the model is too high, the weight vector is adjusted to make it predict lower, and it is continuously adjusted until the neural network model can predict the truly desired target value or a value very close to the truly desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function. They are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value of the loss function, that is, the loss value, the greater the difference. Then the training of the neural network model becomes a process of minimizing this loss as much as possible. In the embodiments of the present application, the loss function is not specifically limited and can be selected according to user needs. For example, the loss function can be but is not limited to the mean square error (MSE) loss function or the mean absolute error (MAE) loss function.

[0197] Exemplarily, taking the case where the loss value #1 is calculated by the MSE loss function as an example, the loss value #1 can be expressed by the following formula:

[0198]

[0199] In the above formula (12.1), MSE represents the loss value #1, y i represents the i-th training image, represents the i-th predicted image, represents the difference between the i-th training image and the i-th predicted image.

[0200] In the above S1230, according to the N feature images #1 and the N motion mask images, the loss value #1 can be calculated. Then, adjusting the parameters of the first network module in the decoder according to the loss value #1 is to make the loss value #1 corresponding to the N feature images #1 as small as possible to better meet the requirements of image deblurring.

[0201] Thus, in the decoder, the parameters of the first network module in the decoder are adjusted using the differences between the N motion mask images corresponding to the N training images among the N training samples and the N feature images #1 corresponding to the N training images obtained by the encoder. After that, the adjusted first network module can output N feature images #2 obtained by optimizing the N feature images #1, that is, the N feature images #2 output by the adjusted first network module can more accurately represent the blurred features in the corresponding training images.

[0202] S1240: The second network module of the decoder processes the N feature images #2 and outputs N predicted label images.

[0203] As an example of the present application, the structure of the initial deblurring network can refer to Figure 9 the network shown in Figure 9 The two upsampling layers 950 shown in, and the content not described in detail here can refer to the relevant descriptions above.

[0204] S1250: Adjust the parameters of the encoder, the parameters of the adjusted first network module, and the parameters of the second network module according to the loss value #2 to obtain a trained deblurring network, where the loss value #2 is determined based on the difference between the i-th training image and the i-th label image.

[0205] In the above S1250, the parameters of the encoder, the parameters of the adjusted first network module, and the parameters of the second network module are adjusted using the loss value #2, that is, the weights of the encoder, the weights of the adjusted first network module, and the weights of the second network module are adjusted using the loss value #2.

[0206] For example, the loss value #2 can be, but is not limited to, the loss value calculated by the electronic device using any one of the following loss functions for the difference between the i-th training image and the i-th label image: MSE loss function or MAE loss function.

[0207] As an example of the present application, when the i-th training image among the N training samples in the above S1210 is a locally blurred image, the trained deblurring network obtained after performing the above S1250 is called a local deblurring network.

[0208] As another example of the present application, when the i-th training image among the N training samples in the above S1210 is a globally blurred image, the trained deblurring network obtained after performing the above S1250 is called a global deblurring network.

[0209] In one example, the parameters of the encoder, the parameters of the adjusted first network module, and the parameters of the second network module are adjusted according to loss value #2 to obtain a trained deblurring network, including: adjusting the parameters of the encoder, the parameters of the adjusted first network module, and the parameters of the second network module according to loss value #2 until a preset training condition is met, ending the model training, and obtaining a trained deblurring network. The iterative stop end condition for performing model training on the initial deblurring network can be but is not limited to at least one of the following conditions: the loss value of the loss function is less than a preset threshold, the current number of iterations meets the requirements of the preset number of iterations, or the current model training time meets the requirements of the preset training duration.

[0210] Thus, adjusting the parameters of the encoder, the parameters of the adjusted first network module, and the parameters of the second network module according to loss value #2 determined from N blurred images and N predicted clear images is to make the loss value #2 corresponding to the N blurred images as small as possible to better meet the requirements of image deblurring.

[0211] It should be understood that the Figure 12 illustrated training method is only illustrative and does not impose any limitation on the training method provided in this application. For example, a trained image segmentation model can also be used to process N training images to obtain N motion mask images corresponding to the N training images.

[0212] In the embodiments of this application, the training samples for the electronic device to train the initial deblurring network include not only the training images and the label images corresponding to the training images, but also the motion mask images corresponding to the training images. Since the blur types of the training samples (e.g., global blur type or local blur type) are different, the corresponding motion mask images are also different. In this way, when training the initial deblurring network based on the corresponding training samples, the network can be guided to learn the blur processing ability of the corresponding type, so that the trained global deblurring network has strong global deblurring ability, and the trained local deblurring network has strong local deblurring ability. Different from the traditional training method of only inputting the training images and label images into the initial network model, when the target deblurring network used in this image deblurring method performs training on the initial deblurring network based on the training samples (i.e., training images, label images, and motion mask images), the deblurring accuracy of the trained target deblurring network can be improved. Thereafter, performing deblurring processing on the image to be processed based on the trained target deblurring network can improve the image deblurring effect, thereby better meeting the user's visual experience.

[0213] Figure 13It is a schematic diagram of an image deblurring method provided by an embodiment of the present application. The image deblurring method provided by the embodiment of the present application can be executed by an electronic device. It can be understood that the electronic device can be implemented as software, or a combination of software and hardware. Exemplarily, the electronic device in the embodiment of the present application can be but is not limited to the Figure 2 illustrated electronic device 100. As Figure 13 shown, the image deblurring method provided by the embodiment of the present application includes S1310 to S1340. Next, S1310 to S1340 will be introduced.

[0214] S1310: The electronic device acquires an image to be processed.

[0215] The image to be processed is an image obtained by the camera of the electronic device shooting a photographed object, and no specific limitation is imposed on the image to be processed. For example, the image to be processed can be the Figure 4 partial blurred image shown in (1) of Figure 5 . For example, the image to be processed can be the

[0216] global blurred image shown in (1) of

[0217] In the embodiment of the present application, no specific limitation is imposed on the acquisition method for the electronic device to acquire the image to be processed.

[0218] As an example of the present application, the electronic device acquires the image to be processed, including: the electronic device uses the camera to shoot the photographed object to obtain the image to be processed. In this implementation manner, the electronic device that shoots the photographed object to obtain the image to be processed is the same electronic device as the electronic device that executes the image deblurring method provided by the embodiment of the present application. Figure 3 The image to be processed in the above implementation manner is the blurred image to be processed in the image deblurring method provided above.

[0219] S1320: The electronic device acquires the jitter amount of the electronic device, where the jitter amount is used to represent the jitter degree of the camera of the electronic device during the period of collecting the image to be processed.

[0220] As an example of the present application, the electronic device acquires the jitter amount of the electronic device, including: the electronic device acquires angular acceleration data, where the angular acceleration data is used to represent the pose data of the electronic device during the period when the camera collects the image to be processed; the electronic device determines the jitter amount based on the angular acceleration data.

[0221] In one example, the gyroscope sensor in the electronic device can detect and record the pose data of the electronic device at different times. In this case, the above angular acceleration data can be obtained by the electronic device from the gyroscope sensor located in the electronic device.

[0222] In the embodiments of the present application, the implementation manner of determining the jitter amount based on the angular acceleration data for the electronic device is not specifically limited.

[0223] In one example, the angular acceleration data in the above implementation manner includes a plurality of angular accelerations, where the plurality of angular accelerations correspond to a plurality of target points in the image to be processed. Each angular acceleration is the pose data of the electronic device when the corresponding target point is captured by the camera. The plurality of target points correspond to a plurality of first two-dimensional positions, and the position of each target point in the image to be processed is the corresponding first two-dimensional position. Moreover, the electronic device determines the jitter amount based on the angular acceleration data, including: the electronic device obtains a plurality of second two-dimensional positions based on the plurality of angular accelerations and the plurality of first two-dimensional positions; the electronic device obtains a plurality of blur values corresponding to the plurality of target points according to the plurality of second two-dimensional positions and the plurality of first two-dimensional positions; the first electronic device determines the jitter amount according to the plurality of blur values.

[0224] In one example, the plurality of target points correspond one-to-one to a plurality of image blocks included in the image to be processed, and each target point is a point in the corresponding image block. For example, each target point is the center point of the corresponding image block. For example, each target point is the lower left corner point of the corresponding image block.

[0225] For example, the image to be processed can be Figure 6 the shown partial blurred image. The partial blurred image is divided into P = 18 image blocks by multiple dotted lines. The shape of each image block is rectangular, and the center point of each image block is the center point of each image block.

[0226] The step in which the above electronic device determines the jitter amount according to the plurality of blur values can, for example, include the following steps: when the smallest blur value among the plurality of blur values is less than a preset threshold, it is determined that the jitter amount satisfies a first jitter degree, where the first jitter degree is less than the preset jitter degree; when the smallest blur value among the plurality of blur values is greater than or equal to the preset threshold, it is determined that the jitter amount satisfies a second jitter degree, where the second jitter degree is greater than the first jitter degree.

[0227] The presentation forms of the first jitter degree and the second jitter degree are not specifically limited. For example, at least one of the first jitter degree and the second jitter degree can be a specific jitter degree. For example, at least one of the first jitter degree and the second jitter degree can be a jitter degree within a preset range, and this is not specifically limited.

[0228] Exemplarily, the multiple target points in the above implementation are the P target points in the image deblurring method provided above Figure 3 and the angular acceleration data is the P first three-dimensional positions in the image deblurring method provided above Figure 3 and the multiple first two-dimensional positions are the P first two-dimensional positions in the method provided above Figure 3 and the multiple second two-dimensional positions are the P second two-dimensional positions in the method provided above Figure 3 .

[0229] There is no specific limitation on the execution order of S1320 and S1310. For example, S1320 can be executed first, and then S1310. For example, S1310 can be executed first, and then S1320.

[0230] S1330: The electronic device determines a target deblurring strategy according to the amount of jitter, where the target deblurring strategy matches the degree of blurring of the image to be processed.

[0231] That the target deblurring strategy matches the degree of blurring of the image to be processed means that the image obtained after the target deblurring strategy performs deblurring on the image to be processed is a clear image, and the clear image can meet the user's deblurring requirements.

[0232] For example, when the image to be processed is a globally blurred image, the target deblurring strategy is a global deblurring strategy. For example, when the image to be processed is a locally blurred image, the target deblurring strategy is a local deblurring strategy.

[0233] As an example of the present application, the target deblurring strategy includes a local deblurring strategy or a global deblurring strategy. The electronic device determines the target deblurring strategy according to the amount of jitter, including: when the amount of jitter meets the first jitter degree, determining the target deblurring strategy as a local deblurring strategy; when the amount of jitter meets the second jitter degree, determining the target deblurring strategy as a global deblurring strategy, where the second jitter degree is greater than the first jitter degree.

[0234] For the relevant descriptions of the first jitter degree and the second jitter degree, reference can be made to the description in S1320 above, which will not be elaborated here.

[0235] Exemplarily, the target deblurring strategy in the above implementation is the strategy for performing deblurring processing on the blurred image to be processed by using a global deblurring network or a local deblurring network in the image deblurring method provided above. The specific implementation process of the above implementation can refer to S350 to S380 in the embodiment shown above Figure 3 and will not be elaborated here. Figure 3

[0236] ​In this way, the electronic device determines a deblurring strategy to be executed on the image to be processed according to the degree of jitter of the electronic device during the period when the image to be processed is captured, so as to ensure that the determined target deblurring strategy matches the image to be processed. Specifically, when the degree of jitter of the electronic device is small during the period when the image to be processed is captured, the deblurring strategy executed on the image to be processed is a local deblurring strategy. When the degree of jitter of the electronic device is large during the period when the image to be processed is captured, the deblurring strategy executed on the image to be processed is a global deblurring strategy.

[0237] S1340: The electronic device performs deblurring processing on the image to be processed according to the target deblurring strategy to obtain a clear image.

[0238] After the electronic device determines the target deblurring strategy according to the jitter amount, for example, the target deblurring strategy is a global deblurring strategy or a local deblurring strategy, the implementation manner of the electronic device performing deblurring processing on the image to be processed according to the global deblurring strategy or the local deblurring strategy is not specifically limited.

[0239] As an example of the present application, the electronic device performs deblurring processing on the image to be processed according to the target deblurring strategy to obtain a clear image, including: the electronic device performs deblurring processing on the image to be processed by using a target deblurring network to obtain a clear image, where the target deblurring network is a neural network model obtained by training an initial deblurring network by using training samples, the training samples include training images, label images, and motion mask images, the training images are blurred images, the label images are clear images corresponding to the training images, and the motion mask images are images including the characteristics of the blurred areas of the training images.

[0240] In an example, the initial deblurring network in the above steps includes an encoder and a decoder, where the decoder includes a first network module and a second network module, and the target deblurring network is specifically a neural network model obtained by training the encoder, the adjusted first network module, and the second network module according to a first loss value, where the first loss value is determined based on the difference between the predicted image and the training image, the predicted image is an image obtained by the second network module performing decoding processing on a second feature image, the second feature image is an image obtained by the adjusted first network module performing feature extraction on a first feature image, the adjusted first network module is obtained by adjusting the parameters of the first network module according to a second loss value, the first feature image is an image obtained by the encoder performing feature extraction on the training image, and the second loss value is determined based on the difference between the first feature image and the motion mask image.

[0241] Specifically, in the case where the training image in the above steps is a locally blurred image, a partial area of the training image is blurred content, the motion mask image includes the features of the partial area, and the target deblurring network is a local deblurring network.

[0242] For example, the training image can be Figure 10 the image shown in (1) of Figure 10 and the motion mask image can be the image shown in (3) of

[0243] Specifically, in the case where the training image in the above steps is a globally blurred image, the entire area of the training image is blurred content, the motion mask image includes the features of the entire area, and the target deblurring network is a global deblurring network.

[0244] For example, the training image can be Figure 11 the image shown in (1) of Figure 11 and the motion mask image can be the image shown in (3) of

[0245] Exemplarily, the training samples in the above implementation manner are the N training samples included in the training data set of the image deblurring method provided above, the encoder is the encoder provided above, the first network module in the decoder is the first network module provided above, the second network module in the decoder is the second network module provided above, the first loss value is the loss value #2 in the image deblurring method provided above, the second loss value is the loss value #1 in the method provided above, the N first feature images are the N feature images #1 in the method provided above, the N second feature images are the N feature images #2 in the method provided above, and the specific implementation process of training the initial deblurring network to obtain the target deblurring network in the above implementation manner can refer to the training method provided above, which will not be elaborated here. Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 12 Figure 12

[0246] As another example of the present application, an electronic device performs deblurring processing on an image to be processed according to a target deblurring strategy to obtain a clear image. Exemplarily, it may include the following steps: The electronic device uses a target deblurring network to perform deblurring processing on the image to be processed to obtain a clear image, where the target deblurring network is a neural network model obtained by training an initial deblurring network using training samples, the training samples include a training image and a label image, the training image is a blurred image, and the label image is the clear image corresponding to the training image.

[0247] In the above steps, the initial deblurring network includes an encoder and a decoder. The target deblurring network is a neural network model obtained by training the initial deblurring network using the difference between the labeled image and the predicted image. The predicted image is obtained by the decoder decoding the feature image acquired from the encoder, and the feature image acquired from the encoder is the image obtained by the encoder extracting features from the training image.

[0248] As an example of the present application, before the electronic device acquires the image to be processed, the method further includes: displaying a shooting interface; the electronic device acquiring the image to be processed includes: in response to a shooting operation on the shooting interface, the electronic device obtaining the image to be processed through a camera; after the electronic device performs deblurring processing on the image to be processed according to the target deblurring strategy to obtain a clear image, the method further includes: the electronic device displaying an interface including the clear image.

[0249] In this way, in response to the user's shooting operation, the electronic device acquires the image to be processed, then the electronic device processes the image to be processed using the target deblurring strategy, and finally the electronic device presents an interface including the clear image to the user. The process of the electronic device performing deblurring processing on the image to be processed is executed in the background, that is, the user is not aware of this process, which can improve the user's visual experience.

[0250] It should be understood that the above Figure 13 illustrated image deblurring method is only illustrative and does not constitute any limitation to the image deblurring method provided by the present application. It should be noted that in the above Figure 13 illustrated image deblurring method, the image to be processed obtained by the electronic device shooting the object to be photographed, and the electronic device executing the image deblurring method are taken as examples for illustration. Optionally, the electronic device executing the image deblurring method and the electronic device obtaining the image to be processed by shooting the object to be photographed are two different electronic devices. In this case, after the electronic device obtaining the image to be processed by shooting the object to be photographed obtains the image to be processed, it can transmit the image to be processed to the electronic device executing the image deblurring method so that the electronic device executing the image deblurring method can acquire the image to be processed.

[0251] In an embodiment of the present application, after the electronic device acquires the image to be processed, the electronic device first determines a strategy for performing deblurring on the image to be processed (the target deblurring strategy) according to the amount of jitter. Since the amount of jitter represents the degree of jitter of the electronic device during the period when the camera of the electronic device acquires the image to be processed, therefore, the target deblurring strategy determined based on the amount of jitter matches the degree of blurring of the image to be processed. Then, the electronic device performs deblurring processing on the image to be processed according to the target deblurring strategy determined based on the amount of jitter to obtain a clear image. This method avoids the problem in the traditional technology that the deblurring processing strategy does not match the degree of blurring of the image to be processed when the electronic device performs deblurring processing on the acquired image to be processed. Therefore, this method can improve the image deblurring effect and thus better meet the user's visual experience.

[0252] In the present application, the application form of the image deblurring method provided above in the present application on the terminal device is not specifically limited. Below, taking the mobile phone applying the image deblurring method provided in the present application as an example, the schematic diagram of the user interface of the mobile phone is described.

[0253] As an example, in response to the user triggering the shooting function of the mobile phone, the image deblurring method provided in the present application is called to perform deblurring processing on the blurred image obtained by the mobile phone shooting. In this case, the user is not aware of the process of the mobile phone performing image deblurring.

[0254] For example, taking Figure 14 Figure (1) as an example, the photographed objects presented on the shooting interface S4 of the mobile phone include people and buildings. In response to the user triggering the camera control 10 of the mobile phone, then, the mobile phone can photograph the photographed objects to obtain an image. Although the mobile phone shakes during the shooting process, since the mobile phone performs the image deblurring method provided in the present application on the acquired blurred image, therefore, the image finally obtained by the mobile phone after photographing the photographed objects is a clear image. For example, Figure 14 the clear image shown in Figure (2) of

[0255] As another example, in response to the user triggering the deblurring control 20 of the mobile phone's album interface, the image deblurring method provided in the present application is called to perform deblurring processing on the blurred image stored in the album. In this case, the user can perceive the process of the mobile phone performing deblurring on the blurred image.

[0256] For example, taking Figure 15Taking the (1) figure in [reference] as an example, the objects to be photographed presented on the shooting interface S7 of the mobile phone include a person and a building. In response to the user triggering the camera control 10 of the mobile phone, then the mobile phone can photograph the object to be photographed to obtain an image. During the shooting process, since the mobile phone shakes, therefore, the image finally obtained after the mobile phone photographs the object to be photographed is a blurred image. For example, Figure 15 The blurred image shown in the photo album interface S8 of the mobile phone shown in the (2) figure in [reference]. After that, the user can perform a de-blurring process on the blurred image in S8 by triggering the de-blurring control 20 in the photo album interface S8 to obtain a clear image. Please refer to Figure 15 The clear image shown in the camera interface S9 of the mobile phone shown in the (2) figure in [reference].

[0257] This application also provides a computer program product, which when executed by a processor implements the image de-blurring method described in any method embodiment of this application.

[0258] This computer program product can be stored in a memory. For example, it is a program that is finally converted into an executable target file that can be executed by a processor after processes such as preprocessing, compilation, assembly, and linking.

[0259] This application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, it implements the image de-blurring method described in any method embodiment of this application. This computer program can be a high-level language program or an executable target program.

[0260] In this application, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following items" or its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0261] It should be understood that in various embodiments of this application, the magnitudes of the sequence numbers of the above processes do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this application.

[0262] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0263] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0264] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0265] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0266] In addition, the functional units in each embodiment of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0267] As described above, the above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. An image deblurring method, characterized in that, Applied to an electronic device, the method includes: Obtain an image to be processed; Obtain the jitter amount of the electronic device, where the jitter amount is used to represent the jitter degree of the electronic device during the period when the camera of the electronic device captures the image to be processed; Determine a target deblurring strategy according to the jitter amount, where the target deblurring strategy matches the blur degree of the image to be processed; Perform deblurring processing on the image to be processed according to the target deblurring strategy to obtain a clear image.

2. The method according to claim 1, wherein The target deblurring strategy includes a local deblurring strategy or a global deblurring strategy, and determining the target deblurring strategy according to the jitter amount includes: When the jitter amount meets the first jitter degree, determine that the target deblurring strategy is the local deblurring strategy; When the jitter amount meets the second jitter degree, determine that the target deblurring strategy is the global deblurring strategy, where the second jitter degree is greater than the first jitter degree.

3. The method according to claim 1 or 2, characterized in that, Performing deblurring processing on the image to be processed according to the target deblurring strategy to obtain a clear image includes: Performing deblurring processing on the image to be processed by using a target deblurring network to obtain a clear image, where the target deblurring network is a neural network model obtained by training an initial deblurring network by using training samples, the training samples include training images, label images, and motion mask images, the training images are blurred images, the label images are clear images corresponding to the training images, and the motion mask images are images including features of the blurred regions of the training images.

4. The method according to claim 3, characterized in that, The initial deblurring network includes an encoder and a decoder, where the decoder includes a first network module and a second network module, and the target deblurring network is specifically a neural network model obtained by training the encoder, the adjusted first network module, and the second network module according to a first loss value, where the first loss value is determined based on the difference between a predicted image and the training image, the predicted image is an image obtained by the second network module performing decoding processing on a second feature image, the second feature image is an image obtained by the adjusted first network module performing feature extraction on a first feature image, the adjusted first network module adjusts the parameters of the first network module according to a second loss value, the first feature image is an image obtained by the encoder performing feature extraction on the training image, and the second loss value is determined based on the difference between the first feature image and the motion mask image.

5. The method according to claim 3 or 4, wherein When the training image is a globally blurred image, the entire region of the training image is blurred content, the motion mask image includes features of the entire region, and the target deblurring network is a global deblurring network; When the training image is a locally blurred image, a partial area of the training image is blurred content, the motion mask image includes features of the partial area, and the target deblurring network is a local deblurring network.

6. The method according to any one of claims 1 to 5, characterized in that The obtaining of the jitter amount of the electronic device includes: Obtaining angular acceleration data, where the angular acceleration data is used to represent the pose data of the electronic device during the period when the camera captures the image to be processed; Determining the jitter amount based on the angular acceleration data.

7. The method according to claim 6, wherein The angular acceleration data includes a plurality of angular accelerations, where the plurality of angular accelerations correspond to a plurality of target points in the image to be processed, each angular acceleration is the pose data of the electronic device when the camera captures the corresponding target point, the plurality of target points correspond to a plurality of first two-dimensional positions, the position of each target point in the image to be processed is the corresponding first two-dimensional position, and the determining the jitter amount based on the angular acceleration data includes: Obtaining a plurality of second two-dimensional positions based on the plurality of angular accelerations and the plurality of first two-dimensional positions; Obtaining a plurality of blur values corresponding to the plurality of target points according to the plurality of second two-dimensional positions and the plurality of first two-dimensional positions; Determining the jitter amount according to the plurality of blur values.

8. The method according to claim 7, wherein The determining the jitter amount according to the plurality of blur values includes: Determining that the jitter amount satisfies a first jitter degree when the minimum blur value among the plurality of blur values is less than a preset threshold; Determining that the jitter amount satisfies a second jitter degree when the minimum blur value among the plurality of blur values is greater than or equal to the preset threshold, where the second jitter degree is greater than the first jitter degree.

9. The method according to any one of claims 1 to 8, characterized in that Before obtaining the image to be processed, the method further includes: Displaying a shooting interface; The obtaining of the image to be processed includes: In response to a shooting operation on the shooting interface, obtaining the image to be processed through the camera; After performing deblurring processing on the image to be processed according to the target deblurring strategy to obtain a clear image, further including: Displaying an interface including the clear image.

10. An image deblurring device, characterized in that, Applied to an electronic device, the image deblurring device includes a processing unit, and the processing unit is used to: Obtain an image to be processed; Obtain the jitter amount of the electronic device, where the jitter amount is used to represent the jitter degree of the electronic device during the period when the camera of the electronic device captures the image to be processed; Determine a target deblurring strategy according to the jitter amount, where the target deblurring strategy matches the blur degree of the image to be processed; Perform deblurring processing on the image to be processed according to the target deblurring strategy to obtain a clear image.

11. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the processor executes the image deblurring method according to any one of claims 1 to 9.

12. A chip system, characterized in that, The chip system includes a processor, and when the processor executes an instruction, the processor executes the image deblurring method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it causes the processor to execute the image deblurring method according to any one of claims 1 to 9.

Images