Image processing and model training method and electronic device

Abstract

Embodiments of the present application provide an image processing method and a model training method and an electronic device. The image processing method comprises: in an alternating current light environment, capturing a first image and a second image for a moving scene; wherein the exposure time of the first image is longer than the exposure time of the second image, and the exposure time of the second image is less than the energy period of the alternating current; then, inputting the first image and the second image into a trained image processing model, and outputting a target image by the image processing model. In this way, in the alternating current light environment, the moving scene can be captured to obtain an image without stripes.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of data processing, and in particular, to an image processing method and a model training method and an electronic device. BACKGROUND

[0002] The alternating current light environment is a high-frequency shooting environment. When shooting in the alternating current light environment, if the exposure time of the camera is lower than the energy cycle of the alternating current, banding phenomenon will occur, that is, the captured image will have alternating light and dark stripes. SUMMARY

[0003] To solve the above technical problems, the present application provides an image processing method and a model training method and an electronic device. In the image processing method, the image captured in the alternating current light environment for the moving scene can be processed to obtain a stripe-free image.

[0004] In a first aspect, the embodiments of the present application provide an image processing method, which comprises: in an alternating current light environment, capturing a first image and a second image for a moving scene; wherein the exposure time of the first image is greater than the exposure time of the second image, and the exposure time of the second image is less than the energy cycle of the alternating current; then inputting the first image and the second image into a trained image processing model, and outputting a target image from the image processing model. In this way, the image captured in the alternating current light environment for the moving scene can be obtained without stripes.

[0005] In addition, the low-light area in the second image is underexposed and loses details, and the high-light area has details; the high-light area in the first image is overexposed and loses details, and the low-light area has details; the trained image processing model can fuse the low-light area in the first image and the high-light area in the second image, so that the target image obtained by fusion has details in each area, and the brightness dynamic range of the target image (that is, the difference between light and dark of the image) is expandable.

[0006] Illustratively, the moving scene refers to a scene in which the relative motion speed between the object to be photographed and the image acquisition device is greater than a relative motion speed threshold.

[0007] Illustratively, the first image and the second image can be one of a RAW (unprocessed) image, an RGB (Red Green Blue) image, and a YUV (Y represents brightness (Luminance, Luma), and U and V are chroma, chrominance (Chrominance, Chroma)) image.

[0008] For example, the first image can be an image that does not satisfy the definition condition. For example, the definition condition can be that the edge width of the photographed object is less than or equal to a preset width threshold, which can be set as 3 pixels or the like according to requirements, and the application does not limit this.

[0009] For example, the edge width of the moving photographed object in the first image is greater than the preset width threshold, and the edge width of the stationary photographed object in the first image is less than or equal to the preset width threshold.

[0010] For example, the edge width of all photographed objects in the first image is greater than the preset width threshold.

[0011] According to the first aspect, the method further comprises: detecting the motion speed of the photographed object and the shaking speed of the image acquisition device; and detecting whether the current shooting scene is a motion scene according to the motion speed and the shaking speed.

[0012] For example, when it is determined according to the motion speed of the photographed object and the shaking speed of the image acquisition device that the relative motion speed of the photographed object and the image acquisition device is greater than a relative motion speed threshold, it can be determined that the current shooting scene is a motion scene.

[0013] For example, when it is determined according to the motion speed of the photographed object and the shaking speed of the image acquisition device that the relative motion speed of the photographed object and the image acquisition device is less than or equal to a relative motion speed threshold, it can be determined that the current shooting scene is not a motion scene.

[0014] According to the first aspect, or any one of the implementations of the first aspect, the first image and the second image are shot for the motion scene, comprising: when it is determined according to the motion speed and the shaking speed that the relative motion speed of the photographed object and the image acquisition device is less than or equal to a preset relative motion speed, the first image is shot for the motion scene according to a first preset exposure time, and the second image is shot for the motion scene according to a second preset exposure time; wherein the first preset exposure time is N1 times of the AC energy period, the second preset exposure time is M1 times of the energy period, N1 is a positive integer, and M1 is a decimal between 0 and 1. In this way, the first image shot is stripe-free, and the second image is striped.

[0015] In an implementation form of the first aspect or any implementation form of the first aspect, the first image and the second image are captured for a motion scene, including: when the relative motion speed between the object and the image capture device is determined to be less than or equal to a preset relative motion speed according to the motion speed and the shaking speed, the first image is captured for the motion scene according to a third preset exposure time, and the second image is captured for the motion scene according to a fourth preset exposure time; the third preset exposure time is N2 times of an AC energy period, the fourth preset exposure time is M2 times of the energy period, N2 is a decimal greater than 1, and M2 is a decimal between 0 and 1. In this way, the captured first image is striped, and the second image is also striped.

[0016] In an implementation form of the first aspect or any implementation form of the first aspect, the first image and the second image are captured for a motion scene, including: when the relative motion speed between the object and the image capture device is determined to be greater than a preset relative motion speed according to the motion speed and the shaking speed, the first image is captured for the motion scene according to a fifth preset exposure time, and the second image is captured for the motion scene according to a sixth preset exposure time; the fifth preset exposure time is N3 times of an AC energy period, the sixth preset exposure time is M3 times of the energy period, N3 and M3 are decimals between 0 and 1, and N3 is greater than M3. In this way, the captured first image is striped, and the second image is also striped.

[0017] In an implementation form of the first aspect or any implementation form of the first aspect, the image processing model is trained based on a first training image and a second training image, the second training image is striped, and the brightness of the first training image is greater than the brightness of the second training image. In this way, the image processing model can be used to process the first image captured according to the first preset exposure time and the second image captured according to the second preset exposure time, to obtain a target image without stripes.

[0018] For example, the image processing model is a first image processing model.

[0019] In an implementation form of the first aspect or any implementation form of the first aspect, the method further includes: obtaining a video sequence, the video sequence being obtained by capturing for a motion scene, the video sequence including images of high dynamic range imaging, and the images in the video sequence being non-striped; selecting one image from the video sequence as a label image; performing frame interpolation on the video sequence, selecting K images from the video sequence after frame interpolation based on the label image for fusion to obtain a first training image; selecting one image from the video sequence after frame interpolation as a base image, reducing the brightness of the base image, and adding stripes to the base image after the brightness is reduced to obtain a second training image. In this way, the first training image and the second training image can be constructed.

[0020] According to a first aspect, or any possible implementation mode of the first aspect, the image processing model is trained based on a first training image and a second training image, the first training image and the second training image both have stripes, the stripe intensity of the first training image is less than the stripe intensity of the second training image, and the brightness of the first training image is greater than the brightness of the second training image. In this way, the image processing model can be used to process the first image captured according to the third preset exposure time and the second image captured according to the fourth preset exposure time, or to process the first image captured according to the fifth preset exposure time and the second image captured according to the sixth preset exposure time, to obtain a target image without stripes.

[0021] Exemplarily, the image processing model is a second image processing model.

[0022] According to the first aspect, or any possible implementation mode of the first aspect, the method further includes: obtaining a video sequence, the video sequence being obtained by capturing a moving scene, the video sequence including images of high dynamic range imaging, and the images in the video sequence being stripe-free; selecting one image from the video sequence as a label image; performing frame interpolation on the video sequence, selecting K images from the video sequence after frame interpolation based on the label image for fusion, and adding stripes meeting a weak stripe condition to the image obtained by fusion to obtain a first training image; selecting one image from the video sequence after frame interpolation as a base image, reducing the brightness of the base image, and adding stripes meeting a strong stripe condition to the base image after brightness reduction to obtain a second training image. In this way, the first training image and the second training image can be constructed.

[0023] According to the first aspect, or any possible implementation mode of the first aspect, the first image and the second image are images captured by the same image acquisition device, and the time interval between capturing the first image and the second image is less than a preset time length; or, the first image and the second image are images captured by different image acquisition devices, and the starting capturing time of the first image and the second image is the same.

[0024] Exemplarily, the first image and the second image are two images continuously captured by the same image acquisition device.

[0025] In a second aspect, the embodiments of the present application provide a model training method, which comprises: generating training data, the training data comprising: a first training image, a second training image and a label image, the second training image having stripes, and the brightness of the first training image being greater than the brightness of the second training image. Then, inputting the training data into an image processing model, performing forward calculation on the first training image and the second training image in the training data by the image processing model, and outputting a fused image; subsequently, calculating a loss function value based on the fused image and the label image in the training data, and adjusting model parameters of the image processing model according to the loss function value. In this way, the image processing model can learn the stripe removal capability and the image fusion capability.

[0026] According to the second aspect, the training data is generated, comprising: first, obtaining a video sequence, the video sequence being obtained by shooting a moving scene, the video sequence comprising images of high dynamic range imaging, and the images in the video sequence being stripe-free. Then, selecting one frame of image from the video sequence as a label image; subsequently, performing frame interpolation on the video sequence, selecting K frames of image from the frame-interpolated video sequence based on the label image for fusion, and obtaining a first training image; and then selecting one frame of image from the frame-interpolated video sequence as a base image, reducing the brightness of the base image, and adding stripes to the base image with reduced brightness to obtain a second training image. In this way, the first training image which does not meet the definition of clarity but is stripe-free, the second training image which meets the definition of clarity but has stripes, and the label image which is clear and stripe-free can be generated.

[0027] According to the second aspect, or any one of the implementation manners of the second aspect above, the training data is generated, comprising: first, obtaining a video sequence, the video sequence being obtained by shooting a moving scene, the video sequence comprising images of high dynamic range imaging, and the images in the video sequence being stripe-free. Then, selecting one frame of image from the video sequence as a label image; subsequently, performing frame interpolation on the video sequence, selecting K frames of image from the frame-interpolated video sequence based on the label image for fusion, and adding stripes to the fused image to meet a weak stripe condition to obtain a first training image; and then selecting one frame of image from the frame-interpolated video sequence as a base image, reducing the brightness of the base image, and adding stripes to the base image with reduced brightness to meet a strong stripe condition to obtain a second training image. In this way, the first training image which does not meet the definition of clarity and has stripes, the second training image which meets the definition of clarity but has stripes, and the label image which is clear and stripe-free can be generated.

[0028] According to the second aspect, or any one of the implementation manners of the second aspect above,

[0029] The weak stripe condition comprises: the stripe intensity being less than an intensity threshold;

[0030] The strong stripe condition includes that the stripe intensity is greater than or equal to an intensity threshold.

[0031] In a third aspect, an embodiment of the present application provides an image acquisition device, which can execute the image processing method in the first aspect or any possible implementation manner of the first aspect.

[0032] The third aspect and any one of the implementation manners of the third aspect correspond to the first aspect and any one of the implementation manners of the first aspect respectively. For details, refer to the technical effects of the first aspect and any one of the implementation manners of the first aspect, which will not be described here again.

[0033] In a fourth aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, the memory being coupled with the processor; the memory stores program instructions, when the program instructions are executed by the processor, the electronic device executes the image processing method in the first aspect or any possible implementation manner of the first aspect.

[0034] The fourth aspect and any one of the implementation manners of the fourth aspect correspond to the first aspect and any one of the implementation manners of the first aspect respectively. For details, refer to the technical effects of the first aspect and any one of the implementation manners of the first aspect, which will not be described here again.

[0035] In a fifth aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, the memory being coupled with the processor; the memory stores program instructions, when the program instructions are executed by the processor, the electronic device executes the model training method in the second aspect or any possible implementation manner of the second aspect.

[0036] The fifth aspect and any one of the implementation manners of the fifth aspect correspond to the second aspect and any one of the implementation manners of the second aspect respectively. For details, refer to the technical effects of the second aspect and any one of the implementation manners of the second aspect, which will not be described here again.

[0037] In a sixth aspect, an embodiment of the present application provides a chip, including one or more interface circuits and one or more processors; the interface circuit is used for receiving a signal from a memory of an electronic device and sending a signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the image processing method in the first aspect or any possible implementation manner of the first aspect.

[0038] The sixth aspect and any possible implementation manner of the sixth aspect correspond to the first aspect and any possible implementation manner of the first aspect respectively. The technical effects of the sixth aspect and any possible implementation manner of the sixth aspect correspond to the technical effects of the first aspect and any possible implementation manner of the first aspect, which will not be described herein.

[0039] In a seventh aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send a signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the model training method in the second aspect or any possible implementation manner of the second aspect.

[0040] The seventh aspect and any possible implementation manner of the seventh aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively. The technical effects of the seventh aspect and any possible implementation manner of the seventh aspect correspond to the technical effects of the second aspect and any possible implementation manner of the second aspect, which will not be described herein.

[0041] In an eighth aspect, an embodiment of the present application provides a computer storage medium, the computer readable storage medium stores a computer program, when the computer program runs on a computer or a processor, the computer or the processor executes the image processing method in the first aspect or any possible implementation manner of the first aspect.

[0042] The eighth aspect and any possible implementation manner of the eighth aspect correspond to the first aspect and any possible implementation manner of the first aspect respectively. The technical effects of the eighth aspect and any possible implementation manner of the eighth aspect correspond to the technical effects of the first aspect and any possible implementation manner of the first aspect, which will not be described herein.

[0043] In a ninth aspect, an embodiment of the present application provides a computer storage medium, the computer readable storage medium stores a computer program, when the computer program runs on a computer or a processor, the computer or the processor executes the model training method in the second aspect or any possible implementation manner of the second aspect.

[0044] The ninth aspect and any possible implementation manner of the ninth aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively. The technical effects of the ninth aspect and any possible implementation manner of the ninth aspect correspond to the technical effects of the second aspect and any possible implementation manner of the second aspect, which will not be described herein.

[0045] In a tenth aspect, an embodiment of the present application provides a computer program product, which contains a software program. When the software program is executed by a computer or a processor, the steps of the image processing method in the first aspect or any possible implementation manner of the first aspect are executed.

[0046] The tenth aspect and any possible implementation manner of the tenth aspect correspond to the first aspect and any possible implementation manner of the first aspect respectively. The technical effects of the tenth aspect and any possible implementation manner of the tenth aspect correspond to the technical effects of the first aspect and any possible implementation manner of the first aspect respectively, which will not be described here.

[0047] In an eleventh aspect, an embodiment of the present application provides a computer program product, which contains a software program. When the software program is executed by a computer or a processor, the steps of the model training method in the second aspect or any possible implementation manner of the second aspect are executed.

[0048] The eleventh aspect and any possible implementation manner of the eleventh aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively. The technical effects of the eleventh aspect and any possible implementation manner of the eleventh aspect correspond to the technical effects of the second aspect and any possible implementation manner of the second aspect respectively, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 An application scenario schematic diagram is exemplarily shown;

[0050] Figure 2 An application scenario schematic diagram is exemplarily shown;

[0051] Figure 3 A training process schematic diagram is exemplarily shown;

[0052] Figure 4 A data generation process schematic diagram is exemplarily shown;

[0053] Figure 5 A training process schematic diagram is exemplarily shown;

[0054] Figure 6 A data generation process schematic diagram is exemplarily shown;

[0055] Figure 7 A processing flow schematic diagram is exemplarily shown;

[0056] Figure 8 An image processing flow schematic diagram is exemplarily shown;

[0057] Figure 9a An image processing process schematic diagram is exemplarily shown;

[0058] Figure 9b An exemplary schematic diagram of an image processing procedure is shown.

[0059] Figure 9c An exemplary schematic diagram of an image processing procedure is shown.

[0060] Figure 9d An exemplary schematic diagram of an image processing procedure is shown.

[0061] Figure 10 An exemplary schematic diagram of a device is shown. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be clearly and completely described in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0063] The term "and / or" in the present application is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone.

[0064] The terms "first" and "second" and the like in the description and claims of the embodiments of the present application are used to distinguish different objects, and are not used to describe the specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, and are not used to describe the specific order of the target objects.

[0065] In the embodiments of the present application, the words such as "exemplary" or "for example" are used to mean an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words such as "exemplary" or "for example" are intended to present the relevant concept in a specific manner.

[0066] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more. For example, a plurality of processing units means two or more processing units; a plurality of systems means two or more systems.

[0067] Figure 1 An exemplary schematic diagram of an application scenario is shown.

[0068] Exemplary, Figure 1 (1)~Figure 1 The light fixture in (4) is connected to AC power.

[0069] Referring to Figure 1 (1), by way of example, one application scenario of the present application is to capture a scene of a moving ping pong ball and / or a moving user in an AC light environment with the image capture device stationary.

[0070] Referring to Figure 1 (2), by way of example, one application scenario of the present application is to capture a scene of a moving tennis ball and / or a moving user in an AC light environment with the image capture device stationary.

[0071] Referring to Figure 1 (3), by way of example, one application scenario of the present application is to capture a scene of a moving volleyball and / or a moving user in an AC light environment with the image capture device stationary.

[0072] Referring to Figure 1 (4), by way of example, one application scenario of the present application is to capture a scene of a running user in an AC light environment with the image capture device stationary.

[0073] It should be understood that Figure 1 By way of example only, one application scenario of the present application can include capturing a scene of various moving objects in an AC light environment with the image capture device stationary.

[0074] Figure 2 A schematic diagram of the application scenario shown by way of example.

[0075] By way of example, Figure 2 (1)- Figure 2 The light fixture in (4) is connected to AC power, Figure 2 (1)- Figure 2 The image capture device in (4) is subject to jitter.

[0076] Referring to Figure 2 (1), by way of example, one application scenario of the present application is to capture a scene of a moving ping pong ball and / or a moving user in an AC light environment with the image capture device subject to jitter.

[0077] Referring to Figure 2 (2), by way of example, one application scenario of the present application is to capture a scene of a moving tennis ball and / or a moving user in an AC light environment with the image capture device subject to jitter.

[0078] Referring to Figure 2(3) For example, one application scenario of the present application is to capture the scene of a moving volleyball and / or a moving user in a case where the image acquisition device is dithered in an AC light environment.

[0079] Referring to Figure 2 (4) For example, one application scenario of the present application is to capture the scene of a house in a case where the image acquisition device is dithered in an AC light environment.

[0080] It should be understood that Figure 3 For example, one application scenario of the present application is to capture the scene of a moving volleyball and / or a moving user in a case where the image acquisition device is dithered in an AC light environment.

[0081] For example, the scene in which the relative motion speed between the object to be captured and the image acquisition device is greater than a relative motion speed threshold value can be referred to as a moving scene. The relative motion speed threshold value can be set as required, and the present application does not limit this. That is, the application scenario of the present application can include the scene of capturing various moving scenes in an AC light environment.

[0082] For example, the image acquisition device can be all devices in which the image sensor works in a rolling shutter mode. Alternatively, the image acquisition device can include all devices in which the image sensor is a CMOS (Complementary Metal Oxide Semiconductor). For example, a mobile phone, a pad, a camera, a camcorder, and the like, and the present application does not limit this.

[0083] For example, the frequency of AC power is different in different regions, and the present application does not limit the frequency of AC power.

[0084] For example, the present application takes into account that in an AC light environment, when a moving scene is captured, the image captured by a short exposure time has stripes and the low-light area loses details but the high-light area has details, and the image captured by a long exposure time has the high-light area losing details but the low-light area has details; therefore, the present application can pre-train an image processing model having a stripe removal capability and an image fusion capability, and then capture one image with a long exposure time and one image with a short exposure time in actual application, and then process them by using the pre-trained image processing model to obtain an image in which each area has details and no stripes.

[0085] For example, when shooting a sports scene, the image taken with a long exposure time may be a clear image or a blurred image. Then, the image processing model with the deblurring, de-streaking and image fusion capabilities can be pre-trained. In actual application, one image with a long exposure time and one image with a short exposure time are taken, and the pre-trained image processing model is used for processing to obtain an image with details, clarity and no streaks in each region. It should be noted that even if the image with a long exposure time is clear, the image processing model with the deblurring, de-streaking and image fusion capabilities can be used to process one image with a long exposure time and one image with a short exposure time to obtain an image with details, clarity and no streaks in each region. The application is described by taking the training of the image processing model with the deblurring, de-streaking and image fusion capabilities as an example.

[0086] For example, when the image meets the clarity condition, it can be determined that the image is clear, and when the image does not meet the clarity condition, it can be determined that the image is blurred. The clarity condition can be set as required, and the application does not limit this. For example, the clarity condition can be that the edge width of the photographed object is less than or equal to a preset width threshold, and the preset width threshold can be set as required, such as 3 pixel points, and the application does not limit this.

[0087] In one possible manner, in actual application, the long exposure time can be set as a first preset exposure time T1, and the short exposure time can be set as a second preset exposure time T2. Wherein, T1=N1*T, T is the energy period of alternating current, and N1 is a positive integer. T2=M1*T, M1 is a decimal between 0 and 1. In this case, the image taken according to the long exposure time is blurred but has no streaks (or is clear and has no streaks), and the image taken according to the short exposure time is clear but has streaks.

[0088] For example, the energy period of alternating current refers to the period of alternating current amplitude absolute value change, which is equal to one-half of the alternating current period. For example, the domestic alternating current frequency is 50 Hz, the alternating current period is 20 ms, and the energy period of alternating current is 10 ms.

[0089] For example, the image processing model can include a first image processing model. The first image processing model can be pre-trained, and then used to process the blurred but streak-free image (or clear and streak-free) and the clear but streaky image to obtain an image with details, clarity and no streaks in each region.

[0090] The training process of the first image processing model is described below.

[0091] Figure 4A training process flowchart is shown as an example.

[0092] S301, generating training data, the training data comprising: a first training image, a second training image and a label image.

[0093] Figure 4 A data generation process flowchart is shown as an example.

[0094] For example, the training data can include multiple groups, and each group of training data includes one frame of first training image, one frame of second training image and one frame of label image.

[0095] Referring to Figure 5 For example, the process of generating training data can refer to the following steps S401-S406:

[0096] S401, obtaining a video sequence.

[0097] In one possible manner, the moving object is photographed under the condition that the image acquisition device is stationary, and a video sequence is obtained, wherein each frame of image of the video sequence is clear and stripe-free. For example, different motion processes of the same object can be photographed to obtain multiple video sequences. For example, motion processes of different objects can be photographed to obtain multiple video sequences.

[0098] In one possible manner, the moving object is photographed under the condition that the image acquisition device is dithered, and a video sequence is obtained, wherein each frame of image of the video sequence is clear and stripe-free. For example, different motion processes of the same object can be photographed to obtain multiple video sequences. For example, motion processes of different objects can be photographed to obtain multiple video sequences.

[0099] In one possible manner, the stationary object is photographed under the condition that the image acquisition device is dithered, and a video sequence is obtained, wherein each frame of image of the video sequence is clear and stripe-free. For example, different stationary objects can be photographed to obtain multiple video sequences.

[0100] It should be noted that the video sequence includes high dynamic range imaging images, for example, during the photographing process, the high dynamic range imaging images can be obtained by converting the shooting environment from indoor to outdoor, etc.

[0101] It should be noted that the present embodiment does not limit the motion speed of the object, the dithering speed of the image acquisition device, the exposure time of the image acquisition device and other shooting device parameters, as long as each frame of image in the obtained video sequence is clear and stripe-free.

[0102] It should be understood that the application can obtain a video sequence by shooting various motion scenes; the images in the video sequence are clear and stripe-free, and the video sequence contains images of high dynamic range imaging.

[0103] Then a group of video sequences can be selected from the plurality of groups of video sequences, and training data can be generated based on the selected video sequences, which can refer to S402-S406:

[0104] S402, selecting a label image from the video sequence.

[0105] For example, a frame of image can be selected from the video sequence as the label image.

[0106] For example, a frame of image can be selected from the video sequence as the label image according to a preset rule. The preset rule can be set according to requirements, such as random selection, selection according to order, etc., which is not limited by the application.

[0107] S403, interpolating the video sequence.

[0108] For example, a plurality of frames of images in the video sequence can be fused to generate a first training image that is blurred but stripe-free. For example, in two adjacent frames of images of a video sequence shot at a certain frame rate (such as 30fps, 60fps), the displacement of the moving photographed object is greater than a preset displacement, so that the moving photographed object in the image obtained by fusing multiple frames will have non-continuous ghosting. Therefore, the video sequence can be interpolated first, and then K (K is a positive integer) frames of images can be selected from the interpolated video sequence for fusion to ensure that the ghosting of the moving photographed object in the image after fusing multiple frames is continuous. For example, the preset displacement can be set according to requirements, such as one pixel, which is not limited by the application.

[0109] For example, the video sequence can be interpolated with the goal that the displacement of the moving photographed object in the adjacent two frames of images after interpolation is less than or equal to the preset displacement.

[0110] For example, the interpolation method can include various methods, such as a quadratic interpolation method, a SepConv (sequence convolution) network interpolation method, etc., which is not limited by the application, and the video sequence can be interpolated according to existing mature interpolation methods.

[0111] S404, selecting K frames of images from the interpolated video sequence for fusion.

[0112] For example, the number of frames K of images used for fusion can be determined according to the requirement of the blur degree of the first training image. For example, the higher the blur degree, the greater the value of K to be determined, and the lower the blur degree, the smaller the value of K to be determined.

[0113] Exemplarily, K frames of images can be selected from the video sequence after the frame insertion based on the tag image in the video sequence for multi-frame fusion. Exemplarily, k1 frames of images before the tag image, the tag image and k2 frames of images after the tag image can be selected as the images for multi-frame fusion. Wherein, k1 and k2 are integers, and k1+k2=K-1. Optionally, k1=k2=(K-1) / 2. That is, K-1 frames of images before the tag image can be selected to form K frames of images with the tag image; or K-1 frames of images after the tag image can be selected to form K frames of images with the tag image; or K-1 frames of images before the tag image and K-1 frames of images after the tag image can be selected to form K frames of images with the tag image.

[0114] It should be understood that the application does not limit the way of selecting K frames of images from the video sequence after the frame insertion.

[0115] Exemplarily, after selecting K frames of images, weighted calculation can be performed on the K frames of images to obtain a frame of image, that is, the first training image. Exemplarily, weighted calculation can be performed on the pixel values of the pixel points corresponding to the K frames of images to obtain the weighted calculation result of the pixel value of each pixel point; wherein the weighted calculation result of the pixel value of each pixel point is the pixel value of each pixel point in the first training image.

[0116] Exemplarily, the weights of the K frames of images can be set as required, such as the weights of the K frames of images being the same, or the weights of the frames corresponding to the closer distance to the tag image in the K frames of images being larger, and the weights of the frames corresponding to the farther distance to the tag image being smaller.

[0117] Exemplarily, when the video sequence is obtained by photographing the moving photographed object and the stationary photographed object under the condition that the image acquisition device is stationary, the moving photographed object in the generated first training image is blurred, and the stationary photographed object is clear, that is, the first training image is a partially blurred but stripe-free image.

[0118] Exemplarily, the partial blurring can mean that only the edge width of the moving photographed object in the image is greater than a preset width threshold, and the edge width of the stationary photographed object in the image is less than or equal to the preset width threshold.

[0119] Exemplarily, when the video sequence is obtained by photographing the moving photographed object and the stationary photographed object under the condition that the image acquisition device is dithered, the moving photographed object and the stationary photographed object in the generated first training image are both blurred, that is, the first training image is a globally blurred but stripe-free image. Wherein, the global blurring can mean that the edge width of all the photographed objects in the image is greater than a preset width threshold.

[0120] For example, when the video sequence is obtained by capturing a still object in a case of shaking of the image capturing device, the still object in the generated first training image is blurred, that is, the first training image is a globally blurred but stripe-free image.

[0121] In this way, a plurality of first training images which are blurred but stripe-free can be obtained, wherein the plurality of first training images can include images which are locally blurred, and / or images which are globally blurred.

[0122] S405, selecting a base image from the interpolated video sequence.

[0123] For example, an image which is spaced by L (L is an integer greater than or equal to 0) frames from the label image can be selected as the base image, and then the second training image is generated based on the base image.

[0124] For example, L can be determined according to a shooting time interval between an image captured according to a long exposure time and an image captured according to a short exposure time in actual application. For example, when the shooting time interval is 10 ms, L can be equal to 1. For example, when the shooting time interval is 20 ms, L can be equal to 2.

[0125] For example, when L is 0, the label image is selected as the base image.

[0126] In a possible manner, when L is greater than 0, an image which is spaced by L frames from the label image can be selected as the base image from images before the label image.

[0127] In a possible manner, when L is greater than 0, an image which is spaced by L frames from the label image can be selected as the base image from images after the label image.

[0128] S406, reducing the brightness of the base image.

[0129] In a possible manner, the base image can be processed by using an inverse gamma curve to reduce the brightness of the base image.

[0130] In a possible manner, a ratio of the first preset exposure time and the second preset exposure time can be determined, and the pixel value of each pixel point in the base image is divided by the ratio to reduce the brightness of the base image.

[0131] S407, adding stripes to the base image after the brightness is reduced.

[0132] Exemplarily, the stripes can be added to the reduced-luminance base image based on a waveform of the alternating current. Exemplarily, the alternating current is a sine wave.

[0133] Exemplarily, a plurality of frequencies can be set for the alternating current to simulate the frequency difference of alternating current in different regions. Exemplarily, the plurality of frequencies set for the alternating current can include the frequency of the alternating current in the target shooting scene.

[0134] Exemplarily, a plurality of different amplitudes can be set for the alternating current to simulate the difference in the light intensity in different scenes, which causes the difference in the light and dark contrast of the stripes.

[0135] Exemplarily, a plurality of different phases can be set for the alternating current, wherein the phase of the alternating current can be any degree between 0° and 360° (the phase can be 0° or 360°) to simulate the case that the stripes located at the edge are complete and / or incomplete in the image shot according to the short exposure time in the actual application.

[0136] In this way, a plurality of sine waves can be obtained, wherein the frequency and / or amplitude and / or phase of each sine wave is different. Each sine wave can be used to generate a stripe image. The following is exemplarily described by taking the generation of a stripe image by using one sine wave as an example.

[0137] Exemplarily, whether the amplitude of the sine wave is offset can be determined according to the maximum luminance value and / or minimum luminance value of the base image to avoid overexposure or underexposure of the base image after adding the stripes. Exemplarily, when the maximum luminance value is greater than or equal to a first preset luminance value, the amplitude of the sine wave can not be offset. When the maximum luminance value is less than the first preset luminance value, the amplitude of the sine wave can be offset by a first preset amplitude; wherein the first preset amplitude is a positive number, which can be set according to the maximum luminance value of the base image, which is not limited in the present application. Exemplarily, when the minimum luminance value is less than or equal to a second preset luminance value, the amplitude of the sine wave can not be offset. When the minimum luminance value is greater than the second preset luminance value, the amplitude of the sine wave can be offset by a second preset amplitude; wherein the second preset amplitude is a negative number, which can be set according to the minimum luminance value of the base image, which is not limited in the present application. Wherein the first preset luminance value is greater than the second preset luminance value, the absolute values of the first preset amplitude and the second preset amplitude can be the same or different, which is not limited in the present application.

[0138] Exemplarily, the stripe image can be generated according to the size of the base image and the frequency, amplitude and phase of the sine wave.

[0139] In one possible manner, the number of pixel rows M (M is a positive integer) generating one complete period of the fringe can be determined, and then a fringe image containing P1 (P1 is a positive integer or a positive decimal number, P1 = H / M) periods of the fringe can be generated according to the size (W*H, W is the width of the base image and H is the height of the base image) of the base image and M, and the frequency, amplitude and phase of the sine wave. For example, when M is less than or equal to the height of the base image, P1 is an integer or a decimal number greater than or equal to 1. When M is greater than the height of the base image, P1 is a positive decimal number less than 1. In this way, a horizontal fringe image can be generated.

[0140] For example, the number of pixel columns N (N is a positive integer) generating one complete period of the fringe can be determined, and then a fringe image containing P2 (P2 is a positive integer or a positive decimal number, P2 = W / N) periods of the fringe can be generated according to the size (W*H, W is the width of the base image and H is the height of the base image) of the base image and N, and the frequency, amplitude and phase of the sine wave. For example, when N is less than or equal to the width of the base image, P2 is an integer or a decimal number greater than or equal to 1. When N is greater than the width of the base image, P2 is a positive decimal number less than 1. In this way, a vertical fringe image can be generated.

[0141] For example, the fringe image can be superimposed with the base image with reduced brightness to obtain a second training image. For example, the horizontal fringe image can be superimposed with the base image with reduced brightness to obtain a second training image with horizontal fringes. For example, the vertical fringe image can be superimposed with the base image with reduced brightness to obtain a second training image with vertical fringes.

[0142] In this way, a plurality of second training images with clear but fringed images can be obtained, wherein the number of fringes, the direction of the fringes, and / or the light-dark contrast of the fringes in different second training images are different.

[0143] For example, one of the plurality of second training images, one of the plurality of first training images, and the label image can be taken as a set of training images, and then a plurality of sets of training data can be obtained.

[0144] For example, one image can be selected again from the images in the video sequence that are not selected as the label image as the label image, and then the plurality of sets of training data can be generated according to S402-S406.

[0145] For example, when the number of label images selected from a certain set of video sequences is greater than a preset frame number and less than or equal to the total number of frames of the video sequence, another set of video sequences can be selected from a plurality of sets of video sequences to generate training data. The preset frame number can be set according to requirements, which is not limited in the present application.

[0146] It should be noted that other ways can also be used to generate the training data, and the present application does not limit the same.

[0147] In a possible manner, two image acquisition devices (image acquisition device 1 and image acquisition device 2, the device parameters of the image acquisition device 1 and the image acquisition device 2 are the same) can be used to shoot a video. For example, the image acquisition device 1 and the image acquisition device 2 are bound, the exposure time of the image acquisition device 1 is less than the exposure time of the image acquisition device 2, the images in the video sequence captured by the image acquisition device 1 are clear and stripe-free images, and the images in the video sequence captured by the image acquisition device 2 are blurred but stripe-free images. For example, the image acquisition device 1 and the image acquisition device 2 can shoot according to the above-mentioned manner to obtain video sequence 1 and video sequence 2 respectively; wherein the image acquisition device 1 and the image acquisition device 2 shoot synchronously, and the image acquisition device 1 and the image acquisition device 2 are the same except for the exposure time.

[0148] For example, the i-th (i is a positive integer) image in the video sequence 1 shot by the image acquisition device 1 can be selected as the label image according to the above-mentioned method, and the i-th image in the video sequence 2 recorded by the image acquisition device 2 can be selected as the first training image.

[0149] For example, the k-th (k is a positive integer) image in the video sequence 1 shot by the image acquisition device 1 can be selected as the base image. For example, the distance between k and i is L.

[0150] In a possible manner, the image acquisition device 1 and the image acquisition device 2 can be used to shoot, and image 1 and image 2 can be obtained correspondingly. For example, the image 1 can be used as the label image, the image 2 can be used as the first training image, and the image 1 can be used as the base image to generate the second training image.

[0151] For example, a group or a batch of training data can be input into the first image processing model each time to train the first image processing model, wherein the batch of training data includes multiple groups of training data. The following will be exemplarily described by taking inputting a group of training data as an example.

[0152] In S302, the training data is input into the first image processing model, the first image processing model performs forward calculation on the first training image and the second training image in the training data, and outputs a fused image.

[0153] Exemplarily, after a set of training data is input into the first image processing model, the first image processing model can process the first training image and the second training image in the set of training data to obtain a frame of image and output the frame of image. For ease of description, the image output by the first image processing model during the training process can be referred to as a fusion image.

[0154] S303, determining a loss function value according to the label image and the fusion image in the training data, and adjusting the model parameters of the first image processing model based on the loss function value.

[0155] Exemplarily, after the first image processing model outputs the fusion image, the fusion image and the label image in the set of training data can be substituted into the loss function for calculation to obtain the loss function value. Then, the model parameters of the first image processing model can be adjusted with the goal of minimizing the loss function value.

[0156] It should be noted that the present application does not limit the loss function used for training the first image processing model.

[0157] Further, according to the above S302-S303, each set of training data is used to train the first image processing model until the loss function value is less than a first loss function threshold, or the training frequency reaches a first training frequency threshold, or the effect of the first image processing model meets a first preset effect condition, the training of the first image processing model is stopped. Exemplarily, the first loss function threshold, the first training frequency threshold and the first preset effect condition can be set according to requirements, and the present application does not limit this.

[0158] In this way, the first image processing model can learn the ability to remove blur, the ability to remove stripes, and the ability of image fusion.

[0159] In one possible way, in actual application, the long exposure time can be set as a third preset exposure time T3, and the short exposure time can be set as a fourth preset exposure time T4. Wherein, T3=N2*T, T is the energy period of alternating current, and N2 is a decimal greater than 1. T4=M2*T, M2 is a decimal between 0 and 1. When N2 is a decimal greater than 1, the energy period of the sine wave during the exposure process is not complete, which will also cause the first image to have stripes. In this case, the first image is blurred and has stripes (or clear but has stripes), and the second image is clear but has stripes.

[0160] In a possible manner, in actual application, the long exposure time length can be set as a fifth preset exposure time length T5, and the short exposure time length can be set as a sixth preset exposure time length T6. Wherein, T5=N3*T, T is an energy period of alternating current, and N3 is a decimal between 0 and 1. T6=M3*T, M3 is a decimal between 0 and 1, and M3 is less than N3. In this case, the first image is blurred and striped (or clear but striped), and the second image is clear but striped.

[0161] For example, the image processing model can include a second image processing model. The second image processing model can be pre-trained, and then used to process the blurred and striped (or clear but striped) image and the clear but striped image to obtain an image with details, clarity and no stripes in each region.

[0162] The training process of the second image processing model is described below.

[0163] Figure 6 The training process is schematically shown.

[0164] S501, generate training data, the training data including: a first training image, a second training image and a label image.

[0165] Figure 6 The data generation process is schematically shown.

[0166] For example, the training data can include multiple groups, and each group of training data includes: a first training image, a second training image and a label image.

[0167] Referring to Figure 7 For example, the training data generation process can refer to the following steps S601-S607:

[0168] S601, obtain a video sequence.

[0169] S602, select a label image from the video sequence.

[0170] S603, frame insertion is performed on the video sequence.

[0171] S604, select K frames of images from the video sequence after frame insertion for fusion.

[0172] For example, S601-S604 can refer to the description of S401-S404 above, and will not be described here.

[0173] S605, add stripes meeting the weak stripe condition to the fused image.

[0174] For example, S604 can obtain the blurred image, and then can add stripes to the blurred image to obtain the first training image which is blurred and has stripes.

[0175] For example, the longer the exposure time, the smaller the brightness difference between the light and dark stripes, that is, the brightness difference between the light and dark stripes in the image taken according to the long exposure time is smaller than the brightness difference between the light and dark stripes in the image taken according to the short exposure time. For example, the brightness difference between the light and dark stripes can be described by stripe intensity, the greater the brightness difference between the light and dark stripes, the greater the stripe intensity, the smaller the brightness difference between the light and dark stripes, the smaller the stripe intensity. For example, the stripes added to the blurred image can satisfy the weak stripe condition, that is, the intensity threshold, that is, the weak stripes are added to the blurred image. For example, the intensity threshold can be set as required, which is not limited in the present application.

[0176] For example, a plurality of stripes with intensity less than the intensity threshold can be added to the blurred image, so that a plurality of first training images with different stripe intensities can be obtained. For example, the way of adding stripes to the blurred image can refer to the description of S406 above, which will not be repeated here.

[0177] In this way, a plurality of first training images which are blurred and have weak stripes can be obtained, wherein the number of stripes, the direction of stripes and / or the intensity of stripes in different first training images are different. And the plurality of first training images can include local blurred images and / or global blurred images.

[0178] S606, selecting a base image from the inserted video sequence.

[0179] For example, S606 can refer to the description of S404 above, which will not be repeated here.

[0180] S607, reducing the brightness of the base image.

[0181] In one possible way, the base image can be processed by using an inverse gamma curve to reduce the brightness of the base image.

[0182] In one possible way, the ratio of the third preset exposure time and the fourth preset exposure time can be determined; the pixel value of each pixel point in the base image is divided by the ratio to reduce the brightness of the base image.

[0183] In one possible way, the ratio of the fifth preset exposure time and the sixth preset exposure time can be determined; the pixel value of each pixel point in the base image is divided by the ratio to reduce the brightness of the base image.

[0184] S608, adding stripes satisfying the strong stripe condition to the base image with reduced brightness.

[0185] Exemplarily, the stripes satisfying the strong stripe condition can be added to the base image with reduced brightness, where the strong stripe condition can refer to a stripe intensity greater than or equal to an intensity threshold, that is, the strong stripes are added to the base image with reduced brightness.

[0186] Exemplarily, a plurality of stripes with intensities greater than or equal to the intensity threshold can be added to the base image with reduced brightness, so that a plurality of second training images can be obtained. Exemplarily, the manner of adding the stripes to the base image can refer to the description of S406 above, which will not be described here.

[0187] In this way, a plurality of second training images with clear but strong stripes can be obtained, where the number of stripes, the direction of the stripes, and / or the light and dark contrast of the stripes are different in different second training images.

[0188] S502, input the training data into the second image processing model, and perform forward calculation on the first training image and the second training image in the training data by the second image processing model, and output a fusion image.

[0189] S503, determine a loss function value according to the label image in the training data and the fusion image, and adjust the model parameters of the second image processing model based on the loss function value.

[0190] Exemplarily, S502-S503 can refer to the description of S302-S303 above, which will not be described here.

[0191] Further, according to the above S502-S503, each group of training data is used to train the second image processing model until the loss function value is less than a second loss function threshold, or the training frequency reaches a second training frequency threshold, or the effect of the second image processing model meets a second preset effect condition, the training of the second image processing model is stopped. Exemplarily, the second loss function threshold, the second training frequency threshold and the second preset effect condition can be set according to requirements, and the present application does not limit this.

[0192] In this way, the second image processing model can learn the ability to remove blur, the ability to remove stripes, and the ability of image fusion.

[0193] The above method can obtain the trained image processing model: the first image processing model and the second image processing model. Then the trained first image processing model and the trained second image processing model can be deployed in the image acquisition device. Further, in actual application, the first image and the second image captured can be processed by the first image processing model or the second image processing model to obtain a clear and stripe-free image.

[0194] It should be noted that the training data of S401 and the training data of S601 can also be mixed according to a preset ratio, and then a mixed training data is used to train an image processing model. The image processing model is deployed in the image acquisition device to process the images captured at different long exposure times and the images captured at short exposure times to obtain images with details, clarity and stripes in each region. The preset ratio can be set as required, and the present application does not limit this.

[0195] It should be noted that the training data of S401 and the training data of S601 can also be mixed according to a preset ratio, and then a mixed training data is used to train an image processing model. The image processing model is deployed in the image acquisition device to process the images captured at different long exposure times and the images captured at short exposure times to obtain images with details, clarity and stripes in each region. The preset ratio can be set as required, and the present application does not limit this.

[0196] Figure 7 The processing flow diagram is shown for illustration.

[0197] Referring to Figure 7 For example, the image acquisition device includes an image sensor, a RAW domain processing module, an RGB domain processing module and a YUV domain processing module. It should be understood that, Figure 7 The image acquisition device shown is only an example of the image acquisition device, and the image acquisition device can have more or fewer components than those shown in the figure, can combine two or more components, or can have a different component configuration. Figure 7 The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0198] For example, after the light is incident on the image sensor through the lens, the image sensor can convert the received optical signal into an electrical signal and output a RAW image to the RAW domain processing module.

[0199] Referring to Figure 7 For example, after obtaining the RAW image, the RAW domain processing module can perform RAW domain processing (such as level correction, bad pixel removal, etc.) on the RAW image to obtain an RGB image.

[0200] Referring to Figure 7After the RGB image is obtained, the RGB domain processing module can perform RGB domain processing (such as demosaicing, white balance adjustment, etc.) on the RGB image, and can obtain a YUV image.

[0201] With reference to the foregoing Figure 8 After the YUV image is obtained, the YUV domain processing module can perform YUV domain processing (such as gamma correction, sharpening, color adjustment, etc.) on the YUV image, and can obtain a to-be-displayed image. Then the image acquisition device can display the to-be-displayed image.

[0202] In one possible manner, if the images in the video sequence used in the training process are RAW images, the image processing model obtained through the training can be integrated in the RAW domain processing module, or can be set before the RAW domain processing module as an independent module.

[0203] In one possible manner, if the images in the video sequence used in the training process are RGB images, the image processing model obtained through the training can be integrated in the RGB domain processing module, or can be set before the RGB domain processing module as an independent module.

[0204] In one possible manner, if the images in the video sequence used in the training process are YUV images, the image processing model obtained through the training can be integrated in the YUV domain processing module, or can be set before the YUV domain processing module as an independent module.

[0205] In one possible manner, if the images in the video sequence used in the training process are images processed in the YUV domain, the image processing model obtained through the training can be set after the YUV domain processing module as an independent module.

[0206] It should be noted that the first image processing model and the second image processing model can be an integral whole, or can be two independent parts. When the first image processing model and the second image processing model are two independent parts, the first image processing model and the second image processing model can be deployed at the same position of the image acquisition device, or can be deployed at different positions of the image acquisition device, and the present application does not limit this.

[0207] Figure 9a The image processing flowchart is shown for illustration.

[0208] S801, in an alternating current light environment, a first image and a second image are captured for a motion scene. The exposure time length of the first image is greater than that of the second image, and the exposure time length of the second image is less than the energy period of the alternating current.

[0209] Exemplarily, before shooting in an alternating current light environment, the current shooting scene can be detected to detect whether the current shooting scene is a motion scene. Exemplarily, the motion speed of the photographed object and the shaking speed of the image acquisition device can be detected, and then whether the current shooting scene is a motion scene is detected according to the motion speed and the shaking speed.

[0210] Exemplarily, before shooting, the image acquisition device is in a preview state, and the image acquisition device can detect the motion speed of the photographed object according to the image collected in the preview state. Exemplarily, the image acquisition device can detect the motion speed of the photographed object according to the current frame image and the last frame image. Exemplarily, the image acquisition device can determine the position of the photographed object in the current frame image and the position of the photographed object in the last frame image, and then determine the motion speed of the photographed object according to the position of the photographed object in the current frame image and the position of the photographed object in the last frame image, and the time interval between the current frame image and the last frame image.

[0211] Exemplarily, before shooting, the image acquisition device can determine the shaking speed of the image acquisition device according to the data collected by the motion detection sensor (such as a gyroscope, an acceleration sensor) in the image acquisition device.

[0212] Exemplarily, when it is determined according to the motion speed of the photographed object and the shaking speed of the image acquisition device that the relative motion speed of the photographed object and the image acquisition device is greater than a relative motion speed threshold, it can be determined that the current shooting scene is a motion scene. When it is determined according to the motion speed of the photographed object and the shaking speed of the image acquisition device that the relative motion speed of the photographed object and the image acquisition device is less than or equal to the relative motion speed threshold, it can be determined that the current shooting scene is not a motion scene. The relative motion speed threshold can be set as required, which is not limited in the present application.

[0213] Exemplarily, when it is detected that the current shooting scene is a motion scene, the exposure time length for shooting the first image and the exposure time length for shooting the second image can be set according to the motion speed of the photographed object and the shaking speed of the image acquisition device.

[0214] In one possible manner, when it is determined according to the motion speed and the shaking speed that the relative motion speed of the photographed object and the image acquisition device is less than or equal to a preset relative motion speed, the exposure time length for shooting the first image can be set as a first preset exposure time length T1, and the exposure time length for shooting the second image can be set as a second preset exposure time length T2. The preset relative motion speed is greater than the relative motion speed threshold, which can be set as required, which is not limited in the present application.

[0215] In a possible implementation, when it is determined according to the motion speed and the shaking speed that the relative motion speed between the photographed object and the image acquisition device is less than or equal to a preset relative motion speed, the exposure time length for shooting the first image can be set as a third preset exposure time length T3, and the exposure time length for shooting the second image can be set as a fourth preset exposure time length T4.

[0216] In a possible implementation, when it is determined according to the motion speed and the shaking speed that the relative motion speed between the photographed object and the image acquisition device is greater than the preset relative motion speed, the exposure time length for shooting the first image can be set as a fifth preset exposure time length T5, and the exposure time length for shooting the second image can be set as a sixth preset exposure time length T6.

[0217] For example, when the exposure time length for shooting the first image is set as the first preset exposure time length T1, and the exposure time length for shooting the second image is set as the second preset exposure time length T2, in an alternating current (AC) light environment, after the image acquisition device receives a shooting operation of a user, when it is detected that the current shooting scene is a motion scene, the image acquisition device can perform shooting according to the first preset exposure time length T1 for the motion scene in response to the operation behavior of the user, to obtain the first image, and perform shooting according to the second preset exposure time length T2 to obtain the second image.

[0218] For example, when the exposure time length for shooting the first image is set as the third preset exposure time length T3, and the exposure time length for shooting the second image is set as the fourth preset exposure time length T4, in an alternating current (AC) light environment, after the image acquisition device receives a shooting operation of a user, when it is detected that the current shooting scene is a motion scene, the image acquisition device can perform shooting according to the third preset exposure time length T3 for the motion scene in response to the operation behavior of the user, to obtain the first image, and perform shooting according to the fourth preset exposure time length T4 to obtain the second image.

[0219] For example, when the exposure time length for shooting the first image is set as the fifth preset exposure time length T5, and the exposure time length for shooting the second image is set as the sixth preset exposure time length T6, in an alternating current (AC) light environment, after the image acquisition device receives a shooting operation of a user, when it is detected that the current shooting scene is a motion scene, the image acquisition device can perform shooting according to the fifth preset exposure time length T5 for the motion scene in response to the operation behavior of the user, to obtain the first image, and perform shooting according to the sixth preset exposure time length T6 to obtain the second image.

[0220] In a possible manner, the same image acquisition device can be used to capture the first image and the second image. The time interval between capturing the first image and the second image is less than a preset time length, where the preset time length can be determined according to the motion speed of the photographed object and / or the shaking speed of the image acquisition device, which is not limited in the present application. Optionally, the first image and the second image can be two consecutive images.

[0221] For example, the first image can be captured first, and then the second image can be captured, or the second image can be captured first, and then the first image can be captured, which is not limited in the present application.

[0222] In a possible manner, two image acquisition devices can be used to capture the first image and the second image at the same time. After obtaining the first image and the second image, the first image and the second image can be calibrated, for example, the color and brightness of the first image and the second image can be calibrated, because the shooting parameters of the two image acquisition devices are different.

[0223] S802, input the first image and the second image into the trained image processing model, and output a target image by the image processing model.

[0224] For example, when the exposure time of the first image is a first preset exposure time T1, and the exposure time of the second image is a second preset exposure time T2, the trained first image processing model can be selected to process the first image and the second image. For example, the first image processing model learns the ability of removing stripes, the ability of removing blur, and the ability of image fusion, and then after inputting the first image and the second image into the first image processing model for processing, the first image processing model can perform the processing of removing stripes, removing blur, and image fusion, to obtain a clear and stripe-free target image and output the target image. In addition, during the fusion process, the areas with better image quality in the images are selected for fusion. Because the low-light area in the second image is underexposed and loses details, and the high-light area has details, the high-light area in the first image is overexposed and loses details, and the low-light area has details, therefore, the low-light area in the first image and the high-light area in the second image are selected for fusion, so that the target image obtained by the fusion has details in each area, and the brightness dynamic range (that is, the difference between light and dark) of the target image is expandable.

[0225] Figure 9a For example, the image processing process is shown in the schematic diagram.

[0226] Referring to Figure 9a For example, Figure 9bIn a1 is a first image, the first image is globally blurred but without stripes, a2 is a second image, the second image is clear but with stripes, a1 and a2 are input to the first image processing model, the first image processing model processes a1 and a2, and outputs a target image a3, a3 is a clear and stripe-free image.

[0227] Figure 9b An image processing process diagram is exemplarily shown.

[0228] Referring to Figure 9b , exemplarily, Figure 9c In b1 is a first image, the first image is locally blurred but without stripes, b2 is a second image, the second image is clear but with stripes, b1 and b2 are input to the first image processing model, the first image processing model processes b1 and b2, and outputs a target image b3, b3 is a clear and stripe-free image.

[0229] Exemplarily, when the exposure time corresponding to the first image is a third preset exposure time T3, and the exposure time corresponding to the second image is a fourth preset exposure time T4, or when the exposure time corresponding to the first image is a fifth preset exposure time T5, and the exposure time corresponding to the second image is a sixth preset exposure time T6, the trained second image processing model can be selected to process the first image and the second image. Exemplarily, the second image processing model learns the de-stripping ability, the de-blurring ability and the image fusion ability, and then inputs the first image and the second image into the second image processing model for processing, and the second image processing model can perform de-stripping, de-blurring and image fusion processing, and output a clear and stripe-free target image. In this way, the obtained target image is a clear, stripe-free, detailed and brightness-dynamic-range-expandable image.

[0230] Figure 9c An image processing process diagram is exemplarily shown.

[0231] Referring to Figure 9c , exemplarily, Figure 9d In c1 is a first image, the first image is globally blurred and has weak stripes, c2 is a second image, the second image is clear but has strong stripes, c1 and c2 are input to the second image processing model, the second image processing model processes c1 and c2, and outputs a target image c3, c3 is a clear and stripe-free image.

[0232] Figure 9d An image processing process diagram is exemplarily shown.

[0233] Referring to Figure 9d , exemplarily, Figure 10The first image d1 is locally blurred and weakly striped, the second image d2 is clear but strongly striped, d1 and d2 are input into the second image processing model, the second image processing model processes d1 and d2, and outputs a target image d3, d3 is a clear and stripe-free image.

[0234] In one example, ​ A schematic block diagram of an apparatus 1000 is shown, which can implement the embodiments of the present application. The apparatus 1000 can include a processor 1001 and a transceiver / transceiver pin 1002, and optionally further include a memory 1003.

[0235] The various components of the apparatus 1000 are coupled together by a bus 1004, which can include a data bus, a power bus, a control bus, and a state signal bus. For the sake of clarity, the various buses are illustrated in Figure as the bus 1004.

[0236] Optionally, the memory 1003 can be used for instructions in the foregoing method embodiments. The processor 1001 can be used to execute the instructions in the memory 1003, and control the receiving pin to receive signals and the sending pin to send signals.

[0237] The apparatus 1000 can be an electronic device or a chip of an electronic device in the above method embodiments.

[0238] All relevant content of each step involved in the above method embodiments can be cited from the function description of the corresponding function module, which will not be repeated here.

[0239] The embodiment also provides a computer storage medium, which stores computer instructions, when the computer instructions run on an electronic device, the electronic device executes the above related method steps to implement the image processing and / or model training method in the above embodiment.

[0240] The embodiment also provides a computer program product, when the computer program product runs on a computer, the computer executes the above related steps to implement the image processing and / or model training method in the above embodiment.

[0241] In addition, the embodiments of the present application also provide an apparatus, which can be a chip, a component or a module, and the apparatus can include a processor and a memory connected thereto. The memory is used to store computer execution instructions, and when the apparatus is running, the processor can execute the computer execution instructions stored in the memory to make the chip execute the image processing and / or model training method in the above method embodiments.

[0242] The electronic device, the computer storage medium, the computer program product or the chip provided in the embodiment are used for executing the corresponding method provided above, and thus the beneficial effects achieved by the electronic device, the computer storage medium, the computer program product or the chip can refer to the beneficial effects of the corresponding method provided above, which will not be described here again.

[0243] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0244] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented by other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0245] The units described as separate components may or may not be physically separate, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place, or can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0246] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0247] Any content of each embodiment of the present application, and any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0248] If the integrated unit is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or say the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, includes several instructions to make a device (which can be a single-chip microcomputer, a chip, etc.) or a processor execute all or part of the steps of the various embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0249] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are only illustrative but not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope of protection of the claims, and all of them belong to the protection of the present application.

[0250] The steps of the method or algorithm described in combination with the disclosure of the embodiments of the present application can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a compact disc (CD-ROM), or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0251] Those skilled in the art can understand that the functions described in the embodiments of the present application in the one or more examples above can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes computer storage medium and communication medium, wherein the communication medium includes any medium that facilitates the transfer of computer program from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0252] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the specific embodiments described above, which are merely illustrative rather than restrictive, and those of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

Claims

1. An image processing method, characterized by, The method comprises: In an alternating current light environment, detecting a motion speed of a photographed object and a shaking speed of an image acquisition device; Shooting a first image and a second image for a motion scene; wherein an exposure time of the first image is longer than an exposure time of the second image, and the exposure time of the second image is shorter than an energy period of the alternating current; Inputting the first image and the second image into a trained image processing model, and outputting a target image from the image processing model; Wherein, the shooting of the first image and the second image for the motion scene comprises: When it is determined according to the motion speed and the shaking speed that a relative motion speed between the photographed object and the image acquisition device is less than or equal to a preset relative motion speed, shooting the first image according to a first preset exposure time and shooting the second image according to a second preset exposure time for the motion scene; Wherein, the first preset exposure time is N1 times of the energy period, the second preset exposure time is M1 times of the energy period, N1 is a number greater than or equal to 1, and M1 is a decimal between 0 and 1.

2. The method of claim 1, wherein, The method further comprises: Detecting whether a current shooting scene is the motion scene according to the motion speed and the shaking speed.

3. The method according to claim 1 or 2, characterized in that, The shooting of the first image and the second image for the motion scene comprises: When it is determined according to the motion speed and the shaking speed that the relative motion speed between the photographed object and the image acquisition device is greater than a preset relative motion speed, shooting the first image according to a fifth preset exposure time and shooting the second image according to a sixth preset exposure time; Wherein, the fifth preset exposure time is N3 times of the energy period, the sixth preset exposure time is M3 times of the energy period, N3 and M3 are decimals between 0 and 1.

4. The method of claim 1, wherein: The image processing model is trained based on a first training image and a second training image, the second training image has banding, and a brightness of the first training image is greater than a brightness of the second training image.

5. The method of claim 4, wherein, The method further comprises: Obtaining a video sequence, the video sequence being obtained by shooting for a motion scene, the video sequence comprising images of high dynamic range imaging, and images in the video sequence being free of banding; Selecting one image from the video sequence as a label image; Frame inserting the video sequence, selecting K images from the frame-inserted video sequence based on the label image, and fusing the K images to obtain the first training image; Selecting one image from the frame-inserted video sequence as a base image, reducing a brightness of the base image, and adding banding to the base image with reduced brightness to obtain the second training image.

6. The method of claim 1 or 3, wherein: The image processing model is trained based on first training images and second training images, the first training images and the second training images both have stripes, the stripe intensity of the first training images is smaller than that of the second training images, and the brightness of the first training images is greater than that of the second training images.

7. The method of claim 6, wherein, The method further comprises: obtaining a video sequence, the video sequence being obtained by shooting a moving scene, the video sequence comprising images of high dynamic range imaging, and the images in the video sequence being stripe-free; selecting a frame of image from the video sequence as a label image; interpolating the video sequence, selecting K frames of image from the interpolated video sequence based on the label image for fusion, and adding stripes meeting a weak stripe condition to the fused image to obtain the first training image; selecting a frame of image from the interpolated video sequence as a base image, reducing the brightness of the base image, and adding stripes meeting a strong stripe condition to the base image with reduced brightness to obtain the second training image.

8. The method of any one of claims 1 to 7, wherein: the first image and the second image are images shot by the same image acquisition device, and the shooting time interval of the first image and the second image is less than a preset time length; or the first image and the second image are images shot by different image acquisition devices, and the starting shooting time of the first image and the second image is the same.

9. A model training method, comprising: The method comprises: generating training data, the training data comprising: first training images, second training images, and a label image, the brightness of the first training images being greater than that of the second training images; inputting the training data into an image processing model, performing forward calculation on the first training images and the second training images in the training data by the image processing model, and outputting a fused image; calculating a loss function value based on the fused image and the label image in the training data, and adjusting model parameters of the image processing model according to the loss function value; wherein, when the first training image is stripe-free, the second training image has stripes; when the first training image has weak stripes, the second training image has strong stripes.

10. The method of claim 9, wherein, The generation of the training data comprises: obtaining a video sequence, the video sequence being obtained by shooting a moving scene, the video sequence comprising images of high dynamic range imaging, and the images in the video sequence being stripe-free; selecting a frame of image from the video sequence as a label image; interpolating the video sequence, selecting K frames of image from the interpolated video sequence based on the label image for fusion, and obtaining the first training image; selecting a frame of image from the interpolated video sequence as a base image, reducing the brightness of the base image, and adding stripes to the base image with reduced brightness to obtain the second training image.

11. The method of claim 9, wherein, The generation of the training data comprises: Acquiring a video sequence, the video sequence being captured for a moving scene, the video sequence comprising images of high dynamic range imaging, the images in the video sequence being stripe-free; Selecting a frame image from the video sequence as a label image; Interpolating the video sequence, selecting K frame images from the interpolated video sequence based on the label image for fusion, and adding stripes meeting a weak stripe condition to the fused image to obtain the first training image; Selecting a frame image from the interpolated video sequence as a base image, reducing the brightness of the base image, and adding stripes meeting a strong stripe condition to the base image after the brightness is reduced to obtain the second training image.

12. The method of claim 11, wherein the weak stripe condition comprises that the stripe intensity is less than an intensity threshold; the strong stripe condition comprises that the stripe intensity is greater than or equal to the intensity threshold.

13. An image capture device, comprising: A device for performing the method of any one of claims 1-8.

14. An electronic device, comprising: Comprising: a memory and a processor, the memory being coupled to the processor; the memory stores program instructions which, when executed by the processor, cause the electronic device to perform the image processing method of any one of claims 1-8.

15. An electronic device, comprising: Comprising: a memory and a processor, the memory being coupled to the processor; the memory stores program instructions which, when executed by the processor, cause the electronic device to perform the model training method of any one of claims 9-12.

16. A chip, characterized by An electronic device comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from the memory of the electronic device and send the signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the image processing method of any one of claims 1-8.

17. A chip, characterized by An electronic device comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from the memory of the electronic device and send the signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the model training method of any one of claims 9-12.

18. A computer storage medium, comprising, The computer readable storage medium stores a computer program, when the computer program runs on a computer or a processor, the computer or the processor executes the method of any one of claims 1-12.

19. A computer program product, characterised in that, The computer program product contains a software program, when the software program is executed by a computer or a processor, the steps of the method of any one of claims 1-12 are executed.

Images