Frame rate adjustment method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a frame rate adjustment method and device, electronic equipment, a storage medium and a program product. The frame rate adjustment method comprises the following steps: in response to a shooting request, obtaining image information of an image currently shot by a camera; based on the image information, a current shooting scene and motion information of the image are determined, and the motion information of the image is at least used for representing the motion condition of a shooting object in the image; and carrying out frame rate adjustment based on the current shooting scene and the motion information of the image. According to the embodiment of the invention, the frame rate can be flexibly adjusted, so that the recorded video is clearer.

Description

Technical Field

[0001] This disclosure relates to the field of communications, and more particularly to a frame rate adjustment method and apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the continuous development of electronic devices, video recording based on camera units has become a standard feature. Currently, when electronic devices use camera units for video recording, the frame rate of the output images is usually fixed. After setting the frame rate of the output images, the display and encoding are performed based on this fixed frame rate, resulting in poor video recording quality. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a frame rate adjustment method and apparatus, electronic device, storage medium, and program product, which can flexibly adjust the frame rate to make the recorded video clearer.

[0004] According to a first aspect of the present disclosure, a frame rate adjustment method is provided, comprising:

[0005] In response to a shooting request, obtain image information of the image currently being captured by the camera;

[0006] Based on the image information, the current shooting scene and the motion information of the image are determined; the motion information of the image is used to characterize at least the motion of the shooting object in the image;

[0007] Frame rate adjustment is performed based on the current shooting scene and the motion information of the image.

[0008] In some embodiments, determining the current shooting scene and the motion information of the image based on the image information includes:

[0009] Based on the grayscale change information of the pixels in the image and the brightness information of the environment in which the subject is located, the current shooting scene is determined;

[0010] Based on the image information, the motion speed of the image and the size of the motion window of the image are determined. The motion speed of the image is used to characterize at least the speed of movement of the photographed object in the image; the motion window is used to characterize at least the image area occupied by the photographed object in the image.

[0011] In some embodiments, determining the motion speed of the image and the size of the motion window of the image based on the image information includes:

[0012] The motion speed of the image is determined based on image information acquired by a contact camera.

[0013] The size of the motion window of the image is determined based on the image information acquired by the event camera.

[0014] In some embodiments, determining the motion speed of the image based on image information acquired by a contact camera includes:

[0015] Based on the image information acquired by the contact camera, the grayscale difference of each pixel in the image and the gradient value corresponding to each pixel are obtained.

[0016] Based on the grayscale difference and the gradient value, the initial motion speed of each pixel is determined;

[0017] The initial motion speed of each pixel is adjusted to obtain the target motion speed of each pixel;

[0018] The target motion speed of each pixel is filtered to obtain the motion speed of the image.

[0019] In some embodiments, adjusting the initial motion speed of each pixel to obtain the target motion speed of each pixel includes:

[0020] The initial motion speed of each pixel is reduced and adjusted to obtain the adjustment speed of each pixel;

[0021] The adjustment speed of each pixel is filtered to obtain the filtered speed of each pixel.

[0022] The filtering speeds of each pixel are weighted to obtain the target motion speed of each pixel.

[0023] In some embodiments, filtering the target motion speed of each pixel to obtain the motion speed of the image includes:

[0024] Select a portion of the target motion velocities from the multiple target motion velocities of the multiple pixels;

[0025] The average value of the motion speed of a portion of the target is taken as the motion speed of the image;

[0026] Among these, the smallest of the target motion speeds is greater than all the target motion speeds other than some of the target motion speeds.

[0027] In some embodiments, the frame rate adjustment based on the current shooting scene and the motion information of the image includes:

[0028] Based on the current shooting scene and the motion information of the image, determine the target frame rate corresponding to the current shooting scene and the motion information of the image;

[0029] If the current frame rate of the video stream is detected to be lower than the target frame rate, the frame rate of the video stream is adjusted based on the target frame rate.

[0030] In some embodiments, the method further includes:

[0031] If the current frame rate is greater than or equal to the target frame rate, stop adjusting the frame rate of the video stream.

[0032] In some embodiments, the video stream includes a display preview stream and / or a video recording stream, and adjusting the frame rate of the video stream based on the target frame rate includes:

[0033] Based on the target frame rate, the frame rate of the display preview stream and / or the video recording stream is adjusted.

[0034] According to a second aspect of the present disclosure, a frame rate adjustment apparatus is provided, comprising:

[0035] The image acquisition module is configured to acquire image information of the currently captured image in response to a shooting request;

[0036] The motion determination module is configured to determine the current shooting scene and the motion information of the image based on the image information; the motion information of the image is used to characterize at least the motion of the shooting object in the image;

[0037] The adjustment module is configured to adjust the frame rate based on the shooting scene and the motion information.

[0038] In some embodiments, the motion determination module includes:

[0039] The scene determination module is configured to determine the current shooting scene based on the grayscale change information of pixels in the image and the brightness information of the environment in which the shooting object is located;

[0040] The speed determination module is configured to determine the motion speed of the image and the size of the motion window of the image based on the image information. The motion speed of the image is used at least to characterize the speed of movement of the photographed object in the image; the motion window is used at least to characterize the image area occupied by the photographed object in the image.

[0041] In some embodiments, the speed determination module is configured to determine the motion speed of the image based on image information acquired by a contact camera, and to determine the size of the motion window of the image based on image information acquired by an event camera.

[0042] In some embodiments, the speed determination module is configured to: acquire the grayscale difference of each pixel in the image and the gradient value corresponding to each pixel based on the image information acquired by the contact camera; determine the initial motion speed of each pixel based on the grayscale difference and the gradient value; adjust the initial motion speed of each pixel to obtain the target motion speed of each pixel; and filter the target motion speed of each pixel to obtain the motion speed of the image.

[0043] In some embodiments, the speed determination module is configured to reduce and adjust the initial motion speed of each pixel to obtain the adjusted speed of each pixel; filter the adjusted speed of each pixel to obtain the filtered speed of each pixel; and perform weighted processing on the filtered speed of each pixel to obtain the target motion speed of each pixel.

[0044] In some embodiments, the speed determination module is configured to select a portion of the target motion speeds from a plurality of target motion speeds of the plurality of pixels; and to take the average value of the portion of the target motion speeds as the motion speed of the image; wherein the smallest of the portion of the target motion speeds is greater than all the target motion speeds other than the portion of the plurality of target motion speeds.

[0045] In some embodiments, the adjustment module is configured to determine a target frame rate corresponding to the motion information of the current shooting scene and the image based on the motion information of the current shooting scene and the image; and to adjust the frame rate of the video stream based on the target frame rate if the current frame rate of the video stream is detected to be less than the target frame rate.

[0046] In some embodiments, the apparatus further includes:

[0047] The stop adjustment module is configured to stop adjusting the frame rate of the video stream when the current frame rate is greater than or equal to the target frame rate.

[0048] In some embodiments, the adjustment module is configured to adjust the frame rate of the display preview stream and / or video recording stream based on the target frame rate.

[0049] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0050] processor;

[0051] Memory used to store computer programs or instructions;

[0052] The processor executes the computer program or instructions to implement the steps of the method described in the first aspect above.

[0053] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, the storage medium storing a computer program or instructions that, when executed by a processor, implement the steps of the method described in the first aspect above.

[0054] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the method described in the first aspect above.

[0055] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0056] In this embodiment, frame rate adjustment is performed based on the current shooting scene and image motion information. This takes into account the impact of the current shooting scene and the motion of the subject on the frame rate, making the frame rate adjustment more accurate and adaptable to different shooting scenes. Furthermore, unlike existing methods that cannot adjust the frame rate, this embodiment adjusts the frame rate during the response to a shooting request, enabling flexible frame rate adjustment and resulting in clearer recorded video.

[0057] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0058] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0059] Figure 1 This is a flowchart illustrating a frame rate adjustment method according to an exemplary embodiment.

[0060] Figure 2 This is a schematic diagram of a frame rate adjustment module according to an exemplary embodiment.

[0061] Figure 3 This is a schematic diagram illustrating an image processing system according to an exemplary embodiment.

[0062] Figure 4 This is a schematic diagram illustrating a motion detection module according to an exemplary embodiment.

[0063] Figure 5 This is a schematic diagram illustrating a frame rate compensation module according to an exemplary embodiment.

[0064] Figure 6 This is a block diagram illustrating a frame rate adjustment device according to an exemplary embodiment.

[0065] Figure 7 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0066] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0067] This disclosure proposes a frame rate adjustment method applicable to video recording scenarios using electronic devices. For example, in a scenario where an electronic device is recording birds in flight, the frame rate can be increased based on this method to capture clear bird footage. As another example, in a scenario where an electronic device is recording slow-motion video, the frame rate adjustment method can acquire more slow-motion frames, resulting in clearer slow-motion recordings.

[0068] Figure 1 This is a flowchart illustrating a frame rate adjustment method according to an exemplary embodiment. Figure 1 As shown, the frame rate adjustment method performed by this electronic device mainly includes the following steps:

[0069] S11. In response to the shooting request, obtain the image information of the image currently being captured by the camera;

[0070] S12. Based on image information, determine the current shooting scene and the motion information of the image; the motion information of the image is at least used to characterize the motion of the shooting object in the image;

[0071] S13. Adjust the frame rate based on the current shooting scene and image motion information; wherein the adjusted frame rate is greater than the original frame rate.

[0072] In this embodiment of the disclosure, the frame rate adjustment method is applied to an electronic device. The electronic device may include a device with a shooting function or a standalone camera device; the device with a shooting function includes smartphones, tablets, laptops, smart bracelets, or smartwatches, etc., and this embodiment of the disclosure does not impose any limitations.

[0073] It should be noted that the aforementioned electronic device includes a camera module with shooting capabilities. This camera module may include a contact camera with an image sensor (CMOS Image Sensor, CIS) and an event camera with a dynamic vision sensor (Dynamic Vision Sensor, DVS).

[0074] For example, the frame rate adjustment method can be performed using a smartphone, that is, the smartphone's contact camera and / or event camera can be used to record images, and the frame rate can be adjusted based on the shooting scene and motion information of the images, so that the video recording is clearer.

[0075] In step S11, the above-mentioned shooting request can be used to request the electronic device to capture the subject in order to achieve video recording.

[0076] It should be noted that the shooting request can be generated in various ways, such as the user interacting with the camera icon on the screen of the electronic device, the user triggering a shortcut key on the electronic device, the user inputting voice input into the electronic device, or the user inputting gestures into the electronic device. The interaction with the screen of the electronic device can include actions such as clicking, double-clicking, or swiping, and this embodiment of the disclosure does not impose any limitations.

[0077] Of course, the shooting request can also be sent by other devices with which communication has been established. For example, the shooting request can be sent to an electronic device via a selfie stick to request the electronic device to record video.

[0078] According to the embodiments of this disclosure, obtaining image information of the image currently captured by the camera may include: obtaining image information of the image currently captured by the current camera through a contact camera and / or a dynamic vision camera in the camera module.

[0079] It should be noted that the above image information can be obtained by electronic devices capturing the subject simultaneously using a contact camera and a dynamic vision camera.

[0080] In step S12, the shooting scene can be divided into football scene S1, sports scene S2, static scene S3, night scene video scene S4, and slow motion scene S5, depending on the different movement of the subject and the brightness of the environment in which the subject is located.

[0081] Among them, the football scene and the sports scene are shooting scenes for subjects in motion. In the football scene, the speed of the subject can be greater than the speed of the subject in the sports mode scene.

[0082] In this embodiment of the disclosure, the current shooting scene can be one of five scenes: football scene, sports scene, static scene, night scene, and slow motion scene. This embodiment of the disclosure does not limit this.

[0083] The above-mentioned determination of the current shooting scene based on image information may include: analyzing the brightness information of the environment in which the shooting object is located based on image information, and determining the current shooting scene based on the brightness information; it may also include: determining the current shooting scene based on the grayscale change information of pixels in the image.

[0084] For example, brightness information can be used to determine whether the current shooting scene is a night scene video scene. As another example, grayscale variation information of each pixel can be used to determine whether the current scene is a static scene or a moving scene.

[0085] In this embodiment of the disclosure, the motion information of the image is used at least to characterize the motion of the object being photographed in the image. It should be noted that the motion of the object causes changes in the grayscale of pixels in the image; therefore, the motion information of the image can also characterize the grayscale changes of pixels in the currently captured image relative to the previous image frame. In some embodiments, the motion information of the image may include: the motion speed of the image and / or the size of the motion window of the image.

[0086] The motion speed of the aforementioned image is used at least to characterize the speed of motion of the subject in the image. It should be noted that different speeds of motion of the subject will result in different rates of grayscale changes in the pixels within the image. Therefore, the motion speed of the image can also characterize the rate of grayscale change at the pixels in the currently captured image relative to the previous image frame. Specifically, the motion speed of the image can be obtained by averaging the target motion speeds of all pixels in the image, by averaging the target motion speeds of a subset of pixels, or by representing the maximum target motion speed among multiple target motion speeds of multiple pixels in the image.

[0087] The motion window of the aforementioned image is used at least to represent the image area occupied by the photographed object. It should be noted that when the photographed object moves, the grayscale of the corresponding pixels in the image changes. Therefore, the motion window can also represent the area enclosed by at least one pixel whose grayscale changes among the pixels corresponding to the photographed object.

[0088] The above-mentioned determination of image motion information based on image information may include: determining image motion information using image information acquired by a contact camera, or it may include: determining image motion information using image information acquired by an event camera and image information acquired by a contact camera together.

[0089] In step S13, the electronic device can first determine whether to adjust the frame rate based on the current shooting scene and the motion information of the image, and then adjust the frame rate of the video stream if it is determined that the frame rate needs to be adjusted.

[0090] In some embodiments, frame rate adjustment is performed based on the current shooting scene and motion information of the image, including:

[0091] Based on the motion information of the current shooting scene and the image, determine the target frame rate corresponding to the motion information of the current shooting scene and the image;

[0092] If the current frame rate of the video stream is detected to be lower than the target frame rate, the frame rate of the video stream is adjusted based on the target frame rate.

[0093] In this embodiment of the disclosure, the electronic device stores a first mapping relationship in advance, which points to the relationship between the current shooting scene, the motion information of the image, and the target frame rate.

[0094] The above-mentioned determination of the target frame rate corresponding to the motion information of the current shooting scene and image based on the motion information of the current shooting scene and image may include: searching for the target frame rate corresponding to the motion information of the current shooting scene and image in the first mapping relationship.

[0095] For example, in the first mapping relationship, we can find a target frame rate of 120 frames per second corresponding to the current shooting scene S1, the motion speed of the image in the image motion information is V1, and the size of the motion window is W1; we can also find a target frame rate of 30 frames per second corresponding to the current shooting scene S3, the motion speed of the image in the image motion information is V3, and the size of the motion window is W3; we can also find a target frame rate of 60 frames per second corresponding to the current shooting scene S5, the motion speed of the image in the image motion information is V5, and the size of the motion window is W5.

[0096] In this embodiment of the disclosure, during the process of responding to a shooting request, the image processing system of the electronic device outputs the aforementioned video stream, which may include a display preview stream and / or a video recording stream. The display preview stream is used for display on the screen, and the video recording stream is used for storage.

[0097] It should be noted that after obtaining the target frame rate, this embodiment of the disclosure can compare the current frame rate with the target frame rate, and perform frame rate adjustment if the current frame rate is lower than the target frame rate. Here, frame rate adjustment can involve interpolating frames into the video stream so that the interpolated frame rate is the target frame rate. In this way, the adjusted frame rate can be greater than the original frame rate, thereby improving the video clarity through frame rate adjustment.

[0098] In some embodiments, if the current frame rate is greater than or equal to the target frame rate, frame rate adjustment of the video stream is stopped.

[0099] In this embodiment of the disclosure, when the current frame rate is greater than or equal to the target frame rate, the current frame rate will not be adjusted, and the preview display and encoding storage can continue based on the current frame rate of the video stream, thus saving power consumption.

[0100] For example, if the target frame rate is 120 frames per second, corresponding to the current shooting scene being a soccer scene S1, the image's motion speed being V1, and the motion window size being W1, and the current frame rate is also 120, then the frame rate of the video stream will not be adjusted.

[0101] If the target frame rate is 30 frames per second (fps) corresponding to the current shooting scene being static (S3), the image motion speed being V3, and the motion window size being W3, and the current frame rate is also 30 fps, then the frame rate of the video stream will not be adjusted. This saves power consumption.

[0102] If the target frame rate is 60 frames per second (fps) for the current shooting scene (night video scene S4), motion speed (V4), and motion window size (W4), but the current frame rate is 15 fps, then frame rate compensation will be activated to adjust the video stream's frame rate to 60 fps. Simultaneously, the display's frame rate can also be adjusted to 60 fps for a smoother display.

[0103] If the target frame rate corresponding to the current shooting scene being a slow-motion scene (S5), a motion speed of (V5), and a motion window size of (W5) is 60 frames per second, and the current frame rate is 15, then frame rate compensation can be activated to adjust the frame rate of the video stream to 60 frames per second. For example, in a slow-motion scene, the frame rate of the recorded video stream can be adjusted from 15 frames per second to 60 frames per second to obtain more slow-motion images and improve the smoothness of video recording.

[0104] In some embodiments, the video stream includes a display preview stream and / or a video recording stream, and frame rate adjustment of the video stream based on a target frame rate includes:

[0105] Adjust the frame rate of the display preview stream and / or video recording stream based on the target frame rate.

[0106] In other words, the embodiments of this disclosure can not only adjust the frame rate of the display preview stream to make the preview image clearer, but also adjust the frame rate of the video recording stream, that is, adjust the encoding input frame rate, thereby making the encoded and stored video clearer.

[0107] In this embodiment of the disclosure, adjusting the frame rate of the display preview stream and the video recording stream based on the target frame rate may include: adjusting the frame rate of both the display preview stream and the video recording stream to the target frame rate through frame interpolation.

[0108] Here, electronic devices can also set the display frame rate of the screen to the target frame rate, so that the adjusted display preview stream can be displayed based on the adjusted display screen, thereby enabling the display screen to have a smoother display effect.

[0109] It should be noted that electronic devices can stop adjusting the frame rate of the video stream if the current frame rate is greater than or equal to the target frame rate; and adjust the frame rate of the video stream to the target frame rate if the current frame rate of the video stream is detected to be less than the target frame rate.

[0110] Of course, the electronic device can also adjust the frame rate of the video stream based on the target frame rate when it detects that the current frame rate of the video stream is different from the target frame rate. That is, when the current frame rate is different from the target frame rate, the frame rate of the video stream is adjusted to the target frame rate. In this embodiment of the disclosure, the different motion information of the current shooting scene and the image correspond to different frame rates required by the electronic device for preview display and video recording. For example, the frame rate required for a static scene is less than the frame rate required for a moving scene. As another example, the faster the image moves, the higher the frame rate required by the electronic device.

[0111] Therefore, the frame rate adjustment based on the motion information of the current shooting scene and the image takes into account the impact of the motion of the current shooting scene and the shooting object on the frame rate. This not only makes the frame rate adjustment more accurate, but also allows for frame rate adjustment based on different current shooting scenes, making the frame rate adjustment universal.

[0112] For example, if the current shooting scene is a football scene S1, then this embodiment of the present disclosure can adjust the frame rate for video recording of the football scene S1; if the current shooting scene is a motion scene S2, then this embodiment of the present disclosure can adjust the frame rate for video recording of the motion scene S2; if the current shooting scene is a static scene S3, then this embodiment of the present disclosure can adjust the frame rate for video recording of the static scene S3; if the current shooting scene is a night scene video scene S4, then this embodiment of the present disclosure can adjust the frame rate for video recording of the night scene video scene S4; if the current shooting scene is a slow motion scene S5, then this embodiment of the present disclosure can adjust the frame rate for video recording of the slow motion scene S5.

[0113] Furthermore, unlike existing methods where the frame rate cannot be adjusted, this embodiment of the present disclosure adjusts the frame rate during the response to a shooting request, enabling flexible frame rate adjustment and resulting in clearer recorded videos.

[0114] In some embodiments, determining the current shooting scene and motion information of the image based on image information includes:

[0115] The current shooting scene is determined based on the grayscale change information of pixels in the image and the brightness information of the environment in which the subject is located.

[0116] Based on image information, the motion speed of the image and the size of the motion window of the image are determined. The motion speed of the image is used to characterize the speed of movement of the photographed object in the image at least. The motion window is used to characterize the image area occupied by the photographed object in the image at least.

[0117] In this embodiment of the disclosure, obtaining the brightness information of the environment in which the subject is located may include: performing brightness analysis on the image information to obtain the brightness information of the environment in which the subject is located, and may also include: obtaining the brightness information of the environment in which the subject is located through a brightness sensor.

[0118] In this embodiment of the disclosure, the grayscale change information of a pixel may include: the grayscale difference of the pixel and the number of pixels that have undergone grayscale change. The grayscale difference of the pixel may be the grayscale difference between pixels at the same location in the currently acquired image and the previously captured image frame.

[0119] It should be noted that the differences in grayscale values ​​of pixels and the number of pixels exhibiting grayscale changes are both related to the movement of the subject being photographed. For example, the larger the grayscale difference of pixels, the faster the subject is moving, which helps determine whether the current shooting scene is a football scene or a static scene.

[0120] In this embodiment of the disclosure, the electronic device may store a second mapping relationship in advance, which points to the relationship between the brightness value of the environment, the grayscale difference of the pixels, the number of pixels that undergo grayscale changes, and the shooting scene.

[0121] The above method of determining the current shooting scene based on the grayscale change information of each pixel in the image and the brightness information of the environment in which the subject is located may include: searching for a shooting scene corresponding to the grayscale change information of each pixel in the image and the brightness information of the environment in which the subject is located in the second mapping relationship, and using the found shooting scene as the current shooting scene.

[0122] In some embodiments, determining the motion speed of the image and the size of the motion window of the image based on image information includes:

[0123] Determine the speed of motion in the image based on image information acquired by a contact camera;

[0124] The size of the motion window of the image is determined based on the image information acquired by the event camera.

[0125] In other words, electronic devices combine contact cameras and event cameras to determine the motion information of images. This not only simplifies computation but also enables more accurate frame rate adjustment based on image motion information.

[0126] In this embodiment of the disclosure, the camera module of the electronic device includes a contact camera and an event camera. The electronic device can capture images of the subject through the CIS of the contact camera to obtain image information captured by the contact camera, and can also capture images of the subject through the DVS of the event camera to obtain image information captured by the event camera.

[0127] It should be noted that the event camera is suitable for monitoring moving scenes. In some embodiments, the event camera can be used to record the intensity, angle, and position information of the imaging light. That is, the embodiments of this disclosure can determine the size of the motion window of the image based on the intensity, angle, and position information in the image information acquired by the event camera.

[0128] For example, in a soccer scene, an electronic device can first obtain multiple pixels in the image corresponding to the moving soccer ball based on the event camera, then use the area enclosed by the multiple pixels as a motion window, and then obtain the size of the motion window by the coordinates of the pixels enclosed by the motion window.

[0129] In some embodiments, determining the motion speed of an image based on image information acquired by a contact camera includes:

[0130] Based on image information acquired by a contact camera, the grayscale difference of each pixel in the image and the gradient value corresponding to each pixel are obtained.

[0131] The initial motion speed of each pixel is determined based on the gray-level difference and gradient value.

[0132] The initial motion speed of each pixel is adjusted to obtain the target motion speed of each pixel;

[0133] The target motion speed of each pixel is filtered to obtain the motion speed of the image.

[0134] The above-mentioned method of obtaining the grayscale difference of each pixel in the image may include: performing difference processing on the grayscale of the same pixel in the image acquired by the contact camera and the previous image frame to obtain the grayscale difference of each pixel in the image.

[0135] The above-mentioned method of obtaining the gradient value corresponding to each pixel can include: converting the image information acquired by the contact camera to grayscale to obtain a grayscale image, and then performing convolution processing on the grayscale image to obtain the gradient value corresponding to each pixel in the image.

[0136] The above method of determining the initial motion speed of each pixel based on grayscale difference and gradient value may include: obtaining the initial motion speed of each pixel based on the ratio between the grayscale difference and the corresponding gradient value of each pixel.

[0137] In this embodiment of the disclosure, after obtaining the initial motion speed of each pixel, it is necessary to adjust the initial motion speed of each pixel to obtain the target motion speed of each pixel, so as to improve the accuracy of speed calculation.

[0138] In some embodiments, adjusting the initial motion speed of each pixel to obtain the target motion speed of each pixel includes:

[0139] The initial motion speed of each pixel is scaled down and adjusted to obtain the adjustment speed of each pixel.

[0140] The adjustment speed of each pixel is filtered to obtain the filtered speed of each pixel.

[0141] The filtering speed of each pixel is weighted to obtain the target motion speed of each pixel.

[0142] The above-mentioned adjustment of the initial motion speed of each pixel to obtain the adjusted speed of each pixel may include: compressing the grayscale of each pixel to a preset grayscale range to reduce the corresponding initial motion speed to obtain the adjusted speed. This can remove noise and improve the accuracy of speed calculation.

[0143] The above-mentioned filtering process for the adjustment speed of each pixel to obtain the filtered speed of each pixel may include: using operators in multiple directions to filter the adjustment speed of each pixel to obtain multiple filtered values, taking the minimum value among the multiple filtered values ​​as the filtered speed of that pixel, and thus obtaining the filtered speed of each pixel.

[0144] The above-mentioned weighted processing of the filtering speed of each pixel to obtain the target motion speed of each pixel may include: obtaining the weight value of each pixel; and obtaining the weighted target motion speed of each pixel based on the filtering speed and the weight value of each pixel.

[0145] It should be noted that, in the embodiments of this disclosure, the weight value of each pixel can be determined based on the image information collected by the event camera, so that the weight value of the pixel corresponding to the shooting object is greater than the weight value of the pixel corresponding to the shooting background.

[0146] Of course, the embodiments of this disclosure can also identify the image information acquired by the contact camera and determine the weight value of each pixel, so that the weight value of the pixel corresponding to the shooting object is greater than the weight value of the pixel corresponding to the shooting background.

[0147] In this embodiment of the disclosure, after obtaining the target motion speed of each pixel, the target motion speed of each pixel can be filtered to obtain the motion speed of the image.

[0148] In some embodiments, filtering the target motion speed of each pixel to obtain the motion speed of the image includes:

[0149] Select a subset of target motion velocities from multiple target motion velocities across multiple pixels;

[0150] The average speed of a subset of moving targets is taken as the speed of motion in the image.

[0151] Among these, the smallest target speed is greater than all other target speeds except for some of the target speeds.

[0152] In this embodiment of the disclosure, the smallest target movement speed among the partial target movement speeds is greater than all target movement speeds other than the partial target movement speeds among the multiple target movement speeds. That is, if the multiple target movement speeds are arranged in descending order to form a queue, the partial target movement speed is the target movement speed at the head of the queue.

[0153] It should be noted that the embodiments of this disclosure can select a portion of the target motion speeds from multiple target motion speeds by histogram transformation, or by queue sorting, and the embodiments of this disclosure do not limit this.

[0154] Of course, in the embodiments of this disclosure, one of the target motion speeds can also be used as the motion speed of the image. For example, if the partial target motion speed includes two target motion speeds, either of the two target motion speeds can be used as the motion speed of the image; if the partial target motion speed includes three or more target motion speeds, the target motion speed with the middle speed value among the partial target motion speeds can be used as the motion speed of the image.

[0155] It should be noted that the ratio between the number of partial target movement velocities and the number of all target movement velocities is n%, and n can be set according to the actual situation. For example, n can be set between 1 and 10. This disclosure is an embodiment and does not limit it.

[0156] In this embodiment of the disclosure, the target frame rate is set by combining the fastest target motion speed among the target motion speeds of each pixel in the image. Therefore, in this embodiment of the disclosure, after obtaining the target motion speed of each pixel, the average value of some target motion speeds is used as the motion speed of the image, which can make the motion speed of the image more accurate.

[0157] To better understand the frame rate adjustment method of the above-described multiple embodiments, the following are examples of embodiments of this disclosure:

[0158] Figure 2 This is a schematic diagram of a frame rate adjustment module according to an exemplary embodiment. Figure 2 As shown, an electronic device for frame rate adjustment may include: a camera module, processing circuitry, and a display screen. The camera module includes contact cameras and event cameras; the processing circuitry includes: an image processing system, a frame rate compensation module, a video encoding module, a display processing engine, and a storage module.

[0159] The image processing system connects a camera module, a frame rate compensation module, and a display processing engine to output a video stream; the frame rate compensation module is used to adjust the frame rate of the video stream based on a target frame rate.

[0160] Figure 3 This is a schematic diagram illustrating an image processing system according to an exemplary embodiment. For example... Figure 3 As shown, the image processing system includes a motion detection module, which connects a contact camera and an event camera. This module is used to determine the motion information of the current shooting scene and the image based on the image information acquired by the contact camera and the image information acquired by the event camera.

[0161] Figure 4 This is a schematic diagram illustrating a motion detection module according to an exemplary embodiment. Figure 4 As shown, the motion detection module includes: a scene recognition module, a pixel motion calculation module, and an image motion calculation module;

[0162] The scene recognition module is used to determine the current shooting scene based on the grayscale change information of pixels in the image and the brightness information of the environment in which the subject is located;

[0163] The pixel motion calculation module is used to obtain the gray-level change difference and the gradient value of each pixel in the image; based on the gray-level change difference and gradient value, the initial motion speed of each pixel is determined; the initial motion speed of each pixel is adjusted to obtain the target motion speed of each pixel.

[0164] The image motion calculation module is used to select a portion of the target motion velocities from multiple target motion velocities of multiple pixels, and take the average value of the portion of target motion velocities as the motion velocity of the image.

[0165] Figure 5 This is a schematic diagram illustrating a frame rate compensation module according to an exemplary embodiment. For example... Figure 5 As shown, the frame rate compensation module is connected to the display frame rate control module and the video encoding module.

[0166] The frame rate compensation module is used to adjust the frame rate of the display preview stream and the video recording stream based on the target frame rate, so as to obtain the frame rate adjusted display preview stream and the frame rate adjusted video recording stream.

[0167] The video encoding module connects the frame rate compensation module and the storage module, and is used to encode the video recording stream after frame rate adjustment for input to the storage module.

[0168] The display frame rate control module, located in the display processing engine, is used to adjust the display frame rate of the screen.

[0169] This disclosure also proposes a frame rate adjustment device. Figure 6 This is a block diagram illustrating a frame rate adjustment device according to an exemplary embodiment. Figure 6 As shown, the frame rate adjustment device 1000 mainly includes:

[0170] The image acquisition module 1001 is configured to acquire image information of the image currently captured by the camera in response to a shooting request;

[0171] The motion determination module 1002 is configured to determine the current shooting scene and the motion information of the image based on the image information; the motion information of the image is at least used to characterize the motion of the shooting object in the image;

[0172] The adjustment module 1003 is configured to adjust the frame rate based on the shooting scene and the motion information.

[0173] In some embodiments, the motion determination module 1002 includes:

[0174] The scene determination module is configured to determine the current shooting scene based on the grayscale change information of pixels in the image and the brightness information of the environment in which the shooting object is located;

[0175] The speed determination module is configured to determine the motion speed of the image and the size of the motion window of the image based on the image information. The motion speed of the image is used at least to characterize the speed of movement of the photographed object in the image; the motion window is used at least to characterize the image area occupied by the photographed object in the image.

[0176] In some embodiments, the speed determination module is configured to determine the motion speed of the image based on image information acquired by a contact camera, and to determine the size of the motion window of the image based on image information acquired by an event camera.

[0177] In some embodiments, the speed determination module is configured to: acquire the grayscale difference of each pixel in the image and the gradient value corresponding to each pixel based on the image information acquired by the contact camera; determine the initial motion speed of each pixel based on the grayscale difference and the gradient value; adjust the initial motion speed of each pixel to obtain the target motion speed of each pixel; and filter the target motion speed of each pixel to obtain the motion speed of the image.

[0178] In some embodiments, the speed determination module is configured to reduce and adjust the initial motion speed of each pixel to obtain the adjusted speed of each pixel; filter the adjusted speed of each pixel to obtain the filtered speed of each pixel; and perform weighted processing on the filtered speed of each pixel to obtain the target motion speed of each pixel.

[0179] In some embodiments, the speed determination module is configured to select a portion of the target motion speeds from a plurality of target motion speeds of the plurality of pixels; and to take the average value of the portion of the target motion speeds as the motion speed of the image; wherein the smallest of the portion of the target motion speeds is greater than all the target motion speeds other than the portion of the plurality of target motion speeds.

[0180] In some embodiments, the adjustment module 1003 is configured to determine a target frame rate corresponding to the motion information of the current shooting scene and the image based on the motion information of the current shooting scene and the image; and to adjust the frame rate of the video stream based on the target frame rate if the current frame rate of the video stream is detected to be less than the target frame rate.

[0181] In some embodiments, the apparatus further includes:

[0182] The stop adjustment module is configured to stop adjusting the frame rate of the video stream when the current frame rate is greater than or equal to the target frame rate.

[0183] In some embodiments, the adjustment module 1003 is configured to adjust the frame rate of the display preview stream and / or the video recording stream based on the target frame rate.

[0184] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0185] Figure 7 This is a structural block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 700 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0186] Reference Figure 7 The electronic device 700 may include one or more of the following components: processing component 702, memory 704, power supply component 706, multimedia component 708, audio component 710, input / output (I / O) interface 712, sensor component 714, and communication component 716.

[0187] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 702 may include one or more processors 720 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0188] Memory 704 is configured to store various types of data to support operation on electronic device 700. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, and videos. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0189] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0190] Multimedia component 708 includes a screen that provides an output interface between electronic device 700 and user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When electronic device 700 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0191] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0192] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0193] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 may detect the on / off state of electronic device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or one of its components, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of electronic device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0194] Communication component 716 is configured to facilitate wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0195] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0196] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including executable instructions or a computer program, which can be executed by a processor 720 of an electronic device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0197] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the frame rate adjustment methods described in the embodiments of this disclosure. For example, the method includes:

[0198] In response to a shooting request, obtain image information of the image currently being captured by the camera;

[0199] Based on the image information, the current shooting scene and the motion information of the image are determined; the motion information of the image is used to characterize at least the motion of the shooting object in the image;

[0200] Frame rate adjustment is performed based on the current shooting scene and the motion information of the image.

[0201] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the frame rate adjustment methods described above in this disclosure.

[0202] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0203] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A frame rate adjustment method, characterized in that, include: In response to a shooting request, obtain image information of the image currently being captured by the camera; Based on the image information, the current shooting scene and the motion information of the image are determined; the motion information of the image is used to characterize at least the motion of the shooting object in the image; Frame rate adjustment is performed based on the current shooting scene and the motion information of the image.

2. The method according to claim 1, characterized in that, The step of determining the current shooting scene and the motion information of the image based on the image information includes: Based on the grayscale change information of the pixels in the image and the brightness information of the environment in which the subject is located, the current shooting scene is determined; Based on the image information, the motion speed of the image and the size of the motion window of the image are determined. The motion speed of the image is used to characterize at least the speed of movement of the photographed object in the image; the motion window is used to characterize at least the image area occupied by the photographed object in the image.

3. The method according to claim 2, characterized in that, Determining the motion speed of the image and the size of the motion window of the image based on the image information includes: Based on the image information acquired by the contact camera, the motion speed of the image is determined; The size of the motion window of the image is determined based on the image information acquired by the event camera.

4. The method according to claim 3, characterized in that, Determining the motion speed of the image based on the image information acquired by the contact camera includes: Based on the image information acquired by the contact camera, the grayscale difference of each pixel in the image and the gradient value corresponding to each pixel are obtained; Based on the grayscale difference and the gradient value, the initial motion speed of each pixel is determined; The initial motion speed of each pixel is adjusted to obtain the target motion speed of each pixel; The target motion speed of each pixel is filtered to obtain the motion speed of the image.

5. The method according to claim 4, characterized in that, The step of adjusting the initial motion speed of each pixel to obtain the target motion speed of each pixel includes: The initial motion speed of each pixel is reduced and adjusted to obtain the adjustment speed of each pixel; The adjustment speed of each pixel is filtered to obtain the filtered speed of each pixel. The filtering speeds of each pixel are weighted to obtain the target motion speed of each pixel.

6. The method according to claim 4, characterized in that, The step of filtering the target motion speed of each pixel to obtain the motion speed of the image includes: Select a portion of the target motion velocities from the multiple target motion velocities of the multiple pixels; The average value of the motion speed of a portion of the target is taken as the motion speed of the image; Among these, the smallest of the target motion speeds is greater than all the target motion speeds other than some of the target motion speeds.

7. The method according to any one of claims 1 to 6, characterized in that, The frame rate adjustment based on the current shooting scene and the motion information of the image includes: Based on the current shooting scene and the motion information of the image, determine the target frame rate corresponding to the current shooting scene and the motion information of the image; If the current frame rate of the video stream is detected to be lower than the target frame rate, the frame rate of the video stream is adjusted based on the target frame rate.

8. The method according to claim 7, characterized in that, The method further includes: If the current frame rate is greater than or equal to the target frame rate, stop adjusting the frame rate of the video stream.

9. The method according to claim 7, characterized in that, The video stream includes a display preview stream and / or a video recording stream, and the step of adjusting the frame rate of the video stream based on the target frame rate includes: Based on the target frame rate, the frame rate of the display preview stream and / or the video recording stream is adjusted.

10. A frame rate adjustment device, characterized in that, include: The image acquisition module is configured to acquire image information of the currently captured image in response to a shooting request; The motion determination module is configured to determine the current shooting scene and the motion information of the image based on the image information; the motion information of the image is used to characterize at least the motion of the shooting object in the image; The adjustment module is configured to adjust the frame rate based on the shooting scene and the motion information.

11. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 9.

Images