Generation device and method of moving image, equipment and storage medium
By downsampling and upsampling of moving images, focusing on capturing large-scale motion changes, the problem of poor motion detection effect in the prior art is solved, accuracy and robustness are improved, and hardware power consumption is saved.
Patent Information
- Application Number
- CN202411275396.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-31
AI Technical Summary
In image processing, the existing motion detection methods are small in the local pixel point range, resulting in poor motion detection effect, which is prone to discontinuity, missed detection, and missed detection problems. In addition to pixel point calculations, hardware power consumption will be increased.
By downsampling and upsampling the moving images, an accurate target moving image is generated, ignoring small-scale details changes, focusing on capturing large-scale motion changes, avoiding increasing pixel point calculations, thereby saving hardware power consumption.
It improves the robustness and accuracy of moving images, avoids problems such as discontinuity and missed detection of motion judgments, and reduces hardware costs.
Smart Images
Figure CN119228838A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a motion image generation device, method, equipment and storage medium. Background Art
[0002] Motion detection is a commonly used technology in the field of image processing technology. In the image processor (Image Signal Processor, ISP), the pixels of the current frame and the previous frame are compared to determine whether there is relative motion between the two frames. The detection results are often used in applications such as time domain filtering and multi-frame fusion. Specifically, when performing motion detection, local pixels at corresponding positions of the previous and next frames are usually used to determine whether there is motion. For example, a 5x5 or 7x7 pixel block around the corresponding pixel. However, since this motion detection method only uses small local pixel blocks, the motion detection effect is poor, some motions cannot be detected, and it is prone to problems such as discontinuous motion judgment, missed detection, and false detection.
[0003] In order to solve the above technical problems, the related technology increases the number of local pixels, that is, large-scale motion detection. The larger the range of pixels used in motion detection, the larger the scale, the more accurate the detection result, and the higher the robustness. Usually, depending on the size of the image and the noise intensity, hundreds of pixels are needed to make accurate motion judgments more stably.
[0004] However, since an ISP is usually a semiconductor integrated circuit, the implementation of the above motion detection method requires consumption of hardware resources. Therefore, when more local pixels are required for motion detection, the power consumption of the hardware will increase. Summary of the invention
[0005] The embodiments of the present application provide a motion image generation device, method, equipment and storage medium, which can obtain a target motion image with high accuracy by downsampling and upsampling the motion image without increasing the calculation of pixel points for motion detection, thereby saving power consumption of hardware. The technical solution is as follows:
[0006] According to a first aspect of an embodiment of the present application, a device for generating a moving image is provided, the device comprising:
[0007] A motion detection module, a down-sampling module, an up-sampling module and a generation module electrically connected in sequence; the motion detection module is also electrically connected to the generation module;
[0008] The upsampling module is used to obtain the N-1th downsampled image; upsample the N-1th downsampled image to obtain the N-1th upsampled image; the N-1th downsampled image is obtained by the downsampling module downsampling the N-1th motion image; the N-1th motion image is obtained by the motion detection module detecting the N-1th image and the N-2th image; N is an integer greater than or equal to 3;
[0009] The generation module is used to filter the N-1th frame upsampled image based on the Nth frame image to obtain the N-1th frame target motion image; the N-1th frame target motion image includes edge information of the Nth frame image; the N-1th frame target motion image represents the motion state of the object in the Nth frame image.
[0010] In a possible implementation, the up-sampling module is further used to obtain a data reading instruction;
[0011] The N-1th frame of down-sampled image is read based on the data reading instruction.
[0012] In a possible implementation, the device further includes a data reading module;
[0013] The data reading module is electrically connected to the up-sampling module;
[0014] The data reading module is used to obtain the data reading instruction; read the N-1th frame of down-sampled image based on the data reading instruction; and send the N-1th frame of down-sampled image to the up-sampling module.
[0015] In a possible implementation, the device further includes a memory, which is electrically connected to the down-sampling module and the data reading module respectively;
[0016] The down-sampling module is used to store the N-1th frame of down-sampled image in the memory;
[0017] The data reading module is used to read the N-1th frame of down-sampled image from the memory based on the data reading instruction.
[0018] In a possible implementation, the motion detection module is used to obtain a plurality of first image regions from the N-1th frame image based on a preset window; obtain a plurality of second image regions from the N-2th frame image based on the preset window; the coordinate range of the first image region is the same as the coordinate range of the corresponding second image region; the coordinate system corresponding to the N-2th frame image and the N-1th frame image is a preset coordinate system;
[0019] Based on each of the first image regions and the corresponding second image region, a sum of absolute differences is calculated; the sum of absolute differences indicates whether there is motion between the corresponding first image region and the corresponding second image region;
[0020] Based on the absolute difference sum corresponding to each of the first image regions, the N-1th frame of motion image is obtained.
[0021] In a possible implementation, the motion detection module is further used to:
[0022] When the absolute difference sum is greater than a preset threshold, the absolute difference sum indicates that there is motion between the first image area and the corresponding second image area; in the N-1th frame of motion image, the first partial area is marked as a motion area; the coordinate range of the first partial area is the same as the coordinate range of the first image area; or
[0023] When the absolute value sum is less than or equal to the preset threshold, the absolute difference sum indicates that there is no motion between the first image area and the corresponding second image area; in the N-1th frame motion image, the first partial area is marked as a non-motion area.
[0024] In a possible implementation, the ratio of the N-1th down-sampled image frame to the Nth image frame is greater than 4 / 1.
[0025] In a possible implementation manner, the size of the N-1th upsampled image is the same as the size of the Nth frame image.
[0026] In a possible implementation, the generating module is used to perform guided filtering on the N-1th frame up-sampled image based on the Nth frame image to obtain the N-1th frame target motion image.
[0027] According to a second aspect of an embodiment of the present application, a method for generating a moving image is provided, the method comprising:
[0028] Acquire the N-1th down-sampled image; up-sample the N-1th down-sampled image to obtain the N-1th up-sampled image; the N-1th down-sampled image is obtained by down-sampling the N-1th motion image by the down-sampling module; the N-1th motion image is obtained by detecting the N-1th image and the N-2th image by the motion detection module; N is an integer greater than or equal to 3;
[0029] The N-1th frame upsampled image is filtered based on the Nth frame image to obtain the N-1th frame target motion image; the N-1th frame target motion image includes edge information of the Nth frame image; the N-1th frame target motion image represents the motion state of the object in the Nth frame image.
[0030] According to a third aspect of an embodiment of the present application, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store at least one program, and the at least one program is loaded by the processor and executes the above-mentioned method for generating a motion image.
[0031] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one program is stored. The at least one program is loaded and executed by a processor to implement the above-mentioned method for generating a motion image.
[0032] In an embodiment of the present application, an apparatus for generating a motion image is provided, wherein the motion image is downsampled by a downsampling module to obtain a downsampled image, the downsampled image is upsampled to obtain an upsampled image, and the upsampled image is filtered based on the Nth frame image to obtain the N-1th frame of target motion image including the edge information of the Nth frame image, thereby improving the robustness and accuracy of the motion image. The downsampled image is more focused on capturing large-scale motion changes, while ignoring small-scale detail changes, thereby avoiding problems such as discontinuity, missed detection, and false detection in motion judgment. In addition, compared to the related art, the accuracy of motion judgment is improved by increasing the size of the preset window, and the embodiment of the present application does not need to increase the pixel point calculation for motion detection, thereby saving hardware power consumption and reducing hardware costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0034] Figure 1 is a schematic diagram of an implementation environment provided according to an embodiment of the present application;
[0035] Figure 2 is a structural schematic diagram of a motion image generation device provided according to an embodiment of the present application;
[0036] Figure 3 is a structural schematic diagram of a first image area provided according to an embodiment of the present application;
[0037] Figure 4 is a schematic structural diagram of a second image area provided according to an embodiment of the present application;
[0038] Figure 5 It is a schematic diagram of a process for determining whether there is motion between two adjacent frames of images provided according to an embodiment of the present application;
[0039] Figure 6 A schematic diagram of generating an N-1th frame of a target motion image according to an embodiment of the present application;
[0040] Figure 7 is a schematic diagram of another structure of a motion image generating device provided in an embodiment of the present application;
[0041] Figure 8 is a flow chart of a method for generating a moving image provided according to an embodiment of the present application;
[0042] Fig. 9 is a schematic diagram of the structure of a terminal provided according to an embodiment of the present application;
[0043] Fig.10 It is a structural diagram of a server provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0044] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0045] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.
[0046] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limitation on the quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms.
[0047] These terms are only used to distinguish one element from another element. For example, without departing from the scope of various examples, a first action can be referred to as a second action, and similarly, a second action can also be referred to as a first action. Both the first action and the second action can be actions, and in some cases, can be separate and different actions.
[0048] Here, at least one means one or more than one, for example, at least one action can be one action, two actions, three actions, or any other action that is an integer greater than or equal to one. And multiple means two or more than two, for example, multiple actions can be two actions, three actions, or any other action that is an integer greater than or equal to two.
[0049] Figure 1 It is a schematic diagram of an implementation environment provided according to an embodiment of the present application, and the implementation environment may include a terminal 101 and a server 102.
[0050] In the terminal 101, an ISP is provided, so as to perform motion detection on the image based on the ISP.
[0051] The terminal 101 may be a smart phone with an ISP, a wearable device, a personal computer, a laptop, a tablet computer, a smart TV, a car terminal, etc.
[0052] The server 102 may be a single server, a server cluster consisting of multiple servers, or a cloud processing center.
[0053] The terminal 101 is connected to the server 102 via a wired or wireless network.
[0054] In some embodiments, the wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, private network or virtual private network. In some embodiments, the data exchanged through the network is represented by technology and / or format including HyperText Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above data communication technology.
[0055] The embodiment of the present application provides a motion image generation device. Based on the motion image generation device, a motion image of two adjacent frames of images is generated. The motion image indicates whether there is motion between the two adjacent frames of images. The device involves a motion detection module, a downsampling module, an upsampling module and a generation module.
[0056] Exemplarily, a motion detection module is used to perform motion detection on two adjacent frames of images to obtain a motion image corresponding to the two adjacent frames of images. A downsampling module is used to downsample the image to obtain a downsampled image. The downsampled image reduces the number of pixels in the image, thereby reducing the resolution. This means that the spatial range represented by each pixel in the downsampled image becomes larger. An upsampling module is used to restore the downsampled image to the original resolution or a higher resolution. A generation module is used to filter the image to obtain a filtered image.
[0057] In some embodiments, the motion detection module is electrically connected to the downsampling module. The downsampling module is used to downsample the motion image to obtain a downsampled image. Specifically, the downsampling module selects pixels at a certain interval to obtain an image with a reduced size, that is, a downsampled image.
[0058] In the embodiment of the present application, when determining whether there is motion between two adjacent frames of images, the downsampled image is more focused on capturing large-scale motion changes. Such as the overall movement and rotation of an object, while ignoring small-scale detail changes. Therefore, motion patterns that may appear complex due to slight differences between pixels will become smoother and simpler in the downsampled image. This smoothing effect describes the motion state between two adjacent frames of images on a larger scale. Therefore, in the embodiment of the present application, it is not necessary to increase the size of the preset window to achieve a description of the motion state between two adjacent frames of images on a larger scale.
[0059] In some embodiments, the down-sampling module is electrically connected to the up-sampling module. The up-sampling module is used to up-sample the down-sampled image to obtain an up-sampled image. The up-sampling module increases the number of pixels or increases the sampling rate of pixels so as to recover higher resolution or denser pixels from lower resolution or fewer pixels. That is, the resolution of the up-sampled image is greater than the resolution of the down-sampled image. The up-sampled image is an image after the down-sampled image is enlarged.
[0060] In some embodiments, the generation module is electrically connected to the up-sampling module. The generation module is used to filter the up-sampling module to obtain the target motion image.
[0061] For example, Figure 2 2 is a schematic diagram of a motion image generation device 200 provided according to an embodiment of the present application, the device comprising:
[0062] The motion detection module 201 , the down-sampling module 202 , the up-sampling module 203 and the generation module 204 are electrically connected in sequence; the motion detection module 201 is also electrically connected to the generation module 204 .
[0063] Among them, the motion detection module 201, the down-sampling module 202, the up-sampling module 203 and the generation module 204 can be integrated inside the ISP. The ISP is implemented based on a semiconductor integrated circuit. The motion detection module 201, the down-sampling module 202, the up-sampling module 203 and the generation module 204 are also implemented based on a semiconductor integrated circuit.
[0064] In some embodiments, the motion detection module 201 is used to obtain the N-2 frame image and the N-1 frame image, perform motion detection on the N-2 frame image and the N-1 frame image, and obtain the N-1 frame motion image. The N-1 frame motion image represents the motion state of the object between the N-2 frame image and the N-1 frame image. The motion state indicates whether the object has moved.
[0065] The motion detection module 201 obtains the N-2 frame image and the N-1 frame image from an external device. The external device is electrically connected to the apparatus. Optionally, the external device is used to store the N-2 frame image and the N-1 frame image or dynamically generate the N-2 frame image and the N-1 frame image.
[0066] Figure 3 It is a structural schematic diagram of a first image area provided according to an embodiment of the present application. Figure 4 It is a structural schematic diagram of a second image area provided according to an embodiment of the present application.
[0067] Combine the following Figure 3 and Figure 4 Motion detection is exemplified.
[0068] In some embodiments, the motion detection module is used to obtain multiple first image areas from the N-1 frame image based on a preset window; obtain multiple second image areas from the N-2 frame image based on the preset window; the coordinate range of the first image area is the same as the corresponding coordinate range of the second image area; the coordinate system corresponding to the N-2 frame image and the N-1 frame image is a preset coordinate system.
[0069] In one embodiment, the motion detection module obtains a preset window. The preset window slides on the N-1th frame image based on the sliding step length, and each time it slides, a first image area on the N-1th frame image is reselected. The preset window slides on the N-2th frame image based on the sliding step length, and each time it slides, a second image area on the N-2th frame image is reselected.
[0070] By processing each first image region and the corresponding second image region, it can be determined whether there is motion between the N-2th frame image and the N-1th frame image. Optionally, the size of the preset window is determined according to actual needs. The sliding step size is also determined according to actual needs.
[0071] For example, the size of the preset window is 5×5. That is, when the preset window slides on the N-1th frame image, it can cover a first image area on the N-1th frame image, that is, 25 pixels.
[0072] Figure 3 Ai in represents the i-th pixel point in the first image area. Figure 4 Bi in represents the i-th pixel point of a second image area. i is an integer greater than or equal to 1 and less than or equal to 25.
[0073] In one example, the motion detection module determines the central pixel of the first image area, and then extracts the pixels around the central pixel to obtain the pixel value of each pixel in the first image area. 13 is the central pixel of the first image area.
[0074] In the related art, the accuracy of the motion image is improved by increasing the size of the preset window. However, the larger the size of the preset window, the larger the size of the first image area and the second image area, that is, the more pixels the first image area and the second image area include. Therefore, when calculating the first image area and the second image area, more pixels need to be calculated, the corresponding calculation logic is more complex, the hardware power consumption is greater, and the larger the hardware area used to store the calculation logic, the higher the hardware manufacturing cost. The embodiment of the present application does not need to increase the size of the preset window. Compared with the related art, it saves computing power, reduces hardware power consumption, and reduces the hardware area used to store the calculation logic, thereby reducing the hardware manufacturing cost.
[0075] Figure 5 It is a flowchart of determining whether there is motion between two adjacent frames of images provided according to an embodiment of the present application.
[0076] Combine the following Figure 5 An exemplary description is given of determining whether there is motion between two adjacent image frames.
[0077] In some embodiments, whether there is motion between the N-1th frame image and the N-2th frame image is determined by determining whether there is motion between each first image region and the second image region.
[0078] Specifically, based on each of the first image areas and the corresponding second image areas, the sum of absolute differences is calculated; the sum of absolute differences indicates whether there is motion between the first image area and the corresponding second image area; based on the sum of absolute differences corresponding to each of the first image areas, the N-1th frame of motion image is obtained.
[0079] In one example, for a first image region and a corresponding second image region, a sum of absolute differences is obtained based on each first pixel and the corresponding second pixel. The first pixel is a pixel of the first image region. The second pixel is a pixel of the second image region. The coordinates of the first pixel are the same as the coordinates of the corresponding second pixel.
[0080] To calculate the SAD, first calculate the absolute difference. The absolute difference is the absolute value of the difference. The difference is the difference between the pixel value of the first pixel and the pixel value of the corresponding second pixel. The SAD is the sum of multiple absolute differences (Sum of Absolute Difference, SAD) corresponding to the first image area and the corresponding second image area.
[0081] Specifically, SAD can be calculated by the following formula:
[0082]
[0083] In one example, the motion detection module is further used for: when the absolute difference sum is greater than a preset threshold, the absolute difference sum indicates that there is motion between the first image area and the corresponding second image area; in the N-1th frame motion image, the first partial area is marked as a motion area; the coordinate range of the first partial area is the same as the coordinate range of the first image area; or
[0084] When the absolute value sum is less than or equal to the preset threshold, the absolute difference sum indicates that there is no motion between the first image area and the corresponding second image area; in the N-1th frame motion image, the first partial area is marked as a non-motion area.
[0085] For example, when the SAD of the first image area is greater than a preset threshold, in the N-1th frame of the motion image, the motion detection module sets the pixel value of each pixel in the first partial area to 1. That is, the first partial area is the foreground area.
[0086] For another example, when the SAD of the first image area is less than or equal to the preset threshold, in the N-1th frame of the motion image, the motion detection module sets the pixel value of each pixel in the first partial area to 0. That is, the second partial area is the background area.
[0087] In one example, depending on the sliding step length, there may be an overlapping area between two adjacent first image areas, which may result in both a motion area and a non-motion area existing in a first partial area.
[0088] In order to solve the above problem, a smaller preset window can be selected to further determine the SAD of each sub-region of the first partial region, thereby determining the motion of each sub-region, so as to improve the accuracy of determining whether the first partial region is a motion region or a non-motion region, thereby improving the accuracy of the N-1th frame motion image. For example, the size of the smaller preset window is 3×3, etc.
[0089] In one example, for the N-1th frame image, the motion detection module slides a preset window once to determine a first image area; for the N-2th frame image, the motion detection module slides a preset window once to determine a second image area; each first pixel in the first image area and each second pixel in the second image are obtained, and the SAD is calculated based on each first pixel and the corresponding second pixel.
[0090] Determine whether the SAD is greater than a preset threshold; if so, there is motion between the first image area and the corresponding second image area. If not, there is no motion between the first image area and the corresponding second image area. Determine whether the N-1th frame image and the N-2th frame image have been traversed; if so, the motion detection module ends the operation. If not, the motion detection module continues to slide the preset window.
[0091] Figure 6 A schematic diagram of generating an N-1th frame of a target motion image is provided according to an embodiment of the present application.
[0092] Combine the following Figure 6 The generation of the target motion image of the N-1th frame is exemplified.
[0093] In some embodiments, the upsampling module is used to obtain the N-1th downsampled image; upsample the N-1th downsampled image to obtain the N-1th upsampled image; the N-1th downsampled image is obtained by the downsampling module downsampling the N-1th motion image; the N-1th motion image is obtained by the motion detection module detecting the N-1th frame image and the N-2th frame image; N is an integer greater than or equal to 3.
[0094] In some embodiments, in order to ensure the large-scale property of the downsampled image, the motion image is scaled down as much as possible.
[0095] In one example, the downsampling module first determines the downsampling multiple S, and divides the N-1 frame of the motion image into multiple S×S windows. If the size of the motion image is not an integer multiple of S, appropriate cropping or padding is required. For each window, the mean of all pixels in it is calculated, and the value is used as the pixel value of the corresponding pixel in the new image. Based on the calculated pixel values, the N-1 frame of the downsampled image with a size of (M / S)×(N / S) is generated.
[0096] Wherein, M is the width of the N-1th frame of the moving image, N is the height of the N-1th frame of the moving image, and S is the downsampling multiple.
[0097] For example, in order to obtain a smoother downsampled image, downsampling is performed using methods such as bilinear downsampling or bicubic downsampling.
[0098] For example, the ratio of the N-1th frame down-sampled image to the Nth frame image is greater than 4 / 1; that is, S is greater than or equal to 4.
[0099] In the related art, whether there is motion between two adjacent frames of images is determined by local areas of the two adjacent frames of images, and the motion detection effect is poor. However, the embodiments of the present application improve the effect of motion detection by downsampling the motion image and determining the motion on a larger scale. In addition, the present application does not need to increase the calculation of pixel points, saving computing power.
[0100] In one example, the upsampling module obtains the target size, which is also the size of the upsampled image of the N-1th frame; for a target pixel in the upsampled image of the N-1th frame, the corresponding interpolation point is found in the downsampled image of the N-1th frame, and several nearest pixel points around the interpolation point are determined, and several nearest pixel points around the interpolation point are calculated to obtain the pixel value of the target pixel point, and the pixel value is filled into the upsampled image of the N-1th frame. Similarly, the pixel value of each target pixel in the upsampled image of the N-1th frame can be obtained, and then the upsampled image of the N-1th frame can be obtained.
[0101] For example, upsampling is performed using methods such as bilinear upsampling or bicubic upsampling.
[0102] In some embodiments, the generation module is used to filter the N-1th frame upsampled image based on the Nth frame image to obtain the N-1th frame target motion image; the N-1th frame target motion image includes edge information of the Nth frame image; the N-1th frame target motion image represents the motion state of the object in the Nth frame image.
[0103] In one example, in order to perform guided filtering on the N-1th upsampled image directly based on the Nth frame image without processing the Nth frame image, the size of the N-1th upsampled image frame is the same as that of the Nth frame image.
[0104] In an example, the generating module is used to perform guided filtering on the N-1th frame up-sampled image based on the Nth frame image to obtain the N-1th frame target motion image.
[0105] The Nth frame image is used as the guide image.
[0106] In one example, multiple local windows are preset, and there is a linear relationship between the Nth frame image and the N-1th frame target motion image in each local window. That is, the pixel value k of a pixel point in the N-1th frame target motion image can be linearly represented by the pixel value m of a pixel point at the corresponding position in the Nth frame image. That is, k = a×m+b.
[0107] In order to determine the coefficients a and b corresponding to a local window, the difference between the target motion image of the N-1th frame and the upsampled image of the N-1th frame is minimized, and the coefficients a and b are obtained by solving the linear regression in combination with the constraints of the linear relationship. Similarly, the coefficients a and b corresponding to each local window can be obtained.
[0108] Based on the average value of coefficient a and the average value of coefficient b corresponding to each local window, the coefficient corresponding to the entire Nth frame image is obtained. and That is to say
[0109] Similarly, based on the pixel value of each pixel in the Nth frame image, the pixel value of the pixel at the corresponding position in the N-1th frame target motion image is obtained.
[0110] It should be understood that the linear coefficients corresponding to different local windows may be different. The local window is an area with a certain pixel point as the center and a radius of a preset value in the Nth frame image. The number of local windows corresponding to the Nth frame image is determined according to the preset value and the size of the Nth frame image.
[0111] In the embodiment of the present application, through guided filtering, the gradient of the target motion image of the N-1 frame is proportional to the gradient of the N-frame image. Therefore, the target motion image of the N-1 frame can effectively restore the edge information of the N-frame image, accurately fit the N-frame image, and have the robustness of large-scale motion images, thereby improving the accuracy of the target motion image of the N-1 frame.
[0112] Figure 7 It is a schematic diagram of another structure of a motion image generation device provided according to an embodiment of the present application.
[0113] Combine the following Figure 7 Another structure of the device for generating a moving image is exemplified.
[0114] In some embodiments, the up-sampling module 203 is further used to obtain a data reading instruction;
[0115] The N-1th frame of down-sampled image is read based on the data reading instruction.
[0116] In one example, a CPU (Central Processing Unit) is provided in the ISP. The CPU sends a data read instruction to the up-sampling module 203, and the up-sampling module 203 reads the N-1th frame of down-sampled image from the memory 205 based on the data read instruction.
[0117] In some embodiments, the device further includes a data reading module 206;
[0118] The data reading module 206 is electrically connected to the up-sampling module 203;
[0119] The data reading module 206 is used to obtain the data reading instruction; read the N-1th frame of down-sampled image based on the data reading instruction; and send the N-1th frame of down-sampled image to the up-sampling module 203 .
[0120] In one example, the CPU sends a data reading instruction to the data reading module 206 , and the data reading module 206 reads the N−1th frame of down-sampled image from the memory 205 based on the data reading instruction and sends it to the up-sampling module 203 .
[0121] In some embodiments, the device further includes a memory 205, wherein the memory 205 is electrically connected to the down-sampling module 202 and the data reading module 206 respectively;
[0122] The down-sampling module 202 is used to store the N-1th frame of down-sampled image in the memory 205;
[0123] The data reading module 206 is used to read the N-1th frame of down-sampled image from the memory 205 based on the data reading instruction.
[0124] In an example, while the upsampling module 203 sends the N-1th upsampled image frame to the generating module 204, the generating module 204 obtains the Nth frame image, so that the generating module 204 performs guided filtering on the N-1th frame image based on the Nth frame image.
[0125] In the related art, the down-sampling module 202 is directly electrically connected to the up-sampling module 203. After the down-sampling module 202 obtains the N-1th frame of down-sampling image, the up-sampling module 203 up-samples the N-1th frame of down-sampling image to obtain the N-1th frame of up-sampling image. That is, the up-sampling module 203 needs to wait for the N-1th frame of down-sampling image before it can run, which wastes hardware resources.
[0126] In the embodiment of the present application, after the memory 205 is set between the down-sampling module 202 and the up-sampling module 203, after obtaining the Nth frame of down-sampled image, the down-sampling module 202 stores the Nth frame of down-sampled image in the memory 205. The up-sampling module 203 reads the N-1th frame of down-sampled image from the memory 205. That is, the up-sampling module 203 does not need to wait for the down-sampling module 202 to output the N-1th frame of down-sampled image, but directly reads the N-1th frame of down-sampled image from the memory 205, thereby reducing the waiting time of the up-sampling module 203, optimizing hardware resources, and saving hardware costs.
[0127] In one example, the CPU controls the motion detection module 201 and the up-sampling module 203 to operate in parallel, so as to further optimize hardware resources.
[0128] For example, the motion detection module 201 obtains the Nth frame image and the N-1th frame image, and obtains the Nth frame motion image based on the Nth frame image and the N-1th frame image; at the same time, the upsampling module 203 obtains the N-1th frame downsampled image, upsamples the N-1th frame downsampled image, and obtains the N-1th frame upsampled image.
[0129] It should be noted that: when the motion image generation device provided in the above embodiment executes the corresponding steps, it only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
[0130] In the embodiment of the present application, a motion image is downsampled by a downsampling module to obtain a downsampled image, the downsampled image is upsampled to obtain an upsampled image, and the upsampled image is filtered based on the Nth frame image to obtain the N-1th frame of target motion image including the edge information of the Nth frame image, thereby improving the robustness and accuracy of the motion image. The downsampled image is more focused on capturing large-scale motion changes, while ignoring small-scale detail changes, thereby avoiding problems such as discontinuity, missed detection, and false detection in motion judgment. In addition, compared to the related art, the accuracy of motion judgment is improved by increasing the size of the preset window, and the embodiment of the present application does not need to increase pixel point calculation, thereby saving hardware power consumption and reducing hardware costs.
[0131] Figure 8 is a flow chart of a method for generating a motion image according to an embodiment of the present application, such as Figure 8 As shown, in the embodiment of the present application, the application is described by taking the application on a terminal with an ISP as an example. The method comprises the following steps:
[0132] In step 801, the terminal obtains the N-1th down-sampled image; up-samples the N-1th down-sampled image to obtain the N-1th up-sampled image; the N-1th down-sampled image is obtained by down-sampling the N-1th motion image by the down-sampling module; the N-1th motion image is obtained by detecting the N-1th image and the N-2th image by the motion detection module; N is an integer greater than or equal to 3;
[0133] In step 802, the terminal filters the N-1th upsampled image based on the Nth frame image to obtain the N-1th target motion image; the N-1th target motion image includes edge information of the Nth frame image; the N-1th target motion image represents the motion state of the object in the Nth frame image.
[0134] In some embodiments, the method further includes: obtaining a data read instruction;
[0135] The N-1th frame of down-sampled image is read based on the data reading instruction.
[0136] In a possible implementation, the method further includes:
[0137] The data reading module is electrically connected to the up-sampling module;
[0138] The data reading module is used to obtain the data reading instruction; read the N-1th frame of down-sampled image based on the data reading instruction; and send the N-1th frame of down-sampled image to the up-sampling module.
[0139] In some embodiments, the method further includes: storing the N-1th frame of down-sampled image in the memory;
[0140] The N-1th frame of down-sampled image is read from the memory based on the data read instruction.
[0141] In some embodiments, the method further includes: acquiring a plurality of first image regions from the N-1 frame image based on a preset window; acquiring a plurality of second image regions from the N-2 frame image based on the preset window; the coordinate range of the first image region is the same as the coordinate range of the corresponding second image region; the coordinate system corresponding to the N-2 frame image and the N-1 frame image is a preset coordinate system;
[0142] Based on each of the first image regions and the corresponding second image region, a sum of absolute differences is calculated; the sum of absolute differences indicates whether there is motion between the corresponding first image region and the corresponding second image region;
[0143] Based on the absolute difference sum corresponding to each of the first image regions, the N-1th frame of motion image is obtained.
[0144] In some embodiments, the method further comprises:
[0145] When the absolute difference sum is greater than a preset threshold, the absolute difference sum indicates that there is motion between the first image area and the corresponding second image area; in the N-1th frame of motion image, the first partial area is marked as a motion area; the coordinate range of the first partial area is the same as the coordinate range of the first image area; or
[0146] When the absolute value sum is less than or equal to the preset threshold, the absolute difference sum indicates that there is no motion between the first image area and the corresponding second image area; in the N-1th frame motion image, the first partial area is marked as a non-motion area.
[0147] In some embodiments, the ratio of the N-1th down-sampled image frame to the Nth frame image is greater than 4 / 1.
[0148] In some embodiments, the size of the N-1th upsampled image is the same as the size of the Nth frame image.
[0149] In some embodiments, the method further includes: performing guided filtering on the N-1th frame up-sampled image based on the Nth frame image to obtain the N-1th frame target motion image.
[0150] In addition, the motion image generation device and the motion image generation method provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the device embodiment, which will not be repeated here.
[0151] In the embodiment of the present application, a motion image is downsampled by a downsampling module to obtain a downsampled image, the downsampled image is upsampled to obtain an upsampled image, and the upsampled image is filtered based on the Nth frame image to obtain the N-1th frame of target motion image including the edge information of the Nth frame image, thereby improving the robustness and accuracy of the motion image. The downsampled image is more focused on capturing large-scale motion changes, while ignoring small-scale detail changes, thereby avoiding problems such as discontinuity, missed detection, and false detection in motion judgment. In addition, compared to the related art, the accuracy of motion judgment is improved by increasing the size of the preset window, and the embodiment of the present application does not need to increase pixel point calculation, thereby saving hardware power consumption and reducing hardware costs.
[0152] An embodiment of the present application further provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the above method when executing the computer program.
[0153] Taking computer equipment as the terminal as an example, Fig. 9 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application, see Fig. 9 The terminal 900 may be a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a notebook computer or a desktop computer. The terminal 900 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.
[0154] Typically, the terminal 900 includes a processor 901 and a memory 902 .
[0155] The processor 901 may include one or more processing cores, such as a 4-core processor, a 5-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0156] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one program code, which is used to be executed by the processor 901 to implement the process for terminal execution in the above method provided by the method embodiment of the present application.
[0157] In some embodiments, the terminal 900 may further optionally include: a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902 and the peripheral device interface 903 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 903 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a display screen 904, a camera assembly 905, an audio circuit 906 and a power supply 907.
[0158] The peripheral device interface 903 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 may be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.
[0159] The display screen 904 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 904 is a touch display screen, the display screen 904 also has the ability to collect touch signals on the surface or above the surface of the display screen 904. The touch signal can be input to the processor 901 as a control signal for processing. At this time, the display screen 904 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 904 can be one, set on the front panel of the terminal 900; in other embodiments, the display screen 904 can be at least two, respectively set on different surfaces of the terminal 900 or in a folding design; in other embodiments, the display screen 904 can be a flexible display screen, set on the curved surface or folding surface of the terminal 900. Even, the display screen 904 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 904 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).
[0160] The camera assembly 905 is used to capture images or videos. In some embodiments, the camera assembly 905 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the blurring function of the non-motion area, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 905 may also include a flash. The flash may be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.
[0161] The audio circuit 906 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 901 for processing. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 900. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 901 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 906 may also include a headphone jack.
[0162] The power supply 907 is used to power various components in the terminal 900. The power supply 907 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 907 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.
[0163] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the terminal 900, and the terminal 900 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.
[0164] Take the computer device as a server as an example. Fig.10 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1001 and one or more memories 1002, wherein the one or more memories 1002 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1001 to implement the above-mentioned motion image generation method. Of course, the server 1000 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1000 may also include other components for implementing device functions, which will not be described in detail here.
[0165] The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for generating an image processing model. Optionally, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.
[0166] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0167] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A motion image generation device, characterized in that: include: A motion detection module, a down-sampling module, an up-sampling module and a generation module electrically connected in sequence; the motion detection module is also electrically connected to the generation module; The up-sampling module is used to obtain the N-1th down-sampled image; up-sample the N-1th down-sampled image to obtain the N-1th up-sampled image; the N-1th down-sampled image is obtained by the down-sampling module down-sampling the N-1th motion image; the N-1th motion image is obtained by the motion detection module detecting the N-1th image and the N-2th image; N is an integer greater than or equal to 3; The generation module is used to filter the N-1th frame upsampled image based on the Nth frame image to obtain the N-1th frame target motion image; the N-1th frame target motion image includes edge information of the Nth frame image; the N-1th frame target motion image represents the motion state of the object in the Nth frame image.
2. The device according to claim 1, characterized in that The up-sampling module is also used to obtain a data reading instruction; The N-1th frame of down-sampled image is read based on the data reading instruction.
3. The device according to claim 1, characterized in that The device further includes a data reading module; The data reading module is electrically connected to the up-sampling module; The data reading module is used to obtain the data reading instruction; read the N-1th frame of down-sampled image based on the data reading instruction; and send the N-1th frame of down-sampled image to the up-sampling module.
4. The device according to claim 3, characterized in that The device further includes a memory, wherein the memory is electrically connected to the down-sampling module and the data reading module respectively; The down-sampling module is used to store the N-1th frame of down-sampled image in the memory; The data reading module is used to read the N-1th frame of down-sampled image from the memory based on the data reading instruction.
5. The device according to claim 1, characterized in that The motion detection module is used to obtain a plurality of first image regions from the N-1th frame image based on a preset window; Acquire multiple second image regions from the N-2 frame image based on the preset window; the coordinate range of the first image region is the same as the coordinate range of the corresponding second image region; the coordinate system corresponding to the N-2 frame image and the N-1 frame image is a preset coordinate system; calculating a sum of absolute differences based on each of the first image regions and the corresponding second image region; the sum of absolute differences indicating whether there is motion between the first image region and the corresponding second image region; The N-1th frame of motion image is obtained based on the sum of absolute differences corresponding to each of the first image regions.
6. The device according to claim 5, characterized in that The motion detection module is further used for: When the sum of absolute differences is greater than a preset threshold, the sum of absolute differences indicates that there is motion between the first image area and the corresponding second image area; in the N-1th frame of motion image, a first partial area is marked as a motion area; and a coordinate range of the first partial area is the same as a coordinate range of the first image area; or When the absolute value sum is less than or equal to the preset threshold, the absolute difference sum indicates that there is no motion between the first image area and the corresponding second image area; in the N-1th frame motion image, the first partial area is marked as a non-motion area.
7. The device according to claim 1, characterized in that The ratio of the N-1th down-sampled image frame to the Nth frame image is greater than 4 / 1.
8. The device according to claim 1, characterized in that The size of the N-1th up-sampled image is the same as the size of the Nth image.
9. The device according to claim 1, characterized in that The generating module is used to perform guided filtering on the N-1th frame up-sampled image based on the Nth frame image to obtain the N-1th frame target motion image.
10. A method for generating a moving image, characterized in that: include: Acquire the N-1th down-sampled image; up-sample the N-1th down-sampled image to obtain the N-1th up-sampled image; the N-1th down-sampled image is obtained by down-sampling the N-1th motion image by the down-sampling module; the N-1th motion image is obtained by detecting the N-1th image and the N-2th image by the motion detection module; N is an integer greater than or equal to 3; The N-1th frame upsampled image is filtered based on the Nth frame image to obtain the N-1th frame target motion image; the N-1th frame target motion image includes edge information of the Nth frame image; the N-1th frame target motion image represents the motion state of the object in the Nth frame image.
11. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory is used to store at least one program, and the at least one program is loaded by the processor and executes the method for generating a moving image as claimed in claim 10.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the method for generating a moving image as claimed in claim 10.