Image processing methods, apparatuses, electronic devices, and computer storage media

Abstract

This embodiment discloses an image processing method, apparatus, electronic device, and computer storage medium. The method includes: acquiring a video stream with particle effects acquired by an image acquisition device; extracting the background of a first target frame image and the background of a second target frame image; extracting feature points in the background of the first target frame image and feature points in the background of the second target frame image; determining motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image based on the positional relationship between the matching feature points in the background of the first target frame image and the background of the second target frame image; and determining the particle trajectory between the first target frame image and the second target frame image based on the motion information.

Description

Technical Field

[0001] This disclosure relates to computer vision processing technology, and more particularly to an image processing method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] In related technologies, particle systems represent techniques for simulating specific blurred phenomena in 3D computer graphics, phenomena whose realistic physical motion is difficult to achieve using other traditional rendering techniques. Phenomena frequently simulated using particle systems include fire, explosions, smoke, water flow, sparks, falling leaves, clouds, fog, snow, dust, meteor trails, or abstract visual effects such as luminous trajectories.

[0003] Particle systems have been widely used in the special effects industry. In traditional methods, designers and animators need to spend a lot of time designing and adjusting parameters for the required particle effects. In related technologies, optical flow can be used to track particles in video. When the image acquisition equipment is stable, the tracked particle trajectories will be relatively fixed and follow multiple smooth curves. However, when the image acquisition equipment is displaced, the accuracy and reliability of particle tracking will be reduced. Summary of the Invention

[0004] This disclosure provides a technical solution for image processing.

[0005] This disclosure provides an image processing method, the method comprising:

[0006] Acquire a video stream with particle effects captured by an image acquisition device;

[0007] Extract the background of the first target frame image and the background of the second target frame image respectively;

[0008] Feature points in the background of the first target frame image and feature points in the background of the second target frame image are extracted; based on the positional relationship between the matched feature points in the background of the first target frame image and the background of the second target frame image, the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image is determined.

[0009] Based on the motion information, the particle trajectory between the first target frame image and the second target frame image is determined.

[0010] In some embodiments, determining the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image based on the positional relationship between the matched feature points in the background of the first target frame image and the background of the second target frame image includes:

[0011] Based on the positional relationship, the image transformation amplitude of the second target frame image relative to the first target frame image is determined, and the image transformation amplitude is used as the motion information; the image transformation amplitude includes rotation and / or translation.

[0012] Understandably, the backgrounds of the first and second target frame images are usually static backgrounds. Therefore, the positional relationship between the matching feature points in the backgrounds of the first and second target frame images can accurately reflect the motion information of the image acquisition device between the acquisition time of the first and second target frame images. That is, based on the positional relationship between the matching feature points in the backgrounds of the first and second target frame images, the motion information of the image acquisition device between the acquisition time of the first and second target frame images can be accurately obtained.

[0013] In some embodiments, determining the image transformation magnitude of the second target frame image relative to the first target frame image based on the positional relationship includes:

[0014] Obtain at least one variable to represent the image transformation magnitude of the first target frame image;

[0015] Determine the transformed image of the first target frame image after image transformation according to the at least one variable; determine the value of the at least one variable with the goal of reducing the positional difference of matching feature points in the background of the transformed image and the background of the second target frame image;

[0016] The image transformation magnitude of the second target frame image relative to the first target frame image is determined based on the value of the at least one variable.

[0017] Understandably, in this embodiment of the present disclosure, when the value of the above-mentioned at least one variable is determined with the goal of reducing the positional difference of matching feature points in the background of the transformed image and the background of the second target frame image, the image transformation magnitude of the second target frame image relative to the first target frame image can be accurately determined based on the value of the above-mentioned at least one variable.

[0018] In some embodiments, the method further includes:

[0019] After determining the matching feature points in the backgrounds of the first target frame image and the second target frame image, remove the incorrectly matched feature points from the matching feature points in the backgrounds of the first target frame image and the second target frame image.

[0020] Understandably, by removing mismatched feature points from the backgrounds of the first and second target frame images, it is beneficial to accurately determine the motion information of the image acquisition device between the acquisition times of the two frames, thereby facilitating accurate particle tracking of particles in the video stream.

[0021] In some embodiments, determining the particle trajectory between the first target frame image and the second target frame image based on the motion information includes:

[0022] Based on the motion information, the foreground in the second target frame image is corrected to obtain the corrected second target frame image;

[0023] Particle tracking is performed on the foreground of the first target frame image and the foreground of the corrected second target frame image to obtain the particle trajectory between the first target frame image and the second target frame image.

[0024] Because of the movement of the image acquisition device, the preliminary results of particle tracking corresponding to the first target frame image and the second target frame image may be incorrect. To address this problem, in this embodiment, the foreground in the second target frame image can be corrected based on the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image. This can decouple the particle motion information between the first target frame image and the corrected second target frame image from the motion information of the image acquisition device, thereby enabling a more accurate determination of the particle trajectory between the two frames.

[0025] In some embodiments, the first target frame image and the second target frame image are two adjacent images in the video stream, and the first target frame image and the second target frame image form a pair of adjacent images; after determining the particle trajectory between the first target frame image and the second target frame image, the method further includes: obtaining the particle trajectory in the video stream based on the particle trajectories of the multiple pairs of adjacent images contained in the video stream.

[0026] As can be seen, the embodiments of this disclosure can perform particle tracking on particles in the video stream based on the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image, thereby obtaining accurate particle trajectories.

[0027] In some embodiments, the method further includes:

[0028] The particle effects of the video stream are reconstructed based on the particle trajectories in the video stream.

[0029] As can be seen, the embodiments of this disclosure can accurately reconstruct the particle effect of the above video stream based on the accurately obtained particle trajectory, so that the reconstructed particle effect is close to the particle effect of the original video stream, and the reconstructed particle effect can accurately reflect the basic information of the particle system.

[0030] This disclosure also provides an image processing apparatus, which includes: an acquisition module, a first processing module, a second processing module, and a third processing module; wherein,

[0031] The acquisition module is used to acquire video streams with particle effects captured by the image acquisition device;

[0032] The first processing module is used to extract the background of the first target frame image and the background of the second target frame image, respectively.

[0033] The second processing module is used to extract feature points in the background of the first target frame image and feature points in the background of the second target frame image; and to determine the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image based on the positional relationship between the matched feature points in the background of the first target frame image and the background of the second target frame image.

[0034] The third processing module is used to determine the particle trajectory between the first target frame image and the second target frame image based on the motion information.

[0035] This disclosure also provides an electronic device, including a processor and a memory for storing a computer program capable of running on the processor; wherein,

[0036] The processor is used to run the computer program to perform any of the above-described image processing methods.

[0037] This disclosure also provides a computer storage medium storing a computer program that, when executed by a processor, implements any of the above-described image processing methods.

[0038] The image processing method, apparatus, electronic device, and computer storage medium proposed in this disclosure include: acquiring a video stream with particle effects acquired by an image acquisition device; determining each group of adjacent images in the video stream, each group of adjacent images including an adjacent first target frame image and a second target frame image; extracting the background of the first target frame image and the background of the second target frame image respectively; extracting feature points in the background of the first target frame image and feature points in the background of the second target frame image; determining motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image based on the positional relationship between the matching feature points in the background of the first target frame image and the background of the second target frame image; determining the particle trajectory between two frames in each group of adjacent images based on the motion information; and obtaining the particle trajectory in the video stream based on the particle trajectory between two frames in each group of adjacent images.

[0039] As can be seen, the backgrounds of the first and second target frame images are usually static. Therefore, the positional change information between the matched feature points in the backgrounds of the first and second target frame images can accurately reflect the motion information of the image acquisition device between the acquisition time of the first and second target frame images. Thus, based on the motion information of the image acquisition device between the acquisition time of the first and second target frame images, the particle trajectory between the two frames can be determined more accurately.

[0040] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0042] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0043] Figure 2a This is a schematic diagram illustrating the imaging principle of the perspective projection method in an embodiment of this disclosure;

[0044] Figure 2b This is a schematic diagram illustrating the imaging principle of orthogonal projection in an embodiment of this disclosure;

[0045] Figure 3a This is a schematic diagram illustrating the change in the translation amount of the image acquisition device in the x-direction in an embodiment of this disclosure;

[0046] Figure 3bThis is a schematic diagram illustrating the change in the translation amount of the image acquisition device in the y-direction in an embodiment of this disclosure;

[0047] Figure 3c This is a schematic diagram illustrating the change in the rotation angle of the image acquisition device in an embodiment of this disclosure;

[0048] Figure 3d This is a schematic diagram showing the relationship between the translation amount of the image acquisition device in the x-direction and the translation amount of the image acquisition device in the y-direction in an embodiment of this disclosure.

[0049] Figure 4a This refers to any frame image in the video stream of this embodiment of the disclosure;

[0050] Figure 4b This is a schematic diagram illustrating the preliminary results of particle tracking according to an embodiment of this disclosure;

[0051] Figure 4c This is a schematic diagram of a particle trajectory according to an embodiment of the present disclosure;

[0052] Figure 4d This is a schematic diagram of another particle trajectory according to an embodiment of the present disclosure;

[0053] Figure 5 This is a flowchart illustrating the process of determining motion information of an image acquisition device between the acquisition times of two image frames in an embodiment of this disclosure.

[0054] Figure 6 This is a schematic diagram of the process for determining the image transformation amplitude in an embodiment of this disclosure;

[0055] Figure 7 This is a schematic diagram illustrating the process of deriving particle trajectories in an embodiment of this disclosure;

[0056] Figure 8 This is a schematic diagram of the composition structure of the image processing apparatus according to an embodiment of the present disclosure;

[0057] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0058] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present disclosure and are not intended to limit the present disclosure. Furthermore, the embodiments provided below are some embodiments for implementing the present disclosure, and not all embodiments for implementing the present disclosure. Unless otherwise specified, the technical solutions described in the embodiments of the present disclosure can be implemented in any combination.

[0059] It should be noted that, in the embodiments of this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of circuitry, processors, programs, or software, etc.) in the method or apparatus that includes that element.

[0060] For example, the image processing method provided in this disclosure includes a series of steps, but the image processing method provided in this disclosure is not limited to the steps described. Similarly, the image processing apparatus provided in this disclosure includes a series of modules, but the apparatus provided in this disclosure is not limited to the modules explicitly described, but may also include modules that need to be set up for obtaining relevant information or processing based on information.

[0061] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0062] The embodiments disclosed herein can be applied to computer systems consisting of terminals and / or servers, and can operate in conjunction with a wide range of other general-purpose or special-purpose computing system environments or configurations. Here, the terminal can be a thin client, a thick client, a handheld or laptop device, a microprocessor-based system, a set-top box, a programmable consumer electronics product, a network personal computer, a minicomputer system, etc., and the server can be a server computer system, a minicomputer system, a mainframe computer system, and a distributed cloud computing environment that includes any of the above systems, etc.

[0063] Electronic devices such as terminals and servers may include program modules for executing instructions. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., and can perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0064] In related technologies, an image acquisition device can be used to capture a video stream with particle effects. Then, optical flow can be used to track the particles in the video to reconstruct the particle system. However, since the video stream is captured while the image acquisition device is moving (e.g., experiencing jitter or significant displacement), the positional changes of the particles in the image are related not only to the particle's displacement in the real scene but also to the displacement of the image acquisition device. Therefore, when using optical flow for particle tracking, particle tracking errors may occur, resulting in lower reliability of the obtained particle trajectories and affecting the reconstruction of the particle system. For example, when the image acquisition device is in motion, the trajectories of particles moving in the same direction as the image acquisition device will become longer, while the trajectories of particles moving in the same direction as the image acquisition device will become shorter.

[0065] In related technologies, video stabilization methods can be used to estimate the motion information of image acquisition devices. This involves finding stable feature points on two image frames and performing corresponding feature point matching to calculate the rigid body transformation between the two frames. This rigid body transformation can then be used to represent the motion information of the image acquisition device, allowing for the correction of particle tracking results based on this motion information. However, in video streams with particle effects, particles such as fireworks are typically in complex motion states. Therefore, it is difficult to decouple the motion of the image acquisition device from the motion of the particles, making it difficult to accurately determine the feature point matching relationship of the image. This hinders the accurate estimation of the motion information of the image acquisition device and consequently, the generation of accurate particle tracking results.

[0066] To address the aforementioned technical problems, an image processing method is proposed in some embodiments of this disclosure.

[0067] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present disclosure, such as... Figure 1 As shown, the process may include:

[0068] Step 101: Acquire a video stream with particle effects captured by an image acquisition device.

[0069] In this embodiment of the disclosure, the image acquisition device can be a device for continuously acquiring images with particle effects. When the image acquisition device continuously acquires images with particle effects, a video stream with particle effects can be obtained. Obviously, the video stream with particle effects includes at least two frames of images with particle effects. The particle effects here can represent effects such as fire, explosion, smoke, water flow, sparks, falling leaves, clouds, fog, snow, dust, meteor trails, etc., or they can be abstract visual effects similar to light trails.

[0070] In some embodiments, the imaging method of the image acquisition device is as follows: Figure 2a As shown in the perspective projection method, objects appear larger when closer and smaller when farther away. When using an image acquisition device to photograph distant subjects such as fireworks, the difference in image quality caused by perspective projection can be considered negligible due to the considerable distance between the device and the subject; that is, the effect of objects appearing larger when closer and smaller when farther away can be ignored. In this case, the imaging method of the image acquisition device can be considered as... Figure 2b As shown, the orthogonal projection method produces a consistent image size even when the distance between objects of the same size and the image acquisition device varies.

[0071] Step 102: Extract the background of the first target frame image and the background of the second target frame image respectively.

[0072] After obtaining the first target frame image and the second target frame image, the background of each image can be separated.

[0073] In this embodiment of the disclosure, for both the first target frame image and the second target frame image, an image segmentation method can be used to segment the foreground and background in the image; the image segmentation method can be a threshold-based segmentation method, a region-based segmentation method, an edge-based segmentation method, or a segmentation method based on a specific theory, etc.; the following uses the first target frame image as an example to illustrate the implementation method of the background of the image.

[0074] In some embodiments, the implementation of separating the background of the first target frame image using a threshold-based segmentation method may include: firstly, using an image segmentation method based on an adaptive threshold to determine the pixels with higher gray values ​​in the first target frame image, and referring to the determined pixels with higher gray values ​​as significant pixels; then, combining the gray value and position change information of significant pixels in at least two frames including the first target frame image, and using a Gaussian mixture model to determine the background in the first target frame image.

[0075] Understandably, image segmentation methods can not only segment the background of an image, but also the foreground; the foreground can include particle images; for example, in the case of an image of fireworks at night, the background can be a background image other than the fireworks, and the foreground can be a fireworks image separated from the background.

[0076] Step 103: Extract feature points from the background of the first target frame image and the background of the second target frame image; determine the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image based on the positional relationship between the matched feature points in the background of the first target frame image and the background of the second target frame image.

[0077] In some embodiments, after obtaining the background of the first target frame image and the background of the second target frame image, feature points in the background of the first target frame image and feature points in the background of the second target frame image can be extracted using the convolutional layer of a neural network.

[0078] After extracting feature points from the background of the first target frame image and the background of the second target frame image, feature point matching processing can be performed on the feature points in the backgrounds of the first and second target frame images to obtain matched feature points in the backgrounds of the first and second target frame images. For example, the feature point matching steps may include: detecting feature points in the backgrounds of the first and second target frame images; calculating descriptors for the feature points in the backgrounds of the first and second target frame images; calculating the distance between the descriptors of the feature points in the backgrounds of the first and second target frame images; and determining matching pairs of feature points in the backgrounds of the first and second target frame images using a nearest neighbor search method. It should be noted that the above description is merely an exemplary illustration of the implementation of feature point matching, and the embodiments disclosed herein are not limited thereto.

[0079] Understandably, the backgrounds of the first target frame image and the second target frame image are usually static backgrounds. Therefore, the positional change information between the matched feature points in the backgrounds of the first and second target frame images can accurately reflect the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image.

[0080] Step 104: Determine the particle trajectory between the first target frame image and the second target frame image based on the motion information.

[0081] In practical applications, steps 101 to 104 can be implemented using a processor in an electronic device. The processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor.

[0082] Understandably, the backgrounds of the first and second target frame images are usually static backgrounds. Therefore, the positional change information between the matched feature points in the backgrounds of the first and second target frame images can accurately reflect the motion information of the image acquisition device between the acquisition time of the first and second target frame images. Thus, based on the motion information of the image acquisition device between the acquisition time of the first and second target frame images, the particle trajectory between the two frames can be determined more accurately.

[0083] In some embodiments of this disclosure, each group of adjacent images can be determined in the video stream. Each group of adjacent images includes an adjacent first target frame image and a second target frame image. Obviously, the first target frame image and the second target frame image are two consecutive frames in the video stream.

[0084] For any two frames in the video stream, the motion information of the image acquisition device between the acquisition times of any two frames can be determined according to the implementation of steps 101 to 103. Thus, the motion information of the image acquisition device during the acquisition period of the video stream can be obtained. The following will demonstrate... Figures 3a to 3d The motion information of the image acquisition device is illustrated by example.

[0085] Figure 3a In the image, the horizontal axis represents the frame number, and the vertical axis represents the translation dx of the image acquisition device in the x-direction. Figure 3b In the image, the horizontal axis represents the frame number, and the vertical axis represents the translation amount dy of the image acquisition device in the y direction; Figure 3c In the image acquisition device, the horizontal axis represents the frame number, and the vertical axis represents the rotation angle da of the image acquisition device, with the unit of rotation angle da being rad. Figure 3dIn the diagram, the horizontal axis represents the translation dx of the image acquisition device in the x-direction, and the vertical axis represents the translation dy of the image acquisition device in the y-direction.

[0086] Figures 3a to 3d In the diagram, dashed lines represent the motion information of the image acquisition device during the video stream acquisition period, while solid lines represent the motion information of the image acquisition device during other periods outside the video stream acquisition period. Through... Figures 3a to 3d This allows for a direct understanding of the translation and rotation amounts of the image acquisition device during the video stream acquisition period.

[0087] In some embodiments, the first target frame image and the second target frame image are two consecutive frames. Understandably, when the first target frame image and the second target frame image are two consecutive frames, the embodiments of this disclosure can accurately obtain the motion information of the image acquisition device between the acquisition times of the two consecutive frames. Furthermore, based on the motion information of the image acquisition device between the acquisition times of any two consecutive frames, it is beneficial to accurately track particles in the video stream subsequently.

[0088] In some embodiments, the first target frame image and the second target frame image are two adjacent images in the video stream, forming a pair of adjacent images. The particle trajectory in the video stream can be obtained based on the particle trajectories of multiple pairs of adjacent images contained within the video stream. In one embodiment, the particle trajectory in the video stream can be obtained by calculating the particle trajectories between pairs of adjacent images in the video stream, thereby obtaining the particle trajectories between multiple adjacent images in the video stream.

[0089] For example, Figure 4a For any frame of the video stream; the motion information of the image acquisition device is Figures 3a to 3d Given the motion information shown, particle tracking can first be performed using optical flow on each frame of the video stream to obtain... Figure 4b The preliminary results of particle tracking are shown, which include multiple particle trajectories.

[0090] In obtaining Figure 4b After showing the preliminary results of particle tracking, the motion information of the image acquisition device during the video stream acquisition period can be used to further refine the tracking. Figure 4b The preliminary results of particle tracking shown are subjected to motion compensation to obtain... Figure 4c The particle trajectory is shown.

[0091] In related technologies, the movement of the image acquisition device can cause errors in the preliminary particle tracking results corresponding to the first and second target frame images, thus affecting the accuracy and reliability of the particle tracking results. However, in this embodiment, particle tracking can be performed on the particles in the video stream based on the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image, thereby obtaining accurate particle trajectories.

[0092] In some embodiments of this disclosure, reference is made to Figure 5 The process of determining the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image may include:

[0093] Step 1031: Based on the above positional relationship, determine the image transformation amplitude of the second target frame image relative to the first target frame image. The image transformation amplitude includes rotation and / or translation.

[0094] Here, the rotation of the second target frame image relative to the first target frame image can be either a counterclockwise rotation angle or a clockwise rotation angle. The translation of the second target frame image relative to the first target frame image can be represented by the number of pixels. In this embodiment, the size of the pixels in the image is fixed when the image resolution remains unchanged.

[0095] Step 1032: Use the image transformation amplitude as motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image.

[0096] Understandably, the backgrounds of the first and second target frame images are usually static backgrounds. Therefore, the positional relationship between the matching feature points in the backgrounds of the first and second target frame images can accurately reflect the motion information of the image acquisition device between the acquisition time of the first and second target frame images. That is, based on the positional relationship between the matching feature points in the backgrounds of the first and second target frame images, the motion information of the image acquisition device between the acquisition time of the first and second target frame images can be accurately obtained.

[0097] In some embodiments of this disclosure, reference is made to Figure 6 The process of determining the image transformation magnitude of the second target frame image relative to the first target frame image may include:

[0098] Step 10311: Obtain at least one variable to represent the image transformation magnitude of the first target frame image.

[0099] Step 10312: Determine the transformed image of the first target frame image after image transformation according to at least one of the above variables; determine the value of at least one of the above variables with the goal of reducing the positional difference of matching feature points in the background of the transformed image and the background of the second target frame image.

[0100] Step 10313: Determine the image transformation magnitude of the second target frame image relative to the first target frame image based on the value of at least one of the above variables.

[0101] Here, at least one of the above variables can be at least one of the rotation and translation amounts mentioned above. For example, the value of the rotation amount R of the second target frame image relative to the first target frame image and the value of the translation amount T of the second target frame image relative to the first target frame image can be determined according to equation (1):

[0102]

[0103] In equation (1), N represents the number of feature points matching the background of the first target frame image and the background of the second target frame image, and p i ' represents the position of the i-th matched feature point in the background of the second target frame image, p i 0 This represents the position of the i-th matched feature point in the background of the first target frame image. ||·||2 represents the 2-norm.

[0104] Understandably, in this embodiment of the present disclosure, when the value of the above-mentioned at least one variable is determined with the goal of reducing the positional difference of matching feature points in the background of the transformed image and the background of the second target frame image, the image transformation magnitude of the second target frame image relative to the first target frame image can be accurately determined based on the value of the above-mentioned at least one variable.

[0105] In some embodiments of this disclosure, after determining the matching feature points in the background of the first target frame image and the background of the second target frame image, the incorrectly matched feature points in the matching feature points in the background of the first target frame image and the background of the second target frame image can be removed to obtain a corrected feature point matching pair; then, based on the corrected feature point matching pair, the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image is determined.

[0106] For example, the Random Sample Consensus (RANSAC) method can be used to remove mismatched feature points from the backgrounds of the first target frame image and the second target frame image.

[0107] Understandably, by removing mismatched feature points from the backgrounds of the first and second target frame images, it is beneficial to accurately determine the motion information of the image acquisition device between the acquisition times of the two frames, thereby facilitating accurate particle tracking of particles in the video stream.

[0108] In some embodiments, when the second target frame image is the image acquired later between the first target frame image and the second target frame image, referencing Figure 7 The process of determining the particle trajectory between the first target frame image and the second target frame image may include:

[0109] Step 1041: Based on the above motion information, the foreground in the second target frame image is corrected to obtain the corrected second target frame image.

[0110] Step 1042: Perform particle tracking on the foreground of the first target frame image and the foreground of the corrected second target frame image to obtain the particle trajectory between the first target frame image and the second target frame image.

[0111] Understandably, the movement of the image acquisition device can cause errors in the preliminary particle tracking results corresponding to the first target frame image and the second target frame image. To address this issue, in this embodiment, the foreground in the second target frame image can be corrected based on the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image. This can decouple the particle motion information between the first target frame image and the corrected second target frame image from the motion information of the image acquisition device, thereby enabling a more accurate determination of the particle trajectory between the two frames.

[0112] For example, the motion information of the image acquisition device is Figures 3a to 3d Given the motion information shown, firstly, the foreground in the image acquired by the image acquisition device is corrected based on the motion information of the image acquisition device during the video stream acquisition period. Then, particle tracking is performed on the corrected image to obtain... Figure 4d The particle trajectory shown; Figure 4c Compared to the particle trajectories shown, Figure 4d The particle trajectories shown are more in line with expectations.

[0113] In some embodiments of this disclosure, the particle effects of the video stream can be reconstructed based on the particle trajectories in the video stream.

[0114] For example, after obtaining the particle trajectories in the video stream, the particle trajectories can be fitted using a quadratic function or a straight line to obtain parameterized particle trajectories; the parameterized particle trajectories can be clustered to obtain trajectory clustering results; each cluster in the trajectory clustering results represents a type of particle, for example, in the case of fireworks particles; each cluster in the trajectory clustering results represents the trajectory of the same type of fireworks.

[0115] After obtaining the trajectory clustering results, motion modeling can be performed on the particles in each cluster of the trajectory clustering results, thereby reconstructing the particle motion information for each cluster of the trajectory clustering results, that is, realizing the reconstruction of the particle effect of the above video stream.

[0116] As can be seen, the embodiments of this disclosure can accurately reconstruct the particle effect of the above video stream based on the accurately obtained particle trajectory, so that the reconstructed particle effect is close to the particle effect of the original video stream, and the reconstructed particle effect can accurately reflect the basic information of the particle system.

[0117] In this embodiment of the disclosure, for video streams acquired when the image acquisition device is displaced, particle tracking can be performed using matching feature points in the background of two frames in the video stream, thereby reconstructing the particle effect of the video stream. In the case of a fireworks video, even if the image acquisition device moves during the filming of the fireworks, the solution of this embodiment can accurately determine the motion information of the image acquisition device, and thus accurately reconstruct the actual pose information of the fireworks in the video stream.

[0118] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0119] Based on the image processing method proposed in the foregoing embodiments, this disclosure proposes an image processing apparatus.

[0120] Figure 8 This is a schematic diagram of the composition structure of the image processing apparatus according to an embodiment of the present disclosure, such as... Figure 8 As shown, the device may include: an acquisition module 800, a first processing module 801, a second processing module 802, and a third processing module 803; wherein,

[0121] The acquisition module 800 is used to acquire a video stream with particle effects captured by an image acquisition device;

[0122] The first processing module 801 is used to extract the background of the first target frame image and the background of the second target frame image, respectively.

[0123] The second processing module 802 is used to extract feature points in the background of the first target frame image and feature points in the background of the second target frame image; and to determine the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image based on the positional relationship between the matched feature points in the background of the first target frame image and the background of the second target frame image.

[0124] The third processing module 803 is used to determine the particle trajectory between the first target frame image and the second target frame image based on the motion information.

[0125] In some embodiments, the second processing module 802 is configured to determine the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image based on the positional relationship between the matched feature points in the background of the first target frame image and the background of the second target frame image, including:

[0126] Based on the positional relationship, the image transformation amplitude of the second target frame image relative to the first target frame image is determined, and the image transformation amplitude is used as the motion information; the image transformation amplitude includes rotation and / or translation.

[0127] In some embodiments, the second processing module 802 is configured to determine the image transformation magnitude of the second target frame image relative to the first target frame image based on the positional relationship, including:

[0128] Obtain at least one variable to represent the image transformation magnitude of the first target frame image;

[0129] Determine the transformed image of the first target frame image after image transformation according to the at least one variable; determine the value of the at least one variable with the goal of reducing the positional difference of matching feature points in the background of the transformed image and the background of the second target frame image;

[0130] The image transformation magnitude of the second target frame image relative to the first target frame image is determined based on the value of the at least one variable.

[0131] In some embodiments, the second processing module 802 is further configured to remove erroneously matched feature points from the backgrounds of the first target frame image and the second target frame image after determining the matching feature points in the backgrounds of the first target frame image and the second target frame image.

[0132] In some embodiments, the third processing module 803 is configured to determine the particle trajectory between the first target frame image and the second target frame image based on the motion information, including:

[0133] Based on the motion information, the foreground in the second target frame image is corrected to obtain the corrected second target frame image;

[0134] Particle tracking is performed on the foreground of the first target frame image and the foreground of the corrected second target frame image to obtain the particle trajectory between the first target frame image and the second target frame image.

[0135] In some embodiments, the first target frame image and the second target frame image are each group of adjacent images in the video stream;

[0136] Accordingly, the third processing module 803 is further configured to, after determining the particle trajectory between the first target frame image and the second target frame image, obtain the particle trajectory in the video stream based on the particle trajectory between two frames in each group of adjacent images.

[0137] In some embodiments, the third processing module 803 is further configured to reconstruct the particle effects of the video stream based on the particle trajectories in the video stream.

[0138] In practical applications, the acquisition module 800, the first processing module 801, the second processing module 802 and the third processing module 803 can all be implemented using a processor in an electronic device. The processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller and microprocessor.

[0139] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0140] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] Specifically, the computer program instructions corresponding to an image processing method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the computer program instructions corresponding to an image processing method in the storage media are read or executed by an electronic device, any one of the image processing methods in the foregoing embodiments is implemented.

[0142] Based on the same technical concept as the foregoing embodiments, see Figure 9 This illustrates an electronic device 90 provided in an embodiment of the present disclosure, which may include: a memory 901 and a processor 902; wherein,

[0143] The memory 901 is used to store computer programs and data;

[0144] The processor 902 is used to execute the computer program stored in the memory to implement any of the image processing methods in the foregoing embodiments.

[0145] In practical applications, the aforementioned memory 901 can be volatile memory, such as RAM; or non-volatile memory, such as ROM, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor 902.

[0146] The processor 902 described above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic device used to implement the above processor function can also be other types, and this disclosure does not specifically limit the embodiments.

[0147] This disclosure also provides a computer program including computer-readable code, which, when run in an electronic device, enables a processor in the electronic device to execute any of the above-described image processing methods.

[0148] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0149] The description of the various embodiments above tends to emphasize the differences between them. Similarities or commonalities can be referenced interchangeably, and for the sake of brevity, will not be repeated here.

[0150] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0151] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0152] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0154] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a video stream with particle effects captured by an image acquisition device; The backgrounds of the first target frame image and the second target frame image are extracted respectively; the first target frame image and the second target frame image are two adjacent frames in the video stream; Feature points in the background of the first target frame image and feature points in the background of the second target frame image are extracted; based on the positional relationship between the matched feature points in the background of the first target frame image and the background of the second target frame image, the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image is determined. Based on the motion information, the particle trajectory between the first target frame image and the second target frame image is determined; the particle trajectory is used to reconstruct the particle effect in the video stream.

2. The method according to claim 1, characterized in that, Determining the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image based on the positional relationship between the matched feature points in the background of the first target frame image and the background of the second target frame image includes: Based on the positional relationship, the image transformation amplitude of the second target frame image relative to the first target frame image is determined, and the image transformation amplitude is used as the motion information; the image transformation amplitude includes rotation and / or translation.

3. The method according to claim 2, characterized in that, Determining the image transformation magnitude of the second target frame image relative to the first target frame image based on the positional relationship includes: Obtain at least one variable to represent the image transformation magnitude of the first target frame image; Determine the transformed image of the first target frame image after image transformation according to the at least one variable; determine the value of the at least one variable with the goal of reducing the positional difference of matching feature points in the background of the transformed image and the background of the second target frame image; The image transformation magnitude of the second target frame image relative to the first target frame image is determined based on the value of the at least one variable.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: After determining the matching feature points in the backgrounds of the first target frame image and the second target frame image, remove the incorrectly matched feature points from the matching feature points in the backgrounds of the first target frame image and the second target frame image.

5. The method according to any one of claims 1 to 4, characterized in that, Determining the particle trajectory between the first target frame image and the second target frame image based on the motion information includes: Based on the motion information, the foreground in the second target frame image is corrected to obtain the corrected second target frame image; Particle tracking is performed on the foreground of the first target frame image and the foreground of the corrected second target frame image to obtain the particle trajectory between the first target frame image and the second target frame image.

6. The method according to any one of claims 1 to 5, characterized in that, The first target frame image and the second target frame image form a pair of adjacent images; After determining the particle trajectory between the first target frame image and the second target frame image, the method further includes: The particle trajectory in the video stream is obtained based on the particle trajectories of multiple sets of adjacent image pairs contained in the video stream.

7. The method according to claim 6, characterized in that, The method further includes: The particle effects of the video stream are reconstructed based on the particle trajectories in the video stream.

8. An image processing apparatus, characterized in that, The device includes: an acquisition module, a first processing module, a second processing module, and a third processing module; wherein, The acquisition module is used to acquire video streams with particle effects captured by the image acquisition device; The first processing module is used to extract the background of the first target frame image and the background of the second target frame image respectively; the first target frame image and the second target frame image are two adjacent frames in the video stream; The second processing module is used to extract feature points in the background of the first target frame image and feature points in the background of the second target frame image; and to determine the motion information of the image acquisition device between the acquisition time of the first target frame image and the acquisition time of the second target frame image based on the positional relationship between the matched feature points in the background of the first target frame image and the background of the second target frame image. The third processing module is used to determine the particle trajectory between the first target frame image and the second target frame image based on the motion information; the particle trajectory is used to reconstruct the particle effect in the video stream.

9. An electronic device, characterized in that, Includes a processor and memory for storing computer programs that can run on the processor; wherein, The processor is used to run the computer program to perform the method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 7.

Images