Image generation method, system, electronic device, storage medium and product

Abstract

The application relates to the field of image processing, and provides a software image generation method, a system, an electronic device, a storage medium and a product.The method comprises the following steps: obtaining a first image frame and a second image frame, and performing image segmentation on the first image frame and the second image frame respectively to obtain foreground regions of the first image frame and the second image frame; judging whether there are matched feature points in the foreground regions of the first image frame and the second image frame; if there are matched feature points, solving a rotation translation matrix based on the matched feature points, otherwise solving the rotation translation matrix based on a default method; correcting the foreground region of the second image frame based on the rotation translation matrix, and generating a composite image according to the first image frame and the corrected second image frame. According to the application, the foreground and the background are separated, the image is synthesized in a targeted manner, a high-definition image can be synthesized, and the image synthesis quality is improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to an image generation method, system, electronic device, storage medium, and product. Background Technology

[0002] With the development of imaging technology for electronic devices, users have higher requirements for image quality, and manufacturers have faced new challenges in image processing for electronic devices.

[0003] Current imaging technologies for electronic devices synthesize images by combining features from the entire screen. This method often results in unnatural composite images, blurred edges, artifacts, and overlaps. Clearly, current imaging methods severely impact the user's viewing experience, causing inconvenience and affecting the user experience. Summary of the Invention

[0004] Therefore, it is necessary to provide an image generation method, system, electronic device, storage medium, and product that can improve image quality in response to the above-mentioned technical problems.

[0005] This application provides an image generation method, including:

[0006] A first image frame and a second image frame are obtained, and image segmentation is performed on the first image frame and the second image frame respectively to obtain the foreground regions of the first image frame and the second image frame.

[0007] Determine whether there are matching feature points in the foreground region of the first image frame and the foreground region of the second image frame.

[0008] If matching feature points exist, the rotation and translation matrix is ​​solved based on the matching feature points; otherwise, the rotation and translation matrix is ​​solved based on the default method.

[0009] The foreground region of the second image frame is corrected based on the rotation and translation matrix, and a composite image is generated based on the first image frame and the corrected second image frame.

[0010] In one embodiment, if matching feature points exist, the rotation and translation matrix is ​​solved based on the matching feature points; otherwise, the rotation and translation matrix is ​​solved based on the default method, including:

[0011] If matching feature points exist, the RANSAC method is used to solve for the matching feature points to obtain the rotation and translation matrix; otherwise, the ECC method is used to iteratively calculate the minimum error of the foreground region of the first and second image frames to obtain the rotation and translation matrix.

[0012] In one embodiment, the foreground regions of both the first image frame and the second image frame include at least one block, and the rotation and translation matrix corresponding to each block is solved based on the matched feature points within each block in the foreground regions of the first image frame and the second image frame.

[0013] In one embodiment, correcting the foreground region of the second image frame based on the rotation and translation matrix includes:

[0014] Each block of the second image frame is corrected based on the rotation and translation matrix corresponding to each block.

[0015] In one embodiment, before determining whether there are matching feature points in the foreground region of the first image frame and the foreground region of the second image frame, the method further includes:

[0016] Obtain feature points from the first image frame and feature points from the second image frame;

[0017] Feature matching is performed on the feature points of the first image frame and the feature points of the second image frame.

[0018] In one embodiment, feature matching of feature points of the first image frame and feature points of the second image frame includes:

[0019] The first method is applied to perform a matching of feature points of the first image frame and feature points of the second image frame.

[0020] Based on the first matching, a second method is applied to perform a second matching on the feature points of the first image frame and the feature points of the second image frame.

[0021] This application provides an image generation system, including:

[0022] The segmentation module is used to obtain a first image frame and a second image frame, and to perform image segmentation on the first image frame and the second image frame respectively to obtain the foreground regions of the first image frame and the second image frame.

[0023] The judgment module is used to determine whether there are matching feature points in the foreground region of the first image frame and the foreground region of the second image frame.

[0024] The solver module is used to solve the rotation and translation matrix based on the matching feature points if they exist, otherwise it solves the rotation and translation matrix based on the default method.

[0025] The compositing module is used to correct the foreground region of the second image frame based on the rotation and translation matrix, and to generate a composite image based on the first image frame and the corrected second image frame.

[0026] This application provides an electronic device, including a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The instruction, the program, the code set, or the instruction set is loaded and executed by the processor to implement the steps of an image generation method provided in any embodiment of this application.

[0027] This application provides a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by the processor of a mobile terminal, enables the mobile terminal to perform the steps of an image generation method provided in any embodiment of this application.

[0028] This application provides a computer program product that, when the instructions in the computer program product are executed by the processor of a mobile terminal, enables the mobile terminal to perform steps that implement an image generation method provided in any embodiment of this application.

[0029] The image generation method, system, electronic device, storage medium, and product provided in this application can synthesize high-resolution images by separating the foreground and background and selectively compositing them. Compared with traditional compositing methods in the prior art, this reduces the problem of poor compositing or low-resolution images caused by significant differences in the sharpness of the foreground and background, thus improving the user experience. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating an image generation method in one embodiment;

[0031] Figure 2 This is a block diagram of the image generation system in one embodiment;

[0032] Figure 3 This is a diagram of the internal structure of an electronic device in one embodiment; Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] In one embodiment, such as Figure 1 As shown, an image generation method is provided. This embodiment illustrates the application of this method to a mobile terminal, so that it can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and can be implemented through the interaction between the terminal and the server.

[0035] Step S101: Obtain a first image frame and a second image frame, and perform image segmentation on the first image frame and the second image frame respectively to obtain the foreground regions of the first image frame and the second image frame.

[0036] Specifically, by acquiring the first and second frames of the image, the foreground and background of the two images are segmented. The foreground includes, but is not limited to, people, still life, and objects, while the background includes, but is not limited to, landscapes and buildings.

[0037] In practice, the first and second frames can be captured by electronic devices, such as smartphones and tablets.

[0038] In the above embodiments, the foreground regions of the first image frame and the second image frame each include at least one block. The rotation and translation matrix corresponding to each block is solved based on the matching feature points in each block of the foreground regions of the first image frame and the second image frame.

[0039] Specifically, the first and second image frames are segmented using a deep learning semantic machine. Each segmented image comprises at least one block, and the number of pixels in the block is related to the foreground regions of the first and second image frames. Based on feature points of the foreground regions of the first and second image frames, the blocks are matched and then rotated or translated to align them.

[0040] In the above steps, the step of correcting the foreground region of the second image frame based on the rotation and translation matrix includes: correcting each block of the second image frame based on the rotation and translation matrix corresponding to each block.

[0041] Specifically, the first image frame is used as the reference frame to correct the second image frame to match the first image frame. Corrections are made to the blocks of the second image frame based on feature points to achieve a better image appearance.

[0042] Step S102: Determine whether there are matching feature points in the foreground region of the first image frame and the foreground region of the second image frame.

[0043] Specifically, it is determined whether feature points are present in the foreground of the first image frame and the foreground of the second image frame. For example, feature points can be simply understood as relatively prominent points in the image frame, such as contour points, bright spots in darker areas, dark spots in brighter areas, Harris corner points, etc.

[0044] Step S103: If matching feature points exist, the rotation and translation matrix is ​​solved based on the matching feature points; otherwise, the rotation and translation matrix is ​​solved based on the default method.

[0045] Specifically, if matching feature points exist, the second image frame is adjusted based on RANSAC. If no feature points exist, iterative calculation is performed using an image registration method that maximizes the enhanced correlation coefficient (ECC).

[0046] Step S104: Correct the foreground region of the second image frame based on the rotation and translation matrix, and generate a composite image based on the first image frame and the corrected second image frame.

[0047] Specifically, the second image frame is corrected according to the corresponding rules, and the corrected second image frame is combined with the first image frame to become a new image frame.

[0048] In the above steps, if matching feature points exist, the rotation and translation matrix is ​​solved based on the matching feature points; otherwise, the rotation and translation matrix is ​​solved using the default method, including:

[0049] If matching feature points exist, the RANSAC method is used to solve for the matching feature points to obtain the rotation and translation matrix; otherwise, the ECC method is used to iteratively calculate the minimum error of the foreground region of the first and second image frames to obtain the rotation and translation matrix.

[0050] Specifically, RANSAC stands for RANdom Sampling Consensus. It iteratively estimates the parameters of a mathematical model from a set of observations containing "outsides." For example, it can find a suitable 2D straight line from a set of observations. Assuming the observations contain both inside and outside points, where the inside points are approximately traversed by the line and the outside points are far from the line, RANSAC can derive a model calculated using only the inside points with sufficiently high probability. ECC stands for Enhanced Correlation Coefficient, which has the advantage of maintaining the luminance distortion of contrast and brightness.

[0051] In the above steps, before determining whether there are matching feature points in the foreground region of the first image frame and the foreground region of the second image frame, the method further includes:

[0052] Obtain feature points from the first image frame and feature points from the second image frame;

[0053] Feature matching is performed on the feature points of the first image frame and the feature points of the second image frame.

[0054] Specifically, feature points in the first and second image frames can be points where the difference in brightness exceeds a preset threshold, points where the color change exceeds a preset threshold, and corner points. Matching the feature points of the two image frames facilitates subsequent image processing.

[0055] In the above steps, the feature matching of feature points of the first image frame and feature points of the second image frame includes:

[0056] The first method is applied to perform a first matching of feature points of the first image frame and feature points of the second image frame; based on the first matching, the second method is applied to perform a second matching of feature points of the first image frame and feature points of the second image frame.

[0057] Specifically, a second precise matching is performed on the matched feature points to achieve better visual results.

[0058] In summary, the image generation method provided in this application can synthesize high-resolution images by separating the foreground and background and selectively compositing them. Compared with traditional compositing methods in the prior art, it reduces the problem of poor compositing or low resolution caused by excessive differences in the sharpness of the foreground and background, thus improving the user experience.

[0059] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Furthermore, Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0060] In one embodiment, such as Figure 2 As shown, an image generation system is provided, including: a segmentation module 210, a judgment module 220, a solution module 230, and a synthesis module 240.

[0061] The segmentation module 210 is used to obtain a first image frame and a second image frame, and to perform image segmentation on the first image frame and the second image frame respectively to obtain the foreground regions of the first image frame and the second image frame.

[0062] The judgment module 220 is used to determine whether there are matching feature points in the foreground region of the first image frame and the foreground region of the second image frame.

[0063] The solver module 230 is used to solve the rotation and translation matrix based on the matching feature points if matching feature points exist, otherwise it solves the rotation and translation matrix based on the default method.

[0064] The compositing module 240 is used to correct the foreground region of the second image frame based on the rotation and translation matrix, and to generate a composite image based on the first image frame and the corrected second image frame.

[0065] In one embodiment, the solving module 230 further includes:

[0066] If matching feature points exist, the RANSAC method is used to solve for the matching feature points to obtain the rotation and translation matrix; otherwise, the ECC method is used to iteratively calculate the minimum error of the foreground region of the first and second image frames to obtain the rotation and translation matrix.

[0067] In summary, the image generation system provided in this application can synthesize high-resolution images by separating the foreground and background and selectively compositing them. Compared with traditional compositing methods in the prior art, it reduces the problem of poor compositing or low resolution caused by significant differences in the sharpness of the foreground and background, thus improving the user experience.

[0068] For specific limitations regarding an image generation system, please refer to the limitations regarding an image generation method above, which will not be repeated here. The various modules in the aforementioned mobile terminal application launch system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0069] In one embodiment, an electronic device is provided, which may be a terminal device, and its internal structure diagram may be as follows: Figure 3 As shown, the terminal device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an application opening method. The display screen can be an LCD screen or a communication e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the terminal device's casing, or an external keyboard, touchpad, or mouse.

[0070] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the communication device to which the present application is applied. Specific communication devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0071] In one embodiment, the image generation system provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 3 The system runs on the computer device shown. The computer device's memory can store the various program modules that make up this image generation system, for example, Figure 2 The diagram shows a segmentation module, a judgment module, a solution module, and a synthesis module. The computer program, comprised of these modules, causes the processor to execute the steps of an image generation method according to various embodiments of this application described in this specification.

[0072] For example, such as Figure 3 The mobile terminal shown can be accessed via, for example... Figure 2 The illustrated application startup system of an electronic device includes a segmentation module that acquires a first image frame and a second image frame, and performs image segmentation on the first and second image frames respectively to obtain the foreground regions of the first and second image frames. A judgment module determines whether matching feature points exist in the foreground regions of the first and second image frames. A solution module solves for a rotation and translation matrix based on the matching feature points if they exist; otherwise, it solves for the rotation and translation matrix using a default method. A synthesis module corrects the foreground region of the second image frame based on the rotation and translation matrix and generates a synthesized image based on the first image frame and the corrected second image frame.

[0073] In one embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: obtaining a first image frame and a second image frame, and performing image segmentation on the first image frame and the second image frame respectively to obtain foreground regions of the first image frame and the second image frame; determining whether there are matching feature points in the foreground regions of the first image frame and the second image frame; if there are matching feature points, solving a rotation and translation matrix based on the matching feature points; otherwise, solving the rotation and translation matrix based on a default method; correcting the foreground region of the second image frame based on the rotation and translation matrix, and generating a composite image based on the first image frame and the corrected second image frame.

[0074] In one embodiment, when the processor executes the computer program, it further implements the following steps: if a matching feature point exists, the matching feature point is solved using the RANSAC method to obtain the rotation and translation matrix; otherwise, the minimum error of the foreground region of the first image frame and the second image frame is iteratively calculated using the ECC method to obtain the rotation and translation matrix.

[0075] In summary, the electronic device provided in this application can synthesize high-resolution images by separating the foreground and background and selectively compositing them. Compared with traditional compositing methods in the prior art, it reduces the problem of poor compositing or low resolution caused by excessive differences in the sharpness of the foreground and background, thus improving the user experience.

[0076] In one embodiment, a non-transitory computer-readable storage medium stores a computer program that, when executed by a processor, performs the following steps: obtaining a first image frame and a second image frame, and performing image segmentation on the first image frame and the second image frame respectively to obtain foreground regions of the first image frame and the second image frame; determining whether there are matching feature points in the foreground regions of the first image frame and the second image frame; if there are matching feature points, solving a rotation and translation matrix based on the matching feature points, otherwise solving the rotation and translation matrix based on a default method; correcting the foreground region of the second image frame based on the rotation and translation matrix, and generating a composite image based on the first image frame and the corrected second image frame.

[0077] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if a matching feature point exists, the matching feature point is solved using the RANSAC method to obtain the rotation and translation matrix; otherwise, the minimum error of the foreground region of the first image frame and the second image frame is iteratively calculated using the ECC method to obtain the rotation and translation matrix.

[0078] In summary, the non-transitory computer-readable storage medium provided in this application can synthesize high-resolution images by separating the foreground and background and selectively compositing the images. Compared with traditional compositing methods in the prior art, this reduces the problem of poor compositing or low-resolution images caused by excessive differences in the sharpness of the foreground and background, thus improving the user experience.

[0079] In one embodiment, a computer program product is provided. When the instructions in the computer program product are executed by the processor of a mobile terminal, the communication device is enabled to perform the following steps: obtaining a first image frame and a second image frame, and performing image segmentation on the first image frame and the second image frame respectively to obtain foreground regions of the first image frame and the second image frame; determining whether there are matching feature points in the foreground regions of the first image frame and the second image frame; if there are matching feature points, solving the rotation and translation matrix based on the matching feature points, otherwise solving the rotation and translation matrix based on a default method; correcting the foreground region of the second image frame based on the rotation and translation matrix, and generating a composite image based on the first image frame and the corrected second image frame.

[0080] In one embodiment, if matching feature points exist, the RANSAC method is used to solve for the matching feature points to obtain the rotation and translation matrix; otherwise, the ECC method is used to iteratively calculate the minimum error of the foreground regions of the first and second image frames to obtain the rotation and translation matrix.

[0081] In summary, by separating the foreground and background and selectively compositing the images, high-resolution images can be synthesized. Compared to traditional compositing methods in existing technologies, this reduces the problem of poor compositing or low-resolution images caused by significant differences in sharpness between the foreground and background, thus improving the user experience.

[0082] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM), etc.

[0083] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image generation method characterized by, The method comprises the following steps: obtaining a first image frame and a second image frame, and performing image segmentation on the first image frame and the second image frame respectively to obtain foreground regions of the first image frame and the second image frame; determining whether there are matching feature points in the foreground regions of the first image frame and the second image frame; if there are matching feature points, solving a rotation and translation matrix based on the matching feature points, otherwise solving the rotation and translation matrix based on a default method; correcting the foreground region of the second image frame based on the rotation and translation matrix, and generating a composite image according to the first image frame and the corrected second image frame; the foreground regions of the first image frame and the second image frame each comprise at least one block, and the blocks are aligned by rotating and translating the blocks according to the matching feature points in the blocks in the foreground regions of the first image frame and the second image frame respectively, and a rotation and translation matrix corresponding to each block is solved, wherein the number of pixels contained in the block is related to the foreground regions of the first image frame and the second image frame; wherein the correction of the foreground region of the second image frame based on the rotation and translation matrix comprises: correcting each block of the second image frame based on the rotation and translation matrix corresponding to the block respectively; the correction of each block of the second image frame based on the rotation and translation matrix corresponding to the block respectively comprises: taking the first image frame as a reference frame to correct the blocks of the second image frame to the first image frame based on the rotation and translation matrix corresponding to the blocks.

2. The image generation method of claim 1, wherein, if there are matching feature points, solving a rotation and translation matrix based on the matching feature points, otherwise solving the rotation and translation matrix based on a default method, comprises: if there are matching feature points, solving the rotation and translation matrix by using a RANSAC method, otherwise iteratively calculating the minimum error of the foreground regions of the first image frame and the second image frame by using an ECC method to obtain the rotation and translation matrix.

3. The image generation method of any of claims 1-2, wherein, Before determining whether there are matching feature points in the foreground regions of the first image frame and the second image frame, the method further comprises: obtaining feature points of the first image frame and feature points of the second image frame; performing feature matching on the feature points of the first image frame and the feature points of the second image frame.

4. The image generation method according to claim 3, characterized by, The feature matching on the feature points of the first image frame and the feature points of the second image frame comprises: applying a first method to perform one-time matching on the feature points of the first image frame and the feature points of the second image frame; on the basis of the one-time matching, applying a second method to perform two-time matching on the feature points of the first image frame and the feature points of the second image frame.

5. An image generation system characterized by comprising: The method comprises the following steps: a segmentation module is configured to obtain a first image frame and a second image frame, and perform image segmentation on the first image frame and the second image frame respectively to obtain foreground regions of the first image frame and the second image frame; a determination module is configured to determine whether there are matching feature points in the foreground regions of the first image frame and the second image frame; The solving module is configured to solve a rotation and translation matrix based on the matched feature points if there are matched feature points, or solve the rotation and translation matrix based on a default method if there are no matched feature points; The synthesizing module is configured to correct the foreground region of the second image frame based on the rotation and translation matrix, and generate a synthesized image according to the first image frame and the corrected second image frame; The foreground region of each of the first image frame and the second image frame includes at least one block, and each block is rotated and translated to align with the corresponding block of the other image frame based on the matched feature points in the block, and a rotation and translation matrix corresponding to each block is solved, wherein the number of pixels contained in each block is related to the foreground region of the first image frame and the second image frame; The synthesizing module is specifically configured to correct each block of the second image frame based on the rotation and translation matrix corresponding to each block. The synthesizing module is specifically configured to take the first image frame as a reference frame, and correct the blocks of the second image frame to the first image frame based on the rotation and translation matrix corresponding to each block.

6. An electronic device, comprising: The apparatus includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the image generation method according to any one of claims 1-4.

7. A non-transitory computer-readable storage medium, comprising: When the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal is enabled to perform the image generation method according to any one of claims 1-4.

8. A computer program product, characterised in that, When the instructions in the computer program product are executed by the processor of the mobile terminal, the mobile terminal is enabled to perform the image generation method according to any one of claims 1-4.

Images