Image processing method and related device

Abstract

Embodiments of the present application disclose an image processing method, which can be applied to a scene of removing other subjects irrelevant to a first subject in an image or a video. The method comprises: acquiring a first image, the first image comprising the first subject, a second subject and a first background, the first background being image content in the first image except the first subject and the second subject; acquiring a first pose; acquiring camera model parameters of a terminal device; acquiring a second image based on the first pose and the camera model parameters, the second image being a background image of a scene corresponding to the first background in a preset 3D map, the second image not comprising the first subject and the second subject; and generating a first target image based on the second image and the first subject in the first image. The first target image with the first subject and a clean background can be obtained by removing other subjects irrelevant to the first subject in the first image.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method and related equipment. Background Technology

[0002] With the rapid iteration and popularization of smartphones and tablets, the built-in camera functions for taking photos and videos have become increasingly rich and powerful, gaining popularity among consumers. Meanwhile, in the internet age, sharing text, pictures, and videos on social media platforms has become a daily habit. Along with rising living standards and the accompanying rise of consumerism, thanks to the convenience of transportation and accommodation, more ordinary people are choosing to travel, using the camera functions of smartphones and other mobile devices to take photos anytime, anywhere and share them on social media platforms such as WeChat and Weibo. Typically, when taking photos at scenic spots or popular photo spots, there are many tourists, especially during holidays. When taking photos / videos to commemorate the occasion, in addition to the main subject, the background of the photos / videos often contains many passersby and objects. These passersby not only obscure the scenery but also make the background appear cluttered, seriously affecting the final product.

[0003] Currently, the traditional method to obtain a photo / video without passersby is to remove passersby / objects through offline post-processing, such as manually retouching photos using image editing tools (e.g., Photoshop / Snapseed) or manually retouching videos using video editing and design tools (e.g., After Effects).

[0004] However, the methods mentioned above are time-consuming, labor-intensive, and require a high level of expertise from the user. How to efficiently remove parts unrelated to the subject when taking photos is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] This application provides an image processing method and related apparatus. It can remove other subjects unrelated to the first subject from a first image to obtain a first target image containing the first subject and with a clean background.

[0006] The first aspect of this application provides an image processing method that can be applied to a scene where other subjects unrelated to a first subject are removed from an image or video. This method can be executed by an image processing device (e.g., a terminal device or a server) or by a component of the image processing device (e.g., a processor, a chip, or a chip system). The method includes: acquiring a first image, the first image including a first subject, a second subject, and a first background, the first background being the image content in the first image excluding the first and second subjects; acquiring a first pose, the first pose being the pose of the terminal device when acquiring the first image; acquiring camera model parameters of the terminal device, the camera model parameters including at least one of the terminal device's field of view and resolution; acquiring a second image based on the first pose and the camera model parameters, the second image being a background image of the scene corresponding to the first background in a preset 3D map, the second image not including the first and second subjects; and generating a first target image based on the second image and the first subject in the first image, the first target image not including the second subject, the first target image including the first subject and a target background, the target background being the image content in the first target image excluding the first subject, and the target background being a sub-region of the second image. The first subject, also known as the photographed subject, can be a person or object occupying the largest pixel area in the first image, or it can be a person or object located close to the terminal device. The first subject can also be a person or object determined in the first image by the user's operation (or understood as a subject the user wants to highlight in the image). Besides the methods mentioned above, there are other possible ways to determine the first subject, which are not limited here. Furthermore, the second subject can also be understood as another subject unrelated to the first subject, or a subject the user wants to remove from the first image, such as passersby or cluttered objects that obstruct the user's view.

[0007] In this embodiment, on the one hand, other subjects unrelated to the first subject can be removed from the first image to obtain a first target image with the first subject and a clean background. On the other hand, since the second image is related to the first pose, the background in the generated first target image has the same viewpoint as the background in the first image.

[0008] Optionally, in one possible implementation of the first aspect, the number of pixels of the first subject is greater than the number of pixels of the second subject. Alternatively, the subject with the larger area is considered the first subject. Or, the subject located in the central region of the first image is defined as the first subject, and the radius of the central region is a preset value. This central region can be a regular region (e.g., rectangle, circle, trapezoid, etc.) or an irregular region; no specific limitation is made here. If the central region is circular, then the preset value * 2 (i.e., the diameter of the central region) is less than the length or width of the image.

[0009] In this possible implementation, determining the first subject based on the area ratio of the main subject can save user operations, and by pre-setting, the efficiency of determining the first subject can be improved.

[0010] Alternatively, in one possible implementation of the first aspect, the aforementioned first subject is determined in response to the user's selection operation.

[0011] In this possible implementation, the first subject can be understood as the subject that the user wants to highlight in the image. The first subject is determined by the user's operation, thereby improving the user experience.

[0012] Alternatively, in one possible implementation of the first aspect, the first subject is positioned closer to the center of the first image relative to the second subject.

[0013] In this possible implementation, using the subject at the center of the image as the first subject can save user operations, and by pre-setting, the efficiency of determining the first subject can be improved.

[0014] Optionally, in one possible implementation of the first aspect, when the terminal device acquires the first image, the terminal device satisfies a preset condition, which includes: the angle between the direction of the plane where the terminal device screen is located and the vertical direction of the terminal device screen is θ, where θ is an acute angle.

[0015] In this possible implementation, the inaccuracy of the subsequently generated second image is avoided due to abnormal pose of the terminal device, or in other words, the mismatch between the subsequently generated second image and the scene content of the first background of the first image is avoided due to abnormal pose of the terminal device.

[0016] Alternatively, in one possible implementation of the first aspect, the value of θ ranges from 0 to 60 degrees.

[0017] Optionally, in one possible implementation of the first aspect, the above step of obtaining camera model parameters of the terminal device includes: displaying a first user interface to the user, the first user interface including parameter options, the parameter options including image resolution and shooting magnification; determining camera model parameters from the parameter options in response to a second operation by the user; or, obtaining pre-set camera model parameters. This can also be understood as converting the shooting magnification selected by the user into a field of view. The shooting magnification can also be expressed in terms of wide-angle, etc.

[0018] In this possible implementation, the camera model parameters can be determined based on the user's operation, and then a second image can be generated based on the camera model parameters set by the user. This can make the generated second image more in line with the user's needs and improve the user experience.

[0019] Optionally, in one possible implementation of the first aspect, the camera model parameters used to acquire the second image are different from the camera model parameters of the terminal device when acquiring the first image.

[0020] In this possible implementation, the size of the second image can be adjusted by adjusting the camera model parameters used to generate the second image, so that the camera model parameters used to generate the second image are not limited to the camera model parameters of the terminal device when acquiring the first image.

[0021] Optionally, in one possible implementation of the first aspect, the above steps: obtaining the second image based on the first pose and camera model parameters include: sending the first image, the first pose, and camera model parameters to the server; receiving the second image sent by the server, wherein the second image is obtained by processing the first image, the first pose, and the camera model parameters.

[0022] In this possible implementation, a second image can be generated by the server and received from the server, thereby reducing the computing power of the terminal device.

[0023] Optionally, in one possible implementation of the first aspect, the above steps: obtaining a second image based on a first pose and camera model parameters, include: generating an intrinsic parameter matrix of the terminal device based on the camera model parameters, the intrinsic parameter matrix being used to represent the transformation relationship between the three-dimensional spatial coordinates and the two-dimensional image coordinates of the terminal device imaging; obtaining a 3D map fragment from a 3D map based on a first background and a first pose, the 3D map fragment corresponding to the background image of the scene where the first background is located; converting multiple 3D points in the 3D map fragment into multiple 2D points and multiple depth values ​​corresponding to the multiple 2D points based on the first pose, the 3D map fragment, and the intrinsic parameter matrix; and generating a second image based on the multiple 2D points and the multiple depth values.

[0024] In this possible implementation, a clean background image that completely corresponds to the terminal device's perspective can be generated online in real time based on the first pose and 3D map fragments, which is more conducive to improving the subsequent fusion effect between the first subject and the second image.

[0025] Optionally, in one possible implementation of the first aspect, the above steps further include: acquiring a third image, wherein each pixel value in the third image is the same; acquiring a second image based on a first pose and camera model parameters, including: acquiring a second pose corresponding to the timestamp of a fourth image, wherein the fourth image is obtained by the server based on the camera model parameters of the terminal device, and the fourth image corresponds to the background image of the scene where the first background is located; calculating a third pose based on the first pose and the second pose; and projecting the fourth image onto the third image based on the depth information of the fourth image and the third pose to obtain the second image.

[0026] In this possible implementation, the second image can be predicted using historical information (i.e., the fourth image) and the first pose, so that the predicted second image has the same viewpoint as the first image.

[0027] Optionally, in one possible implementation of the first aspect, the above steps further include: obtaining a historical pose, wherein the historical pose is the pose of the terminal device when acquiring the fifth image before acquiring the first image, and the fifth image corresponds to the background image of the scene where the first background is located; obtaining a second image based on the first pose and camera model parameters, including: smoothing the first pose based on the historical pose to obtain a fourth pose; and obtaining the second image based on the fourth pose and camera model parameters.

[0028] In this possible implementation, the first pose can be smoothed using historical poses, and the second image can be obtained based on the smoothed first pose (i.e., the fourth pose). When the user is recording video, the second image can be smoothly transitioned with the image of the previous frame, thereby improving the user experience.

[0029] Optionally, in one possible implementation of the first aspect, the above step of generating a first target image based on a second image and a first subject in a first image includes: determining the position information of the first subject in the first image; and generating the first target image based on the position information, the second image, and the first subject.

[0030] In this possible implementation, by obtaining the position information of the first subject in the first image, the first target image without the second subject in the background can be generated better by using the position information.

[0031] Optionally, in one possible implementation of the first aspect, the above steps of generating a first target image based on location information, a second image, and a first subject include: separating the first subject from the first image; and fusing the second image and the first subject to obtain the first target image.

[0032] In this possible implementation, by separating the first subject from the first image and fusing it with the second image, a first target image with a clean background containing the first subject can be obtained.

[0033] Optionally, in one possible implementation of the first aspect, the first subject is a person; separating the first subject in the first image includes: obtaining the facial region of the person in the first image; using the facial region as a seed point to separate the person and obtain the first subject.

[0034] One possible implementation is to use the facial region as a seed point for character separation, which can ensure that the separated characters are more accurate and avoid separating parts that are irrelevant to the character.

[0035] Optionally, in one possible implementation of the first aspect, the above steps further include: obtaining a first average brightness value of each pixel in the first background; obtaining a second average brightness value of each pixel in the second image; generating a first target image based on the second image and the first subject in the first image, including: correcting the second image based on the first average brightness value, the second average brightness value, the overflow prevention function saturate, and the pixel values ​​of each pixel in the second image, wherein the overflow prevention function is used to determine that each pixel value in the corrected second image is between 0 and 255; generating the first target image based on the corrected second image and the first subject in the first image.

[0036] In this possible implementation, before merging the first subject and the second image, the brightness of the second image is adjusted to match the brightness of the replaced background (i.e., the first background), thereby avoiding excessive brightness differences between the first image and the first target image and improving the user experience.

[0037] Optionally, in one possible implementation of the first aspect, the above steps further include: displaying a first target image to a user; responding to a third operation by the user, adjusting the parameters of the first target image to obtain a second target image, the parameters including at least one of color temperature, brightness, blurring, and resolution; and displaying the second target image to the user.

[0038] In this possible implementation, the parameters of the generated first target image can be adjusted according to the user's operation, thereby obtaining a second target image with the parameters required by the user.

[0039] Optionally, in one possible implementation of the first aspect, the camera model parameters mentioned above also include the shooting type of the terminal device, which corresponds to taking a picture or recording a video; if the shooting type corresponds to recording a video, the method further includes: acquiring a video frame, wherein the first image or the first target image is a frame in the video frame; and adjusting the frame rate of the video frame to a preset frame rate.

[0040] In this possible implementation, the frame rate of the video frames can be adjusted according to the user's operation, thereby obtaining the video with the frame rate required by the user.

[0041] A second aspect of this application provides an image processing method. This method can be applied to scenes where other subjects unrelated to a first subject are removed from an image or video. The method can be executed by an image processing device (e.g., a server) or by a component of the image processing device (e.g., a processor, a chip, or a chip system). The method includes: receiving a first image, a first pose, and camera model parameters sent by a terminal device; the first image includes a first subject, a second subject, and a first background; the first background is the image content in the first image excluding the first and second subjects; the first pose is the pose of the terminal device when acquiring the first image; the camera model parameters include at least one of the terminal device's field of view and resolution; acquiring a 3D map fragment based on the first background and the first pose; generating a second image based on the first pose, the 3D map fragment, and the camera model parameters; the second image is a background image of a scene corresponding to the first background in a preset 3D map, and the second image does not include the first and second subjects; and sending the second image to the terminal device.

[0042] In this embodiment, on the one hand, a clean background image that perfectly corresponds to the terminal device's perspective can be generated online in real time based on the first pose and 3D map fragments, which is more conducive to improving the subsequent fusion effect between the first subject and the second image. On the other hand, since the second image is related to the first pose, the perspective of the second image in the generated first target image is consistent with that of the first background in the first image.

[0043] Optionally, in one possible implementation of the second aspect, the above step of obtaining a 3D map fragment based on a first background and a first pose includes: obtaining a 3D map fragment corresponding to the first background from a preset 3D map based on the first pose.

[0044] In this possible implementation, the intrinsic parameter matrix is ​​calculated by the terminal device, reducing the computing power consumption of the server.

[0045] Optionally, in one possible implementation of the second aspect, the above steps: generating a second image based on the first pose, the 3D map fragment, and the camera model parameters, include: generating an intrinsic parameter matrix based on the camera model parameters, the intrinsic parameter matrix being used to represent the transformation relationship between the three-dimensional spatial coordinates of the terminal device imaging and the two-dimensional image coordinates; converting multiple 3D points in the 3D map fragment into multiple 2D points and multiple depth values ​​corresponding to the multiple 2D points based on the first pose, the 3D map fragment, and the intrinsic parameter matrix; and generating a second image based on the multiple 2D points and the multiple depth values.

[0046] In this possible implementation, the server calculates the intrinsic parameter matrix, reducing the computational power consumption of the terminal device. By maintaining a high-precision map and using the first pose to obtain a 3D map fragment corresponding to the first background from the 3D map, the obtained 3D map fragment can be made to match the perspective of the first background in the first image. Furthermore, a clean background image perfectly corresponding to the terminal device's perspective can be generated online in real-time based on the first pose and the 3D map fragment, further improving the subsequent fusion effect between the first subject and the second image.

[0047] A third aspect of this application provides an image processing apparatus that can be applied to a scenario of removing other subjects unrelated to a first subject from an image or video. The image processing apparatus can be a terminal device or a server. The image processing apparatus includes: a first acquisition unit for acquiring a first image, the first image including a first subject, a second subject, and a first background, the first background being the image content in the first image excluding the first and second subjects; a second acquisition unit for acquiring a first pose, the first pose being the pose of the terminal device when acquiring the first image; a third acquisition unit for acquiring camera model parameters of the terminal device, the camera model parameters including at least one of the terminal device's field of view and resolution; a fourth acquisition unit for acquiring a second image based on the first pose and the camera model parameters, the second image being a background image of the scene corresponding to the first background in a preset 3D map, the second image not including the first and second subjects; and a generation unit for generating a first target image based on the second image and the first subject in the first image, the first target image not including the second subject, the first target image including the first subject and a target background, the target background being the image content in the first target image excluding the first subject, and the target background being a sub-region of the second image.

[0048] Optionally, in one possible implementation of the third aspect, the number of pixels of the first subject is greater than the number of pixels of the second subject.

[0049] Alternatively, in one possible implementation of the third aspect, the aforementioned first entity is determined in response to the user's selection operation.

[0050] Alternatively, in one possible implementation of the third aspect, the first subject is positioned closer to the center of the first image relative to the second subject.

[0051] Optionally, in one possible implementation of the third aspect, when the terminal device acquires the first image, the terminal device satisfies a preset condition, which includes: the angle between the direction of the plane where the terminal device screen is located and the vertical direction of the terminal device screen is θ, where θ is an acute angle.

[0052] Alternatively, in one possible implementation of the third aspect, the value of θ ranges from 0 to 60 degrees.

[0053] Optionally, in one possible implementation of the third aspect, the aforementioned third acquisition unit is specifically used to display a first user interface to the user, the first user interface including parameter options, the parameter options including image resolution and shooting magnification; the third acquisition unit is specifically used to determine camera model parameters from the parameter options in response to the user's second operation; or, the third acquisition unit is specifically used to acquire pre-set camera model parameters.

[0054] Alternatively, in one possible implementation of the third aspect, the camera model parameters used to acquire the second image are different from the camera model parameters of the terminal device when acquiring the first image.

[0055] Optionally, in one possible implementation of the third aspect, the fourth acquisition unit described above is specifically used to send the first image, the first pose, and camera model parameters to the server; the fourth acquisition unit is specifically used to receive the second image sent by the server, the second image being obtained by processing the first image, the first pose, and the camera model parameters.

[0056] Optionally, in one possible implementation of the third aspect, the aforementioned fourth acquisition unit is specifically used to generate an intrinsic parameter matrix of the terminal device based on camera model parameters. The intrinsic parameter matrix is ​​used to represent the transformation relationship between the three-dimensional spatial coordinates and the two-dimensional image coordinates of the terminal device imaging. The fourth acquisition unit is specifically used to acquire a 3D map fragment from the 3D map based on the first background and the first pose. The 3D map fragment corresponds to the background image of the scene where the first background is located. The fourth acquisition unit is specifically used to convert multiple 3D points in the 3D map fragment into multiple 2D points and multiple depth values ​​corresponding to the multiple 2D points based on the first pose, the 3D map fragment, and the intrinsic parameter matrix. The fourth acquisition unit is specifically used to generate a second image based on the multiple 2D points and the multiple depth values.

[0057] Optionally, in one possible implementation of the third aspect, the first acquisition unit is further configured to acquire a third image, wherein each pixel value in the third image is the same; the fourth acquisition unit is specifically configured to acquire a second pose corresponding to the timestamp of the fourth image, wherein the fourth image is obtained by the server based on the camera model parameters of the terminal device, and the fourth image corresponds to the background image of the scene where the first background is located; the fourth acquisition unit is specifically configured to calculate a third pose based on the first pose and the second pose; the fourth acquisition unit is specifically configured to project the fourth image onto the third image based on the depth information of the fourth image and the third pose to obtain a second image.

[0058] Optionally, in one possible implementation of the third aspect, the second acquisition unit described above is further used to acquire a historical pose, which is the pose of the terminal device when acquiring the fifth image before acquiring the first image, and the fifth image corresponds to the background image of the scene where the first background is located; the fourth acquisition unit is specifically used to smooth the first pose based on the historical pose to obtain the fourth pose; the fourth acquisition unit is specifically used to acquire the second image based on the fourth pose and camera model parameters.

[0059] Optionally, in one possible implementation of the third aspect, the aforementioned generating unit is specifically used to determine the position information of the first subject in the first image; the generating unit is specifically used to generate a first target image based on the position information, the second image, and the first subject.

[0060] Optionally, in one possible implementation of the third aspect, the aforementioned generating unit is specifically used to separate the first subject in the first image; the generating unit is specifically used to fuse the second image and the first subject based on location information to obtain the first target image.

[0061] Optionally, in one possible implementation of the third aspect, the first subject is a person; the generation unit is specifically used to obtain the facial region of the person in the first image; the generation unit is specifically used to separate the person using the facial region as a seed point to obtain the first subject.

[0062] Optionally, in one possible implementation of the third aspect, the image processing device further includes: a fifth acquisition unit, configured to acquire a first average brightness value of each pixel in the first background; the fifth acquisition unit is further configured to acquire a second average brightness value of each pixel in the second image; a generation unit, specifically configured to modify the second image based on the first average brightness value, the second average brightness value, the overflow prevention function saturate, and the pixel values ​​of each pixel in the second image, wherein the overflow prevention function is used to determine that each pixel value in the modified second image is between 0 and 255; and a generation unit, specifically configured to generate a first target image based on the modified second image and a first subject in the first image.

[0063] Optionally, in one possible implementation of the third aspect, the image processing device further includes: a display unit for displaying a first target image to a user; an adjustment unit for adjusting parameters of the first target image in response to a third operation by the user to obtain a second target image, the parameters including at least one of color temperature, brightness, blurring, and resolution; and the display unit is further configured to display the second target image to the user.

[0064] Optionally, in one possible implementation of the third aspect, the camera model parameters mentioned above also include the shooting type of the terminal device, which corresponds to taking a picture or recording a video; if the shooting type corresponds to recording a video, the first acquisition unit is further used to acquire a video frame, wherein the first image or the first target image is a frame in the video frame; the generation unit is specifically used to adjust the frame rate of the video frame to a preset frame rate.

[0065] A fourth aspect of this application provides an image processing apparatus that can be applied to a scenario of removing other subjects unrelated to a first subject from an image or video. The image processing apparatus can be a server. The image processing apparatus includes: a receiving unit for receiving a first image, a first pose, and camera model parameters sent by a terminal device; the first image includes a first subject, a second subject, and a first background; the first background is the image content in the first image excluding the first and second subjects; the first pose is the pose of the terminal device when acquiring the first image; the camera model parameters include at least one of the terminal device's field of view and resolution; an acquisition unit for acquiring a 3D map fragment based on the first background and the first pose; a generation unit for generating a second image based on the first pose, the 3D map fragment, and the camera model parameters; the second image is a background image of the scene corresponding to the first background in a preset 3D map, and the second image does not include the first and second subjects; and a sending unit for sending the second image to the terminal device.

[0066] Optionally, in one possible implementation of the fourth aspect, the aforementioned acquisition unit is specifically used to acquire a 3D map fragment corresponding to the first background from a preset 3D map based on the first pose.

[0067] Optionally, in one possible implementation of the fourth aspect, the aforementioned generation unit is specifically used to generate an intrinsic parameter matrix based on camera model parameters, the intrinsic parameter matrix being used to represent the transformation relationship between the three-dimensional spatial coordinates and the two-dimensional image coordinates of the terminal device imaging; the generation unit is specifically used to convert multiple 3D points in the 3D map fragment into multiple 2D points and multiple depth values ​​corresponding to the multiple 2D points based on the first pose, the 3D map fragment, and the intrinsic parameter matrix; the generation unit is specifically used to generate a second image based on the multiple 2D points and the multiple depth values.

[0068] The fifth aspect of this application provides an image processing apparatus that performs the method in the first aspect or any possible implementation thereof, or performs the method in the second aspect or any possible implementation thereof.

[0069] A sixth aspect of this application provides an image processing apparatus, comprising: a processor coupled to a memory for storing a program or instructions, wherein when the program or instructions are executed by the processor, the image processing apparatus implements the method of the first aspect or any possible implementation thereof, or implements the method of the second aspect or any possible implementation thereof.

[0070] The seventh aspect of this application provides a computer-readable medium having a computer program or instructions stored thereon, which, when run on a computer, cause the computer to perform the methods of the first aspect or any possible implementation thereof, or cause the computer to perform the methods of the second aspect or any possible implementation thereof.

[0071] The eighth aspect of this application provides a computer program product that, when executed on a computer, causes the computer to perform the methods of the first aspect or any possible implementation thereof, or causes the computer to perform the methods of the second aspect or any possible implementation thereof.

[0072] The technical effects of the third, fifth, sixth, seventh, and eighth aspects or any of their possible implementations can be found in the first aspect or the technical effects of different possible implementations of the first aspect, and will not be repeated here.

[0073] The technical effects of the fourth, fifth, sixth, seventh, and eighth aspects, or any one of their possible implementations, can be found in the second aspect or the technical effects of different possible implementations of the second aspect, and will not be repeated here.

[0074] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: acquiring a first image including a first subject; acquiring a first pose; acquiring camera model parameters of the terminal device; acquiring a second image based on the first pose and camera model parameters, the second image being the background of the scene corresponding to the first background in a preset 3D map, the first background including other subjects besides the first subject compared to the second image; generating a first target image based on the second image and the first subject in the first image. On the one hand, other subjects unrelated to the first subject can be removed from the first image to obtain a first target image with the first subject and a clean background. On the other hand, since the second image is related to the first pose, the generated first target image has the same viewpoint as the first background in the first image. Attached Figure Description

[0075] Figure 1 A schematic diagram of a communication system provided in an embodiment of the present invention;

[0076] Figure 2 A schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0077] Figure 3 A flowchart illustrating the image processing method provided in this application embodiment;

[0078] Figure 4 Example diagram of the first image provided for embodiments of this application;

[0079] Figure 5 A schematic diagram of a 6DOF pose of a terminal device provided in an embodiment of this application;

[0080] Figures 6-7 A schematic diagram of the user interface displayed on several terminal devices provided in the embodiments of this application;

[0081] Figure 8 Example diagram of a 3D map fragment provided in the embodiments of this application;

[0082] Figure 9 An example diagram illustrating the mapping from a 3D map fragment to a second image provided in an embodiment of this application;

[0083] Figure 10 Example diagram of the second image provided in the embodiments of this application;

[0084] Figure 11 Example diagram of the first main body image provided in the embodiments of this application;

[0085] Figure 12 Example diagram of the first target image provided in the embodiments of this application;

[0086] Figure 13 This is a schematic diagram of another communication system provided in an embodiment of the present invention;

[0087] Figures 14-17 Several structural schematic diagrams of the image processing device provided in the embodiments of this application are shown. Detailed Implementation

[0088] This application provides an image processing method and related apparatus. It can remove other subjects unrelated to the first subject from a first image to obtain a first target image containing the first subject and with a clean background.

[0089] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Those skilled in the art will understand that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the term "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0090] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0091] To facilitate understanding of the technical solutions provided in this application, some concepts will be introduced below.

[0092] 1. Field of view (FOV)

[0093] In optical engineering, the field of view, also known as the field of view angle, determines the range of vision of an optical instrument. Alternatively, it can be understood as the angle formed by the two edges of the maximum range through which the image of the target object can pass through the lens, with the lens as the vertex. The larger the field of view angle, the wider the field of vision, but the lower the optical magnification. Simply put, if the target object is beyond this angle, it will not be captured by the lens.

[0094] 2. Pixel value

[0095] A pixel value in an image can be a red-green-blue (RGB) color value, which can be a long integer representing the color. For example, a pixel value of 256*Red+100*Green+76*Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. Within each color component, the smaller the value, the lower the brightness; the larger the value, the higher the brightness. For grayscale images, the pixel value can be a grayscale value.

[0096] 3. Convolutional Neural Networks

[0097] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution between the same trainable filter and an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature maps, each composed of rectangularly arranged neural units. Neural units within the same feature map share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of an image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in an image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0098] Convolutional kernels can be initialized as matrices of random size. During the training of the convolutional neural network, the kernels can learn to acquire appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers of the convolutional neural network, while also reducing the risk of overfitting. The separation network, recognition network, detection network, depth estimation network, and other networks in the embodiments of this application can all be CNNs.

[0099] Please see Figure 1 This application provides a communication system, which includes a terminal device 101 and a server 102.

[0100] The terminal device 101 can be a device with photo and / or video recording capabilities, such as a camera, mobile phone, tablet, personal computer (PC), television, smart bracelet, smartwatch, or other terminal with photo or video recording functions. It is understood that... Figure 1 The terminal device 101 mentioned is just an example, and no specific limit is specified here.

[0101] Server 102, also known as a cloud device or cloud server, receives data sent by terminal device 101, processes the received data to obtain a processing result, and then sends the processing result back to terminal device 101. Furthermore, terminal device 101 can display the processing result to the user. Alternatively, terminal device 102 can further process the processing result and display the further processed result to the user.

[0102] Optionally, the terminal device 101 and the server 102 can interact through a communication network using any communication mechanism / standard. The communication network can be a wide area network (WAN), a local area network (LAN), a point-to-point connection, or any combination thereof. Specifically, the communication network can include a wireless network, a wired network, or a combination of wireless and wired networks. The wireless network includes, but is not limited to, any one or more combinations of: 5th-Generation (5G) systems, Long Term Evolution (LTE) systems, Global System for Mobile Communication (GSM) or Code Division Multiple Access (CDMA) networks, Wideband Code Division Multiple Access (WCDMA) networks, Wireless Fidelity (WiFi), Bluetooth, Zigbee, Radio Frequency Identification (RFID), long-range (Lora) wireless communication, and near-field communication (NFC). The wired network can include fiber optic communication networks or networks composed of coaxial cables.

[0103] Taking mobile phones as an example, Figure 2 The diagram shown is a partial structural representation of a mobile phone, a terminal device provided in an embodiment of this application. (Reference) Figure 2 The mobile phone includes components such as: a radio frequency (RF) circuit 210, a memory 220, an input unit 230, a display unit 240, a sensor 251, a camera 252, an audio circuit 260, a wireless fidelity (WiFi) module 270, a processor 280, and a power supply 290. Those skilled in the art will understand that... Figure 2The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0104] The following is combined Figure 2 A detailed introduction to each component of a mobile phone:

[0105] RF circuit 210 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 280; additionally, it transmits uplink data to the base station. Typically, RF circuit 210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 210 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0106] The memory 220 can be used to store software programs and modules. The processor 280 executes various functions and data processing of the mobile phone by running the software programs and modules stored in the memory 220. The memory 220 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 220 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0107] The input unit 230 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 230 may include a touch panel 231 and other input devices 232. The touch panel 231, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 231), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 231 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 280, and can also receive and execute commands sent by the processor 280. In addition, the touch panel 231 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 231, the input unit 230 may also include other input devices 232. Specifically, other input devices 232 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0108] The display unit 240 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 240 may include a display panel 241, which may optionally be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar display. Furthermore, a touch panel 231 may cover the display panel 241. When the touch panel 231 detects a touch operation on or near it, it transmits the information to the processor 280 to determine the type of touch event. Subsequently, the processor 280 provides corresponding visual output on the display panel 241 based on the type of touch event. Although in Figure 2 In this embodiment, the touch panel 231 and the display panel 241 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 231 and the display panel 241 can be integrated to realize the input and output functions of the mobile phone.

[0109] The mobile phone may also include at least one sensor 251, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 241 according to the ambient light level, and the proximity sensor can turn off the display panel 241 and / or backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, inertial measurement units (IMUs), and simultaneous localization and mapping (SLAM) sensors, will not be described in detail here.

[0110] Camera 252 is used to capture images and / or videos and transmit them to display panel 241 so that display panel 241 can display the images and / or videos to the user. Furthermore, camera 252 can be a monocular or binocular camera, located at the front (i.e., front-facing camera) or rear (i.e., rear-facing camera) of the phone's main body casing. Additionally, camera 252 can be an ultra-wide-angle camera, a wide-angle camera, or a telephoto camera, etc., and the specific type is not limited here.

[0111] Audio circuit 260, speaker 261, and microphone 262 provide an audio interface between the user and the mobile phone. Audio circuit 260 converts received audio data into electrical signals and transmits them to speaker 261, where speaker 261 converts them into sound signals for output. On the other hand, microphone 262 converts collected sound signals into electrical signals, which are received by audio circuit 260, converted into audio data, and then processed by processor 280 before being transmitted via RF circuit 210 to, for example, another mobile phone, or the audio data can be output to memory 220 for further processing.

[0112] WiFi is a short-range wireless transmission technology. Mobile phones, through their WiFi module 270, can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 2 WiFi module 270 is shown, but it is understood that it is not an essential component of a mobile phone.

[0113] The processor 280 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 220, and calls data stored in the memory 220 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 280 may include one or more processing units; preferably, the processor 280 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 280.

[0114] The mobile phone also includes a power supply 290 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 280 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0115] Although not shown, mobile phones may also include Bluetooth modules, etc., which will not be elaborated here.

[0116] In this embodiment of the application, the processor 280 included in the mobile phone can perform the functions of the terminal device in this embodiment of the application, which will not be described in detail here.

[0117] Currently, the traditional method for obtaining a photo / video without passersby is through offline post-processing to remove them. This includes manual photo retouching using image editing tools (such as Photoshop / Snapseed) or manual video retouching using video editing and design tools (such as After Effects). However, these methods are time-consuming, labor-intensive, and require a high level of expertise from the user. How to efficiently remove irrelevant parts from the subject when taking a photo is a pressing technical problem that needs to be solved.

[0118] To address the aforementioned issues, this application provides an image processing method that can process images in real time when a user takes a picture. This method can remove other subjects unrelated to the first subject from the first image, resulting in a target image with the first subject and a clean background.

[0119] The image processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. For ease of description, a mobile phone is used as an example of the terminal device.

[0120] The image processing method provided in this application can be executed by an image processing device or by a component of the image processing device (such as a processor, chip, or chip system). The image processing device can be a server or a terminal device; the following description assumes the image processing device is a terminal device. Of course, the image processing method can also be executed jointly by the terminal device and the server; this is not limited here.

[0121] Please see Figure 3 One embodiment of the image processing method in this application includes steps 301 to 305.

[0122] Step 301: Obtain a first image, which includes a first subject, a second subject, and a first background.

[0123] In this application embodiment, there are multiple ways for the terminal device to acquire the first image. It can be by capturing the first image through the terminal device, by receiving the first image sent by other devices, or by selecting the first image from the database, etc. The specific method is not limited here.

[0124] The first image in this embodiment includes a first subject, a second subject, and a first background. The first background is the image content in the first image excluding the first and second subjects. The subject can refer to a person or object, etc., and is not specifically limited here. The first subject can also be called the photographed subject. The first subject can be a person or object occupying the largest pixel area in the first image, or it can be a person or object located near the terminal device (or camera). The first subject can also be a person or object determined in the first image by the user's operation (or understood as a subject the user wants to highlight in the image). Besides the methods mentioned above, there can be other possible ways to determine the first subject, and is not specifically limited here. Furthermore, the other subjects besides the first subject can also be understood as other subjects unrelated to the first subject, or subjects the user wants to remove from the first image, such as passersby, cluttered objects that affect the user's vision, etc. In this embodiment, the first subject can be understood as the photographed object or person, etc., and is not specifically limited here. This embodiment only describes the first subject as a person.

[0125] For example, the first subject and the second subject are people, and the first image, the first subject in the first image, and the second subject can be as follows: Figure 4 As shown.

[0126] There are several ways to determine the first subject in the embodiments of this application, which are described below:

[0127] The first method is to determine the first subject based on the user's first action.

[0128] In this method, a first image is displayed to the user, and the first subject is determined based on the user's first action on the first image. This first action can be clicking, dragging, etc. This method can be understood as allowing the user to determine the subject they want to highlight in the image through their first action.

[0129] The second method involves determining the primary subject through pre-defined rules.

[0130] 1. The preset rule is: the subject with the larger number of pixels between the first subject and the second subject is the first subject.

[0131] This approach can be understood as defining the subject that occupies the largest area of ​​the first image between the first subject and the second subject as the first subject.

[0132] For example, such as Figure 4 As shown, the first image includes two main subjects, with the first subject occupying a larger area.

[0133] 2. The preset rule is: the subject located in the center region of the first image is determined as the first subject, and the radius of the center region is a preset value.

[0134] The central region can be a regular region (e.g., rectangle, circle, trapezoid, etc.) or an irregular region; no specific limitation is made here. If the central region is circular, then the preset value * 2 (i.e., the diameter of the central region) is less than the length or width of the image.

[0135] 3. The preset rule is: the first subject is closer to the center of the first image than the second subject. Or, it can be understood as determining the subject located in the central region of the first image as the first subject, and the radius of the central region is a preset value.

[0136] It is understandable that the above preset rules are just examples. In actual applications, there may be other forms of preset rules. For example, the first subject may also be a person or object close to the terminal device (or camera). The preset rules are not limited here.

[0137] Step 302: Obtain the first pose, which is the pose of the terminal device when acquiring the first image.

[0138] Optionally, when the terminal device acquires the first image, the terminal device meets preset conditions, including: the angle between the direction of the plane where the terminal device screen is located and the vertical direction of the terminal device screen is θ, where θ is a right angle or an acute angle. The value of θ can be from 0 to 60 degrees or from 0 to 90 degrees, etc.

[0139] The aforementioned preset conditions can be understood as detecting whether the terminal device's pose is abnormal. Specifically, this can be achieved by using IMU and magnetometer data through an extended Kalman filter (EKF) to obtain the terminal device's 3-degree-of-freedom (3DOF) pose, thereby calculating the angle between the vertical direction of the terminal device's screen and the direction of gravity. If the preset conditions are met, the terminal device's pose is determined to be normal, and the first pose is acquired. If the preset conditions are not met, the terminal device's pose is determined to be unfavorable for shooting. The 3DOF pose can be XYZ, or it can be roll (R), pitch (P), and yaw (Y). XYZ represents three translation axes; X can also be understood as X representing the left-right direction (or "motion"), Y representing the forward-backward direction (or "wave"), and Z representing the up-down direction (or "rise and fall").

[0140] The first pose in this embodiment is a 6-degree-of-freedom (6DOF) pose, which includes XYZ and RPY axes. This first pose can be understood as translation in three directions and rotation about three axes. For example, the first pose of the terminal device is as follows: Figure 5 As shown.

[0141] Optionally, the first pose is represented by a 4×4 matrix in the special euclidean group (SE3) space.

[0142] There are multiple ways to obtain the first pose in the embodiments of this application. It can be through pose calculation methods such as SLAM, visual positioning system (VPS), and visual inertial odometry (VIO). The specific method is not limited here.

[0143] Step 303: Obtain the camera model parameters of the terminal device.

[0144] In this embodiment, the terminal device can obtain camera model parameters in various ways. These methods include determining the camera model parameters by responding to a second user operation, receiving camera model parameters sent by other devices, or selecting camera model parameters from a database. No specific method is limited here. Of course, the camera model parameters can also be pre-set or default, such as factory-set camera model parameters or camera model parameters derived through online calibration.

[0145] In this embodiment, the camera model parameters used to acquire the second image may differ from the camera model parameters of the terminal device used to acquire the first image. This can also be understood as allowing the camera model parameters to be adjusted at any time according to the user's actual needs, thereby adjusting the size of the second image.

[0146] The following description uses the example of how the terminal device obtains camera model parameters as a way to respond to the user's second operation.

[0147] Optionally, the terminal device displays a first user interface to the user, the first user interface including parameter options, such as image resolution, shooting magnification, wide angle, etc. In response to a second operation by the user, the terminal device determines camera model parameters from the parameter options, including but not limited to field of view (FOV) and / or resolution. This can be understood as the terminal device determining the field of view based on the shooting magnification or wide angle selected by the user.

[0148] For example, please refer to Figure 6 In step 'a', the terminal device displays a first user interface 601, which includes parameter options such as a camera switching mode option 602, a gallery 603, a wide-angle / magnification option 604, and a settings option 605. The camera switching mode option 602 allows the user to select either the rear or front camera. The gallery 603 stores images taken by the user. The wide-angle / magnification option 604 allows the user to set the wide-angle angle. The settings option 605 allows the user to set parameters related to photos, videos, etc. Furthermore, the user can click and swipe down on the wide-angle / magnification option 604. Figure 6 As shown in b, switch to wide-angle mode. Alternatively, users can click settings option 605, and the terminal device will respond to the user's click by displaying... Figure 7 The interface includes parameter settings for photos (such as aspect ratio, color temperature, brightness, and blur) and parameter settings for videos (such as video resolution and frame rate). Users can click icon 701 to set the photo aspect ratio, icon 702 to set the video resolution, and icon 703 to set the video frame rate. It is understandable that... Figure 6 The 4:3 aspect ratio for photos, 1080p video resolution, and video frame rate shown in the example (c) are just examples. In practical applications, video resolutions can also be 720p, 4K, 8K, etc. Photo aspect ratios can also be 1:1, full-screen, etc., but this is not a specific limitation here. It is understandable that users can also choose recording modes, etc. The above description only uses the user-selected photo mode as an example.

[0149] In addition, by using the camera switching mode options selected by the user (e.g., front camera and rear camera), shooting mode (e.g., photo and video), and effect options (e.g., wide angle, bokeh, brightness, color temperature, photo ratio, resolution, frame rate, etc.), the user's touch swipe trajectory (or options) on the screen can be converted into a value of 0-1 (which can be called the Slide value). The Slide value is then mapped to the FOV value, brightness gain value, color temperature gain value, bokeh kernel radius value, etc., so that the terminal device can determine the FOV, brightness gain, color temperature gain, virtual kernel radius, etc.

[0150] Step 304: Obtain the second image based on the first pose and camera model parameters. The second image does not include the first subject and the second subject.

[0151] In this embodiment, the second image is a background image of the scene corresponding to the first background in a preset 3D map. The second image does not include the first subject or other subjects unrelated to the first subject (including the second subject). Furthermore, there are multiple ways for the terminal device to obtain the second image based on the first pose and camera model parameters, which are described below:

[0152] The first method involves the terminal device acquiring a second image based on the first image, the first pose, and camera model parameters.

[0153] Optionally, the terminal device acquires camera model parameters and generates a second image based on the first image, the first pose, and the camera model parameters.

[0154] In this application embodiment, the terminal device obtains the second image based on the first image, the first pose, and camera model parameters in various ways, which are described below:

[0155] 1. The terminal device generates a second image based on the first image, the first pose, and the camera model parameters.

[0156] Optionally, a high-precision map (i.e., a 3D map) can be pre-built or acquired, and a dataset of multiple locations / buildings can be collected using sensors (including buildings in the first image, or the location where the first image was taken). Humans in the image are detected and labeled as mask regions, and local visual features are extracted from non-mask regions in the image. A 3D map is then constructed using mapping techniques. These mapping techniques can be SLAM or structure from motion (SFM), etc., and are not limited here. The constructed 3D map can be an image, image features, RGB points, triangular patches, etc., and is not limited here.

[0157] Optionally, the terminal device obtains a 3D map fragment from the aforementioned 3D map based on the first background and the first pose of the first image. This 3D map fragment corresponds to the background image of the scene where the first background is located, or it can be understood as the 3D fragment corresponding to the scene content in the first image. After obtaining the camera model parameters, the terminal device can generate a camera intrinsic matrix based on the camera model parameters. This intrinsic matrix represents the transformation relationship between the three-dimensional spatial coordinates of the terminal device's imaging and the two-dimensional image coordinates. Then, based on the first pose, the 3D map fragment, and the intrinsic matrix, multiple 3D points in the 3D map fragment are converted into multiple 2D points and multiple depth values ​​corresponding to the multiple 2D points. A second image is generated based on the multiple 2D points and multiple depth values.

[0158] Alternatively, the aforementioned intrinsic parameter matrix K can be expressed as:

[0159]

[0160]

[0161] FOV∈[60°, 180°)

[0162] Resolution includes width and height.

[0163] Alternatively, 2D points can be generated using the following formula.

[0164] uv = K * Tcw * Pw;

[0165] Where uv is the two-dimensional image coordinate, K is the intrinsic parameter matrix, Tcw is the first pose, and Pw is the point or face vertex in the 3D map segment.

[0166] For example, 3D map fragments such as Figure 8 As shown, the process of converting multiple 3D points in a 3D map fragment into 2D points can be as follows: Figure 9 As shown. Taking four points as an example, X1, X2, X3, and X4 in a 3D map fragment are mapped to x1, x2, x3, and x4 in a 2D image by connecting these four points to the UV image coordinate system. This completes the conversion from 3D points to 2D points, resulting in... Figure 10 The second image shown.

[0167] 2. Obtain the second image from the server.

[0168] Optionally, after acquiring the first image, the first pose, and camera model parameters, the terminal device can send the first image, the first pose, and the camera model parameters to the server. The server generates an intrinsic parameter matrix for the terminal device based on the received camera model parameters. This intrinsic parameter matrix represents the transformation relationship between the three-dimensional spatial coordinates of the terminal device's imaging and the two-dimensional image coordinates. Then, based on the first pose, the 3D map fragment, and the intrinsic parameter matrix, multiple 3D points in the 3D map fragment are converted into multiple 2D points and multiple depth values ​​corresponding to the multiple 2D points. A second image is then generated based on the multiple 2D points and the multiple depth values.

[0169] In this method, the way the server generates the second image based on the first image, the first pose, and the camera model parameters is similar to the description of the terminal device generating the second image based on the first image, the first pose, and the camera model parameters. The difference is whether the generation of the second image is performed by the terminal device or by the server.

[0170] Understandably, the terminal device can first obtain the intrinsic parameter matrix based on the camera model parameters, and then send the first image, the first pose, and the intrinsic parameter matrix to the server. The server then generates the second image based on the first image, the first pose, and the intrinsic parameter matrix.

[0171] The second method involves the terminal device predicting the second image.

[0172] Optionally, the terminal device acquires a third image in which all pixel values ​​are identical. For example, the pixel values ​​in the third image are either 0 or 255. In this embodiment, the terminal device can acquire the third image in various ways, such as by receiving a third image sent by another device, by selecting a third image from a database, or by creating a third image itself; no specific method is limited here.

[0173] Optionally, the height and width of the third image are the same as those of the first image.

[0174] Optionally, a second pose corresponding to the timestamp of the fourth image is obtained. This fourth image is obtained by the server or terminal device based on the intrinsic parameter matrix (or camera model parameters). The generation method of the fourth image is similar to that of the second image, and will not be repeated here. The fourth image and the second image correspond to the same scene, or it can be understood that the fourth image corresponds to the background image of the scene where the first background is located.

[0175] The same scenario in the embodiments of this application can be understood as at least one of the following properties being satisfied:

[0176] 1. The same scene can refer to two images (e.g., the fourth image and the first background of the first image) having the same content. For example, the content (or area, region) overlapping between the fourth image and the first background is greater than or equal to 30%.

[0177] 2. "Same scene" can refer to the following: the distance between the first position of the terminal device when acquiring one image (corresponding to the fourth image) and the second position of the terminal device when acquiring another image (i.e., the first image) is less than a certain threshold (e.g., the distance between the first and second positions is 1 meter, the threshold is 2 meters, meaning the distance is less than the threshold, then the fourth image and the first background can be determined to be images acquired in the same scene); and / or the overlap angle of the field of view of the two images (e.g., the image corresponding to the fourth image and the first image) is greater than a certain threshold (e.g., the overlap angle between the first and second viewpoints is greater than 30 degrees); and / or the difference in the rotation angle of the device acquiring the two images is less than a certain threshold. The rotation angle can be the horizontal rotation angle of the device or the downward rotation angle of the camera.

[0178] The location mentioned above can be a relative location or a geographic location. If the location is a relative location, the relative location between the first device and the second location can be determined by establishing a scene model or other means. If the location is a geographic location, it can be the location of the terminal device determined based on the Global Positioning System (GPS) or the BeiDou Navigation Satellite System.

[0179] 3. The same scene can also be judged based on light intensity. For example, the similarity between the weather type when the fourth image was captured and the weather type when the first image was captured can be used to determine if the fourth and first images belong to the same scene. For instance, if both the fourth and first images were captured on sunny days, then they belong to the same scene. If both the fourth and first images were captured on sunny days, then they do not belong to the same scene.

[0180] 4. The same scene can also refer to the texture similarity between the fourth image and the first background being greater than or equal to a certain threshold. Generally, this method needs to be combined with the other methods mentioned above for determination.

[0181] It is understandable that the above method of determining whether the fourth image and the first background (or the first image) are the same scene is just an example. In actual applications, there may be other methods, which are not limited here.

[0182] Furthermore, after obtaining the second pose, a third pose can be generated based on the first and second poses, and then the fourth image can be projected onto the third image based on the third pose and the depth information of the fourth image to obtain the second image.

[0183] Optionally, the above-mentioned generation of the third pose based on the first pose and the second pose can specifically be: Third pose = First pose / Second pose. This third pose can also be understood as a relative pose.

[0184] In addition, the specific method for projecting the fourth image onto the third image to obtain the second image based on the third pose and the depth information of the fourth image can be as follows: for each pixel position P1(u,v) in the fourth image, calculate the pixel position P2 and the depth value d2 projected onto the third image by P1(u,v) using the following formula.

[0185] Pc = Third pose * (P1 * d1)

[0186] P2 = K * Pc;

[0187] d2=Pc[2]

[0188] Where Pc is a three-dimensional point, d1 is the depth of P1, and Pc[2] represents P C The value of the third dimension in a three-dimensional point.

[0189] Optionally, if the RGB value at position P2 in the third image is empty, or if the RGB value at position P2 in the third image is not empty, but d2 is less than the depth in P2, then the RGB color value stored in P1 is assigned to P2. If the RGB value at position P2 in the third image is not empty, but d2 is greater than or equal to the depth in P2, then the RGB color value at position P2 is not modified, or it can be understood that no assignment is needed.

[0190] This method can be understood as predicting the second image based on historical background information and relative pose.

[0191] Optionally, the second method described above (i.e., the terminal device predicts the second image) can be a method for obtaining the second image when the terminal device is unable to communicate with the server due to network abnormalities.

[0192] It is understandable that the two methods for obtaining the second image mentioned above are just examples. In practical applications, other methods can also be used to obtain the second image, which are not limited here.

[0193] Step 305: Obtain a first target image based on the first subject in the second image and the first image. The first target image includes the first subject and a target background, and the target background is a sub-region of the second image.

[0194] After acquiring the second image, the terminal device can obtain the first target image based on the second image and the first subject in the first image. This can also be understood as replacing the first background in the first image with the second image to obtain the first target image. The first target image includes the first subject and a target background. The target background is the image content in the first target image excluding the first subject. The target background corresponds to the second image (or can be understood as a sub-region of the second image), and the first target image does not include the second subject.

[0195] Optionally, the position information of the first subject in the first image is first determined. Then, a first target image is generated based on the position information, the second image, and the first subject. The position information can be two-dimensional coordinates of the first image, which can determine which pixel values ​​in the second image the first subject replaces. This ensures that the position of the first subject in the first target image is approximately the same as its position in the first image, thereby improving the user's visual experience.

[0196] Furthermore, the terminal device separates the first subject from the first image to obtain the first subject, and fuses the first subject with the second image according to the location information to obtain the first target image.

[0197] In this embodiment, the method for separating the first subject from the first image can be by using a neural network to separate the first image to obtain the first subject image, or by determining the first subject through user operation, or by other methods, which are not limited here. For details on how to determine the first subject, please refer to the description of step 302 above, which will not be repeated here.

[0198] Optionally, the terminal device determining the first subject based on the user's operation may specifically include: the terminal device receiving the user's operation and responding to the operation to determine the terminal device's screen coordinates or screen area. The screen or screen area displays a first image, and the operation can be a click, double-click, drag, etc., for example, the user double-clicks on the first image or selects a first subject area. Then, the terminal device can determine the first subject in the first image and separate the first subject from the first image.

[0199] In one possible implementation, the user clicks on the first image displayed on the screen, the terminal device records the coordinates of the user's click on the screen as Pos(x,y), and performs face detection on the first image in an area centered at Pos(x,y) with a radius of a preset threshold. If a face is detected, the face is determined to be the face of the first subject.

[0200] In another possible implementation, the terminal device performs face detection using the entire first image as the detection area. If at least two faces are detected, and the current image is taken with the front camera, the face with the larger face area in the detection area is selected as the first subject's face. If at least two faces are detected, the terminal device can determine the first subject's face based on the user's action. This action could be the user tapping the screen or entering information, etc., and is not limited here.

[0201] Optionally, after determining the face (or facial region) of the first subject, the face of the first subject is used as a seed point to perform human body separation, thereby separating the first subject corresponding to the first subject in the first image to obtain the first subject image.

[0202] For example, the first subject image is as follows: Figure 11 As shown.

[0203] It is understood that there are various methods for face detection and / or human body separation (e.g., Mask R-CNN model) in the embodiments of this application. The above description is only an example and is not limited here.

[0204] Furthermore, the terminal device can also correct the first subject image and the second image based on user operations or preset threshold values ​​of parameters, so that the parameters of the first subject image and the second image are consistent, or that the difference between the first subject image and the second image is less than a threshold or the same. Here, parameters are used to indicate image features, such as brightness, color temperature, sharpness, and bokeh kernel radius. The following description only uses the correction of the second image's parameters based on the first subject image as an example. It is understood that in practical applications, the parameters of the first subject image can also be corrected based on the second image; this is not specifically limited here.

[0205] Optionally, the terminal device can obtain the first average brightness value of each pixel in the background portion (i.e., the first background) of the first image, where the background portion is the image other than the first subject in the first image, and obtain the second average brightness value of each pixel in the second image. Then, based on the first average brightness value, the second average brightness value, the saturate function (to prevent overflow), and the pixel values ​​of each pixel in the second image, the terminal device corrects the second image. The saturate function is used to determine that each pixel value in the corrected second image is between 0 and 225. The terminal device then generates a first target image based on the corrected second image and the first subject of the first image.

[0206] For example, the corrected second image is obtained using the following formula.

[0207] RGB(u',v')=Saturate[RGB(u,v)*V Ratio ]

[0208]

[0209] Wherein, V1 is the first average brightness value of the background part of the first image, the background part is the image in the first image excluding the first subject, V2 is the second average brightness value of the second image, RGB(u',v') is the pixel value of each point in the corrected second image, and RGB(u,v) is the pixel value of each point in the uncorrected second image.

[0210] Optionally, the terminal device can determine the brightness gain Vgain based on the user's operation, and multiply the RGB three-channel values ​​of the fourth image by Vgain respectively to obtain the corrected fourth image. For example, the corrected fourth image can be obtained using the following formula.

[0211] RGB(u',v')=Saturate[RGB(u,v)*(1+V gain )];

[0212] Wherein, RGB(u',v') represents the pixel value of each point in the corrected second image, and RGB(u,v) represents the pixel value of each point in the uncorrected second image.

[0213] Optionally, the terminal device can determine the color temperature gain Hgain based on user operation or pre-set parameters. If Hgain is greater than or equal to 0.5, the R channel value of the second image is multiplied. If Hgain is less than 0.5, the B channel value of the second image is multiplied. For example, the corrected fourth image is obtained using the following formula.

[0214] R(u',v')=Saturate[R(u,v)*(0.5+H gain )];

[0215] B(u',v')=Saturate[B(u,v)*(0.5+H gain )];

[0216] Where R(u',v') is the R channel value in the corrected second image, and R(u,v) is the R channel value in the uncorrected second image. B(u',v') is the B channel value in the corrected second image, and B(u,v) is the B channel value in the uncorrected second image.

[0217] Optionally, the terminal device can use super-resolution (SR) to upscale the lower-resolution image in the first subject image and the second image, thereby making the resolution of the first subject image and the second image the same.

[0218] Optionally, the terminal device can determine the virtual kernel radius based on user operation or pre-setting, and use the virtual kernel radius as the Gaussian filtering radius to perform Gaussian filtering on the RGB three-channel values ​​of the second image.

[0219] It is understood that the correction operation performed by the terminal device on the second image can be at least one of the above-mentioned corrections to the brightness, color temperature, sharpness, and blur kernel radius of the second image, and no specific limitation is made here.

[0220] After acquiring the first subject image and the second image (or the corrected second image), the terminal device can fuse the first subject image and the second image (or the corrected second image) to obtain the first target image. Alternatively, the first target image can be obtained by replacing the first background in the first image with the second image (or the corrected second image). The first background is an image in the first image other than the first subject, and the specific method is not limited here. The fusion technique used to fuse the first subject image and the second image (or the corrected second image) can be based on a neural network or other methods, and the specific method is not limited here.

[0221] Optionally, the pixel values ​​of the first subject image can be assigned to the corresponding areas of the second image (or the modified second image), and filtering operations (such as average filtering, bilateral filtering, etc.) can be performed on the edge areas of the first subject image to generate the first target image.

[0222] For example, the first target image is as follows: Figure 12 As shown.

[0223] In one possible implementation, steps 301 to 305 occur during the shooting preview stream, meaning the first image is the preview image and the first target image is the image after the shutter button is pressed. This approach can be applied to image processing scenarios where the user is shooting in real time. In another possible implementation, steps 301 to 305 occur after the shooting preview stream, which can also be understood as the first image being the image after the shutter button is pressed. This approach can be applied to offline image processing scenarios.

[0224] In this embodiment, on the one hand, other subjects unrelated to the first subject can be removed from the first image to obtain a first target image with the first subject and a clean background. On the other hand, since the second image is related to the first pose, the second image in the generated first target image has the same viewpoint as the first background in the first image. Furthermore, parameters such as sharpness, brightness, and color temperature can be adjusted between the second image and the first subject image to ensure that the sharpness, brightness, and color temperature of the fused first target image are consistent, thereby improving the user experience.

[0225] Optionally, the above method may further include acquiring a historical pose, which is the pose of the terminal device when acquiring the fifth image before acquiring the first image, and the first image and the fifth image correspond to the same scene (refer to the previous description). Therefore, acquiring the second image based on the first pose in the above method may specifically include: smoothing the first pose based on the historical pose to obtain a fourth pose, and acquiring the second image based on the fourth pose. Alternatively, it can be understood as using the fourth pose as the basis for the above method. Figure 3 The second pose in the illustrated embodiment is then processed.

[0226] In the case of video recording, this method allows the second image generated based on the fourth pose to transition naturally with the first background of the previous frame image.

[0227] It should be noted that if the user initially selects recording mode, the above method also includes acquiring video frames, where the first target image is a frame from the video frame. The terminal device can then adjust the video frame rate to a preset frame rate or adjust the video resolution to a preset resolution. Of course, the frame rate or video resolution can also be determined based on the user's actions; specific details are not limited here.

[0228] Optionally, after acquiring the first target image, the terminal device can display the first target image to the user and, in response to the user's second operation, adjust the parameters of the first target image to obtain the second target image. These parameters include at least one of color temperature, brightness, blurring, resolution, and frame rate. In other words, the parameters of the generated image can be adjusted before or after the user generates the first target image. This method can improve the image display effect and user experience.

[0229] The following describes a communication system and an image processing method applied to the communication system provided by embodiments of this application. This image processing method is jointly executed by a terminal device and a server. It is mainly applied to scenarios where the terminal device takes photos or records videos. The terminal device captures one or more first subjects, and a clean fourth image is generated by real-time precise pose calculation of the terminal device combined with a high-precision map from a cloud server. This fourth image is then fused with the first subject image and post-processed. This process can remove other subjects unrelated to the first subject from the original image, resulting in a target image or video with the first subject and a clean background.

[0230] Please see Figure 13 This application provides another communication system. The communication system includes a server 1301 and a terminal device 1302.

[0231] The server 1301 includes a first computing unit 101, a first storage unit 102, and a first network transmission unit 103. The first computing unit 101 runs the operating system of the server 1301 and generates images. The first storage unit 102 stores high-precision maps. The first network transmission unit 103 transmits data with the terminal device 102. The terminal device 1302 includes a sensor unit 201, a second computing unit 202, an interactive display unit 203, a second storage unit 204, and a second network transmission unit 205. The sensor unit 201 acquires the position, pose, images / videos, etc., of the terminal device 1302. The second computing unit 202 runs the operating system of the terminal device 1302 and processes images (e.g., pose calculation, image fusion, etc.). The interactive display unit 203 allows the terminal device 1302 to interact with the user. The second storage unit 204 stores algorithm data, etc., of the terminal device 1302. The second network transmission unit 205 transmits data with the server 1301.

[0232] Optionally, the first computing unit 101 may include a central processing unit (CPU), a graphics processing unit (GPU) / neural-network processing unit (NPU), cache, registers, etc. The first storage unit 102 may include RAM and external storage. The first network transmission unit 103 may include an upload module, a download module, encoding / decoding, and privacy encryption. The upload module is used to transmit data to the terminal device 1302, and the download module is used to receive data transmitted by the terminal device 1302. Privacy encryption is used to encrypt the data to be sent. Encoding / decoding is used to encode or decode the encrypted data to be sent or received.

[0233] Optionally, the sensor unit 201 includes at least one of the following: a visual sensor (e.g., a camera), a global positioning system (GPS), an inertial measurement unit (IMU), a magnetometer, a depth sensor / laser sensor, and a network unit (e.g., WiFi, 4G / 5G). The depth sensor / laser sensor can be a time-of-flight (TOF) or light detection and ranging (Lidar) sensor, etc., and is not specifically limited here. The visual sensor (e.g., a camera) is used to acquire 2D graphic information of the scene. GPS is used to acquire the latitude and longitude geographic location information of the terminal device. The IMU is used to acquire motion information of the terminal device, such as linear acceleration and angular velocity. The magnetometer is used to acquire the yaw angle of the terminal device. The depth sensor / laser sensor is used to acquire depth information of the scene. The network unit (e.g., WiFi, 4G / 5G) is used for communication and data exchange between the terminal device and the server. The second computing unit 202 may include a CPU, GPU / NPU, cache, registers, etc. The interactive display unit 203 includes a display screen, touchpad, speaker / microphone, vibrator, etc., used to acquire user input information and implement the effects of the rendering algorithm (e.g., images before and after processing). The second storage unit 204 may include memory and external storage. It is used for storing algorithm data, reading and writing user-local and temporary data, etc. The second network transmission unit 205 may include an upload module, a download module, encoding / decoding, and privacy encryption. The upload module is used to transmit data to the server 1301, and the download module is used to receive data transmitted by the server 1301. Privacy encryption is used to encrypt the data to be sent. Encoding / decoding is used to encode or decode the encrypted data to be sent or received.

[0234] The process of generating target images by each unit in this communication system is described in detail below.

[0235] Sensor unit 201 is used to acquire a first image including a first subject and a first pose when the terminal device 1302 acquires the first image. It then sends the first image and the first pose to second computing unit 202. The first image includes at least two subjects and scene content. Interactive display unit 203 is used to determine the camera model parameters selected by the user and send the camera model parameters to second computing unit 202. Second computing unit 202 is used to receive the first image and the first pose sent by sensor unit 201, and to receive the camera model parameters sent by interactive display unit 203.

[0236] If the network connection between server 1301 and terminal device 1302 is normal, the second computing unit 202 sends the first image, the first pose, and camera model parameters to the second storage unit 204. The second storage unit 204 sends the first image, the first pose, and camera model parameters to the second network transmission unit 205. The second network transmission unit 205 sends the first image, the first pose, and camera model parameters to the first network transmission unit 103. The first network transmission unit 103 sends the first image, the first pose, and camera model parameters to the first storage unit 102. The first computing unit 101 obtains the first image, the first pose, and camera model parameters through the first storage unit 102 and generates a second image. The second image is then transmitted to the second computing unit 202.

[0237] If the network connection between server 1301 and terminal device 1302 is abnormal, the second computing unit 202 is used to generate a second image based on the first pose, the intrinsic parameter matrix of the terminal device, and the fourth image stored in the second storage unit 204.

[0238] The second computing unit 202 is also used to separate the first subject from the first image to obtain a first subject image, and to fuse the second image and the first subject image to obtain a first target image.

[0239] In this embodiment, sensor unit 201 is used to capture one or more first subjects. The second computing unit 202 calculates the pose in real time and combines it with the high-precision map stored in the first storage unit 102 to generate a clean fourth image. Then, the image is fused and post-processed with the first subject image. Other subjects unrelated to the first subject can be removed from the original image (i.e. the first image) to obtain a target image or target video with the first subject and a clean background.

[0240] It is understandable that the functions of each unit in this communication system are the same as those described above. Figure 3 The descriptions in the illustrated embodiments are similar and will not be repeated here.

[0241] The image processing method in the embodiments of this application has been described above. The image processing device in the embodiments of this application is described below. Please refer to [link / reference]. Figure 14 One embodiment of the image processing device (terminal device or server) in this application includes:

[0242] The first acquisition unit 1401 is used to acquire a first image, the first image including a first subject, a second subject and a first background, the first background being the image content in the first image other than the first subject and the second subject;

[0243] The second acquisition unit 1402 is used to acquire the first pose, which is the pose of the terminal device when acquiring the first image.

[0244] The third acquisition unit 1403 is used to acquire camera model parameters of the terminal device, the camera model parameters including at least one of the field of view and resolution of the terminal device.

[0245] The fourth acquisition unit 1404 is used to acquire a second image based on the first pose and camera model parameters. The second image is a background image of the scene corresponding to the first background in a preset 3D map. The second image does not include the first subject and the second subject.

[0246] The generation unit 1405 is used to generate a first target image based on a second image and a first subject in a first image. The first target image does not include the second subject. The first target image includes the first subject and a target background. The target background is the image content in the first target image other than the first subject. The target background is a sub-region of the second image.

[0247] In this embodiment, the operations performed by each unit in the image processing device are the same as described above. Figures 3-13 The embodiments shown are similar and will not be repeated here.

[0248] In this embodiment, on one hand, the generation unit 1405 can remove other subjects unrelated to the first subject from the first image to obtain a first target image with the first subject and a clean background. On the other hand, since the second image acquired by the fourth acquisition unit 1405 is related to the first pose, the second image in the first target image generated by the generation unit 1405 has the same viewpoint as the first background in the first image.

[0249] Please see Figure 15 Another embodiment of the image processing device (terminal device or server) in this application includes:

[0250] The first acquisition unit 1501 is used to acquire a first image, the first image including a first subject, a second subject and a first background, the first background being the image content in the first image other than the first subject and the second subject;

[0251] The second acquisition unit 1502 is used to acquire the first pose, which is the pose of the terminal device when acquiring the first image.

[0252] The third acquisition unit 1503 is used to acquire camera model parameters of the terminal device, the camera model parameters including at least one of the field of view and resolution of the terminal device.

[0253] The fourth acquisition unit 1504 is used to acquire a second image based on the first pose and camera model parameters. The second image is a background image of the scene corresponding to the first background in a preset 3D map. The second image does not include the first subject and the second subject.

[0254] The generation unit 1505 is used to generate a first target image based on a second image and a first subject in a first image. The first target image does not include the second subject. The first target image includes the first subject and a target background. The target background is the image content in the first target image other than the first subject. The target background is a sub-region of the second image.

[0255] In this embodiment, the image processing device may further include:

[0256] The fifth acquisition unit 1506 is used to acquire the third image, in which all pixel values ​​are the same;

[0257] Display unit 1507 is used to display the first target image to the user;

[0258] Adjustment unit 1508 is configured to respond to a second user operation by adjusting parameters of the first target image to obtain a second target image. These parameters include at least one of color temperature, brightness, blurring, and resolution.

[0259] In this embodiment, the operations performed by each unit in the image processing device are the same as described above. Figures 3-13 The embodiments shown are similar and will not be repeated here.

[0260] In this embodiment, on one hand, the generation unit 1505 can remove other subjects unrelated to the first subject from the first image to obtain a first target image with the first subject and a clean background. On the other hand, since the second image acquired by the fourth acquisition unit 1504 is related to the first pose, the second image in the first target image generated by the generation unit 1505 has the same viewpoint as the first background in the first image. The adjustment unit 1508 can also determine the parameters of the image according to the user's operation, which can improve the user experience.

[0261] Please see Figure 16 Another embodiment of the image processing device (e.g., including a server) in this application includes:

[0262] The receiving unit 1601 is used to receive a first image, a first pose, and camera model parameters sent by the terminal device. The first image includes a first subject, a second subject, and a first background. The first background is the image content in the first image other than the first subject and the second subject. The first pose is the pose of the terminal device when it captures the first image. The camera model parameters include at least one of the field of view and resolution of the terminal device.

[0263] Acquisition unit 1602 is used to acquire 3D map fragments based on scene content and first pose.

[0264] The generation unit 1603 is used to generate a second image based on the first pose, a 3D map fragment and camera model parameters. The second image is a background image of the scene corresponding to the first background in the preset 3D map. The second image does not include the first subject and the second subject.

[0265] The sending unit 1604 is used to send a second image to the terminal device.

[0266] In this embodiment, the operations performed by each unit in the image processing device are the same as described above. Figures 3-13 The embodiments shown are similar and will not be repeated here.

[0267] In this embodiment, on the one hand, the generation unit 1603 can generate a clean background image that completely corresponds to the terminal device's perspective in real time based on the first pose and the 3D map fragment, which is more conducive to improving the fusion effect of the subsequent first subject image and the second image. On the other hand, since the second image is related to the first pose, the second image in the first target image generated by the generation unit 1603 has the same perspective as the first background in the first image.

[0268] See Figure 17 This application provides a schematic diagram of another image processing device. The image processing device may include a processor 1701, a memory 1702, and a communication interface 1703. The processor 1701, memory 1702, and communication interface 1703 are interconnected via lines. The memory 1702 stores program instructions and data.

[0269] The aforementioned are stored in memory 1702 Figures 3-13 In the corresponding implementation shown, the program instructions and data corresponding to the steps executed by the device are described.

[0270] Processor 1701, for performing the aforementioned Figures 3-13 The steps performed by the device are shown in any of the embodiments illustrated.

[0271] Communication interface 1703 can be used for receiving and sending data, and for performing the aforementioned tasks. Figures 3-13 The steps related to acquiring, sending, and receiving in any of the embodiments shown.

[0272] In one implementation, the image processing device may include, relative to Figure 17 More or fewer components are merely illustrative in this application and are not intended to limit the scope of the application.

[0273] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0274] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0275] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented wholly or partially through software, hardware, firmware, or any combination thereof.

[0276] When the integrated unit is implemented using software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0277] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a first image, the first image including a first subject, a second subject and a first background, the first background being the image content in the first image other than the first subject and the second subject; Obtain the first pose, which is the pose of the terminal device when it acquires the first image; Obtain the camera model parameters of the terminal device, wherein the camera model parameters include at least one of the field of view and resolution of the terminal device; A second image is obtained based on the first pose and the camera model parameters. The second image is a background image of the scene corresponding to the first background in a preset 3D map. The second image does not include the first subject and the second subject. A first target image is generated based on the second image and the first subject in the first image. The first target image does not include the second subject. The first target image includes the first subject and a target background. The target background is the image content in the first target image other than the first subject. The target background is a sub-region of the second image.

2. The method according to claim 1, characterized in that, The number of pixels in the first subject is greater than the number of pixels in the second subject.

3. The method according to claim 1, characterized in that, The first entity is determined in response to the user's selection operation.

4. The method according to claim 1, characterized in that, The first subject is located closer to the center of the first image than the second subject.

5. The method according to claim 1, characterized in that, When the terminal device acquires the first image, the terminal device meets a preset condition, which includes: the angle between the direction of the plane where the terminal device screen is located and the vertical direction of the terminal device screen is θ, where θ is an acute angle.

6. The method according to claim 5, characterized in that, The value of θ ranges from 0 to 60 degrees.

7. The method according to claim 1, characterized in that, The step of obtaining the camera model parameters of the terminal device includes: A first user interface is displayed to the user, the first user interface including parameter options, the parameter options including image resolution and shooting magnification; In response to the user's second action, the camera model parameters are determined from the parameter options; or, Obtain the pre-set camera model parameters.

8. The method according to claim 1, characterized in that, The camera model parameters used to acquire the second image are different from the camera model parameters of the terminal device when acquiring the first image.

9. The method according to any one of claims 1 to 8, characterized in that, The step of obtaining the second image based on the first pose and the camera model parameters includes: Send the first image, the first pose, and the camera model parameters to the server; The system receives the second image sent by the server, which is obtained by processing the first image, the first pose, and the camera model parameters.

10. The method according to any one of claims 1 to 8, characterized in that, The step of obtaining the second image based on the first pose and the camera model parameters includes: The intrinsic parameter matrix of the terminal device is generated based on the camera model parameters. The intrinsic parameter matrix is ​​used to represent the transformation relationship between the three-dimensional spatial coordinates and the two-dimensional image coordinates of the terminal device imaging. Based on the first background and the first pose, a 3D map fragment is obtained from the 3D map, and the 3D map fragment corresponds to the background image of the scene where the first background is located; Based on the first pose, the 3D map fragment, and the intrinsic parameter matrix, multiple 3D points in the 3D map fragment are converted into multiple 2D points and multiple depth values ​​corresponding to the multiple 2D points; The second image is generated based on the plurality of 2D points and the plurality of depth values.

11. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Obtain a third image, wherein all pixel values ​​in the third image are the same; The step of obtaining the second image based on the first pose and the camera model parameters includes: Obtain the second pose corresponding to the timestamp of the fourth image, wherein the fourth image is obtained by the server based on the camera model parameters of the terminal device, and the fourth image corresponds to the background image of the scene where the first background is located. Calculate the third pose based on the first pose and the second pose; Based on the depth information of the fourth image and the third pose, the fourth image is projected onto the third image to obtain the second image.

12. The method according to any one of claims 1 to 8, characterized in that, The method further includes: The historical pose is the pose of the terminal device when it acquired the fifth image before acquiring the first image, and the fifth image corresponds to the background image of the scene where the first background is located. The step of obtaining the second image based on the first pose and the camera model parameters includes: Based on the historical pose, the first pose is smoothed to obtain the fourth pose. The second image is obtained based on the fourth pose and the camera model parameters.

13. The method according to any one of claims 1 to 8, characterized in that, The step of generating a first target image based on the second image and the first subject in the first image includes: Determine the position information of the first subject in the first image; The first target image is generated based on the location information, the second image, and the first subject.

14. The method according to claim 13, characterized in that, The step of generating the first target image based on the location information, the second image, and the first subject includes: Separate the first subject from the first image; The first target image is obtained by fusing the second image with the first subject based on the location information.

15. The method according to claim 14, characterized in that, The first subject is a person; separating the first subject from the first image includes: Obtain the facial region of the person in the first image; Using the facial region as a seed point, the person is separated to obtain the first subject.

16. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Obtain the first average brightness value of each pixel in the first background; Obtain the second average brightness value of each pixel in the second image; The step of generating a first target image based on the second image and the first subject in the first image includes: The second image is corrected based on the first average brightness value, the second average brightness value, the anti-overflow function saturate, and the pixel values ​​of each pixel in the second image. The anti-overflow function is used to determine that each pixel value in the corrected second image is between 0 and 255. The first target image is generated based on the corrected second image and the first subject in the first image.

17. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Display the first target image to the user; In response to the user's third operation, the parameters of the first target image are adjusted to obtain the second target image, wherein the parameters include at least one of color temperature, brightness, blurring and resolution; The second target image is displayed to the user.

18. The method according to any one of claims 1 to 8, characterized in that, The camera model parameters also include the shooting type of the terminal device, which corresponds to taking a photo or recording a video. If the shooting type corresponds to the video recording, the method further includes: Acquire video frames, wherein the first image or the first target image is a frame in the video frames; Adjust the frame rate of the video frame to the preset frame rate.

19. An image processing method, characterized in that, The method includes: The terminal device receives a first image, a first pose, and camera model parameters. The first image includes a first subject, a second subject, and a first background. The first background is the image content in the first image other than the first subject and the second subject. The first pose is the pose of the terminal device when it captures the first image. The camera model parameters include at least one of the field of view and resolution of the terminal device. 3D map fragments are obtained based on the first background and the first pose; A second image is generated based on the first pose, the 3D map fragment, and the camera model parameters. The second image is a background image of the scene corresponding to the first background in the preset 3D map. The second image does not include the first subject and the second subject. The second image is sent to the terminal device.

20. The method according to claim 19, characterized in that, The process of obtaining a 3D map fragment based on the first background and the first pose includes: Based on the first pose, the 3D map fragment corresponding to the first background is obtained from a preset 3D map.

21. The method according to claim 19 or 20, characterized in that, The generation of the second image based on the first pose, the 3D map fragment, and the camera model parameters includes: An intrinsic parameter matrix is ​​generated based on the camera model parameters. The intrinsic parameter matrix is ​​used to represent the transformation relationship between the three-dimensional spatial coordinates and the two-dimensional image coordinates of the terminal device imaging. Based on the first pose, the 3D map fragment, and the intrinsic parameter matrix, multiple 3D points in the 3D map fragment are converted into multiple 2D points and multiple depth values ​​corresponding to the multiple 2D points; The second image is generated based on the plurality of 2D points and the plurality of depth values.

22. An image processing device, characterized in that, The image processing device includes: The first acquisition unit is used to acquire a first image, the first image including a first subject, a second subject and a first background, the first background being the image content in the first image other than the first subject and the second subject; The second acquisition unit is used to acquire the first pose, which is the pose of the terminal device when it acquires the first image. The third acquisition unit is used to acquire camera model parameters of the terminal device, wherein the camera model parameters include at least one of the field of view and resolution of the terminal device. The fourth acquisition unit is used to acquire a second image based on the first pose and the camera model parameters. The second image is a background image of the scene corresponding to the first background in a preset 3D map. The second image does not include the first subject and the second subject. The generation unit is configured to generate a first target image based on the second image and the first subject in the first image, wherein the first target image does not include the second subject, the first target image includes the first subject and a target background, the target background is the image content in the first target image other than the first subject, and the target background is a sub-region of the second image.

23. The device according to claim 22, characterized in that, The number of pixels in the first subject is greater than the number of pixels in the second subject.

24. The device according to claim 22, characterized in that, The first entity is determined in response to the user's selection operation.

25. The device according to claim 22, characterized in that, The first subject is located closer to the center of the first image than the second subject.

26. The device according to claim 22, characterized in that, When the terminal device acquires the first image, the terminal device meets a preset condition, which includes: the angle between the direction of the plane where the terminal device screen is located and the vertical direction of the terminal device screen is θ, where θ is an acute angle.

27. The device according to claim 26, characterized in that, The value of θ ranges from 0 to 60 degrees.

28. The device according to claim 22, characterized in that, The third acquisition unit is specifically used to display a first user interface to the user. The first user interface includes parameter options, including image resolution and shooting magnification. The third acquisition unit is specifically used to determine the camera model parameters from the parameter options in response to the user's second operation; or, The third acquisition unit is specifically used to acquire the pre-set camera model parameters.

29. The device according to claim 22, characterized in that, The camera model parameters used to acquire the second image are different from the camera model parameters of the terminal device when acquiring the first image.

30. The device according to any one of claims 22 to 29, characterized in that, The fourth acquisition unit is specifically used to send the first image, the first pose, and the camera model parameters to the server. The fourth acquisition unit is specifically used to receive the second image sent by the server, the second image being obtained by processing the first image, the first pose, and the camera model parameters.

31. The device according to any one of claims 22 to 29, characterized in that, The fourth acquisition unit is specifically used to generate the intrinsic parameter matrix of the terminal device based on the camera model parameters. The intrinsic parameter matrix is ​​used to represent the transformation relationship between the three-dimensional spatial coordinates and the two-dimensional image coordinates of the terminal device imaging. The fourth acquisition unit is specifically used to acquire a 3D map fragment from the 3D map based on the first background and the first pose, wherein the 3D map fragment corresponds to the background image of the scene where the first background is located; The fourth acquisition unit is specifically used to convert multiple 3D points in the 3D map fragment into multiple 2D points and multiple depth values ​​corresponding to the multiple 2D points based on the first pose, the 3D map fragment and the intrinsic parameter matrix. The fourth acquisition unit is specifically used to generate the second image based on the plurality of 2D points and the plurality of depth values.

32. The device according to any one of claims 22 to 29, characterized in that, The first acquisition unit is further configured to acquire a third image, wherein each pixel value in the third image is the same; The fourth acquisition unit is specifically used to acquire the second pose corresponding to the timestamp of the fourth image. The fourth image is obtained by the server according to the camera model parameters of the terminal device. The fourth image corresponds to the background image of the scene where the first background is located. The fourth acquisition unit is specifically used to calculate the third pose based on the first pose and the second pose; The fourth acquisition unit is specifically used to project the fourth image onto the third image based on the depth information of the fourth image and the third pose to obtain the second image.

33. The device according to any one of claims 22 to 29, characterized in that, The second acquisition unit is further configured to acquire historical pose, wherein the historical pose is the pose of the terminal device when acquiring the fifth image before acquiring the first image, and the fifth image corresponds to the background image of the scene in which the first background is located. The fourth acquisition unit is specifically used to smooth the first pose based on the historical pose to obtain the fourth pose. The fourth acquisition unit is specifically used to acquire the second image based on the fourth pose and the camera model parameters.

34. The device according to any one of claims 22 to 29, characterized in that, The generating unit is specifically used to determine the position information of the first subject in the first image; The generation unit is specifically used to generate the first target image based on the location information, the second image, and the first subject.

35. The device according to claim 34, characterized in that, The generating unit is specifically used to separate the first subject from the first image; The generation unit is specifically used to fuse the second image and the first subject based on the location information to obtain the first target image.

36. The device according to claim 35, characterized in that, The first subject is a person; the generating unit is specifically used to obtain the facial region of the person in the first image; The generation unit is specifically used to separate the person using the facial region as a seed point to obtain the first subject.

37. The device according to any one of claims 22 to 29, characterized in that, The image processing device further includes: The fifth acquisition unit is used to acquire the first average brightness value of each pixel in the first background. The fifth acquisition unit is further configured to acquire the second average brightness value of each pixel in the second image; The generation unit is specifically used to correct the second image based on the first average brightness, the second average brightness, the anti-overflow function saturate, and the pixel values ​​of each pixel in the second image. The anti-overflow function is used to determine that each pixel value in the corrected second image is between 0 and 255. The generation unit is specifically used to generate the first target image based on the corrected second image and the first subject in the first image.

38. The device according to any one of claims 22 to 29, characterized in that, The image processing device further includes: A display unit is used to display the first target image to the user; An adjustment unit is configured to respond to a third operation by the user to adjust the parameters of the first target image to obtain a second target image, wherein the parameters include at least one of color temperature, brightness, blurring, and resolution. The display unit is also used to display the second target image to the user.

39. The device according to any one of claims 22 to 29, characterized in that, The camera model parameters also include the shooting type of the terminal device, which corresponds to taking a photo or recording a video. If the shooting type corresponds to the video recording, the first acquisition unit is further configured to acquire video frames, wherein the first image or the first target image is a frame in the video frames; The generation unit is specifically used to adjust the frame rate of the video frame to a preset frame rate.

40. An image processing device, characterized in that, The image processing device includes: The receiving unit is used to receive a first image, a first pose, and camera model parameters sent by a terminal device. The first image includes a first subject, a second subject, and a first background. The first background is the image content in the first image other than the first subject and the second subject. The first pose is the pose of the terminal device when it acquires the first image. The camera model parameters include at least one of the field of view and resolution of the terminal device. The acquisition unit is used to acquire a 3D map fragment based on the first background and the first pose; The generation unit is used to generate a second image based on the first pose, the 3D map fragment, and the camera model parameters. The second image is a background image in a preset 3D map corresponding to the scene where the first background is located. The second image does not include the first subject and the second subject. A sending unit is used to send the second image to the terminal device.

41. The device according to claim 40, characterized in that, The acquisition unit is specifically used to acquire the 3D map fragment corresponding to the first background from a preset 3D map based on the first pose.

42. The device according to claim 40 or 41, characterized in that, The generation unit is specifically used to generate an intrinsic parameter matrix based on the camera model parameters. The intrinsic parameter matrix is ​​used to represent the transformation relationship between the three-dimensional spatial coordinates and the two-dimensional image coordinates of the terminal device imaging. The generation unit is specifically used to convert multiple 3D points in the 3D map fragment into multiple 2D points and multiple depth values ​​corresponding to the multiple 2D points based on the first pose, the 3D map fragment and the intrinsic parameter matrix. The generation unit is specifically used to generate the second image based on the plurality of 2D points and the plurality of depth values.

43. An image processing device, characterized in that, include: A processor coupled to a memory for storing programs or instructions that, when executed by the processor, cause the image processing device to perform the method as claimed in any one of claims 1 to 18, or cause the image processing device to perform the method as claimed in any one of claims 19 to 21.

44. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 18, or cause the computer to perform the method as claimed in any one of claims 19 to 21.

45. A computer program product, characterized in that, When the computer program product is executed on a computer, it causes the computer to perform the method as described in any one of claims 1 to 18, or causes the computer to perform the method as described in any one of claims 19 to 21.

Images