Image processing method, apparatus, and device

Abstract

An image processing method and apparatus and a device are provided, which relate to the field of image processing. After obtaining an image to be processed, a line segment to be corrected is obtained from the line segment included in the image to be processed by filtering based on at least one of IMU information and field of view included in the metadata. The image to be processed includes at least one of vertical line segments and horizontal line segments in a first scenario captured by a photographing device. The line segment to be corrected and the feature region of the image to be processed are corrected to obtain a processed image, and the feature region includes a background region, a portrait region and an edge region. Since the IMU information and field of view describe the characteristic information of the photographing device that captures the image to be processed, the line segment to be corrected can be accurately located based on the IMU information and field of view, and the line segment to be corrected and the feature region of the image to be processed are corrected so that the line segment in the processed image better follows the distribution characteristics of the horizontal line segment and the vertical line segment in the first scenario captured by the photographing device. This effectively improves the accuracy of correcting the tilt-shift distortion phenomenon of the image.

Description

[Technical field]

[0001] The present application relates to the field of image processing, and in particular to image processing methods and apparatus, and devices. [Background technology]

[0002] Generally, when a user uses a camera to take a picture, due to non-fixed reasons such as the tilt of the camera, tilt-shift distortion (e.g., tilt and convergence) occurs in the image on straight lines in the captured scenario. As a result, the composition of the image becomes unattractive. For example, in an image of a building taken from a low angle, the boundary lines of the exterior walls converge closely at the top of the building.

[0003] Currently, for images with tilt-shift distortion, a global correction is performed based on image content. However, for non-square content with special characteristics, the above correction method may cause correction errors, so the accuracy of the correction method is low. Summary of the Invention

[0004] The present application provides an image processing method and apparatus, as well as a device, for solving the problem of correction errors that occur when a global correction based on image content is performed on an image with tilt-shift distortion, and for improving the accuracy of performing tilt-shift distortion correction on an image.

[0005] According to a first aspect, there is provided an image processing method, the method being executed by a computing device. After acquiring an image to be processed, the computing device , Processing Based on at least one of the field of view and inertial measurement unit (IMU) information included in the metadata of the processed image Filtering, obtain a correction target line segment from a line segment included in the processing target image. The processing target image includes at least one of a vertical line segment and a horizontal line segment in a first scenario of the processing target image captured by the photographing device. It can be understood that the line segment included in the processing target image may be a line segment in the first scenario that is not tilted in the image, or a line segment in the first scenario that is tilted in the image. Furthermore, the computing device can correct the correction target line segment and a feature region of the processing target image to obtain a processed image, where the feature region includes a background region, a portrait region, and an edge region.

[0006] In this way, since the IMU information and the field of view describe the characteristic information of the image capture device that captures the target image, the line segments that need to be corrected are accurately located based on the IMU information and the field of view, the line segments to be corrected are selected from the line segments included in the target image, and the line segments to be corrected and the feature regions of the target image are corrected so that the line segments in the processed image better follow the distribution characteristics of the horizontal and vertical line segments in the first scenario captured by the image capture device. Compared with performing a global correction on the target image, performing a targeted correction on the line segments in the target image effectively improves the accuracy of performing tilt-shift distortion correction on the image.

[0007] For example, IMU information includes the horizontal and pitch tilt angles of the camera that captures the image to be processed. The field of view represents the maximum range that the camera can observe. The field of view is visually represented by an angle. The wider the field of view, the larger the observation range.

[0008] If the image capture device captures the first scenario from a low or high angle and a vertical line segment in the first scenario is tilted in the image, the image to be processed includes a line segment that is tilted in the image to be processed and is a vertical line segment in the first scenario.

[0009] If the imaging device captures a first scenario horizontally and vertical and horizontal line segments in the first scenario are tilted in the image, the image to be processed includes line segments that are tilted in the image to be processed and are vertical and horizontal line segments in the first scenario.

[0010] It should be noted that the computing device that performs the image processing of this embodiment of the present application may be a photographing device (e.g., a smartphone) that has a photographing function, or may be another device that has an image processing function (e.g., a server, a cloud device, an edge device, etc.).

[0011] The line segments to be corrected include at least one of the vertical line segments and the horizontal line segments in the first scenario. It can be understood that the line segments to be corrected include some or all of the line segments that are tilted in the image to be processed and are at least one of the vertical line segments and the horizontal line segments in the first scenario. For example, the image to be processed includes a building image and a portrait with tilt-shift distortion, and the line segments to be corrected include at least one of the vertical line segments and the horizontal line segments in the building that are tilted in the image, and the line segments to be corrected do not include the line segments that are tilted in the portrait. Only the tilted line segments in the building image are corrected, and the effect of longer legs is maintained in the portrait, creating a more beautiful image composition.

[0012] In a possible implementation, the image capture device displays the tilt-shift distortion control, and the method further comprises: after the image capture device receives an operation performed by a user on the tilt-shift distortion control, the image capture device , Processing Based on the metadata of the processed image Filteringand obtaining a correction target line segment from the line segments included in the processing target image, and correcting the correction target line segment and the feature region of the processing target image to obtain a processed image. The tilt-shift distortion control is performed on the image in a human-computer interactive manner, thereby reducing the complexity of the operation steps for a user to perform tilt-shift distortion correction on the image, and increasing the flexibility of performing tilt-shift distortion control on the image. Therefore, the user experience of processing the image is improved.

[0013] In another possible implementation, the method further includes: displaying the processed image after correcting the target line segments and the feature regions of the target image to obtain the processed image, so that a user can more intuitively see the image on which the tilt-shift distortion correction has been performed, which improves the user experience of processing the image.

[0014] In another possible implementation, the line segments to be corrected are filtered based on a line segment filtering policy that is formulated based on the content contained in the metadata. The line segment filtering policy indicates how to perform line segment filtering on the line segments contained in the processed image. For example, , Processing Based on the metadata of the processed image Filtering It is possible to obtain a line segment to be corrected from the line segments included in the image to be processed. , Me Based on line segment filtering policies determined based on the data Filtering , obtaining the corrected line segments from the line segments contained in the processed image. Thus, different line segment filtering policies are adapted based on the content contained in the metadata to improve the flexibility and accuracy of performing line segment filtering on the line segments contained in the processed image.

[0015] Example 1: Metadata includes field of view and IMU information. The line segment filtering policy dictates to perform line segment filtering based on the field of view and IMU information. Me Based on line segment filtering policies determined based on the data Filtering Obtaining a correction target line segment from the line segments included in the processing target image is to perform a global adjustment on M line segments included in the processing target image based on the field of view and the IMU information to obtain M adjusted line segments, and the M adjusted line segments satisfy the line segment angle distribution characteristic of the first scenario; ,La Based on in-segment slope threshold Filtering , obtaining N line segments to be corrected from the M adjusted line segments. The range of the line segment tilt threshold includes a horizontal line segment tilt threshold and a vertical line segment tilt threshold, where M and N are both positive integers, and N is less than or equal to M. In this way, since the IMU information indicates the tilt angle of the image capture device, the image capture device performs filtering on the tilted line segments in the image to be processed based on the IMU information and the field of view. This helps to improve the accuracy of the line segments, and further improves the accuracy of performing tilt-shift distortion correction on the image.

[0016] Example 2: Metadata does not include field of view Me Based on line segment filtering policies determined based on the data Filtering It is possible to obtain a line segment to be corrected from the line segments included in the image to be processed. , Processing Based on the image content of the target image Filtering and obtaining a correction target line segment from a line segment included in the processing target image, where the image content includes a line segment angle distribution feature. In this way, the image capture device performs an accurate line segment filtering based on the tilt angle of the line segment in the image content to further improve the accuracy of performing tilt-shift distortion correction on the image.

[0017] Example 3: Metadata includes field of view Muga , does not include IMU information Me Based on line segment filtering policies determined based on the data Filtering It is possible to obtain a line segment to be corrected from the line segments included in the image to be processed. , visual Based on the field and the image content of the image to be processed Filtering The method includes obtaining a correction target line segment from a line segment included in a processing target image, and the image content includes a line segment angle distribution feature. The inclination angle of the line segment varies within different field of view ranges, so that the wider the field of view, the larger the inclination angle of the edge line segment in the image. The image capture device performs an appropriate line segment filtering based on the field of view and the inclination angle of the line segment in the image content to further improve the accuracy of performing tilt-shift distortion correction on the image.

[0018] In another possible implementation, the method further includes prompting a user to select a line segment to be corrected from the recommended line segments obtained by filtering based on the image content of the image to be processed. Sometimes, a user may capture an image from a low angle or a high angle for composition effect. In this case, the user may not want to correct a tilted line segment in the image. In this way, the image capture device provides a user interaction line recommendation policy to prompt a user whether to perform line segment correction, and performs tilt-shift distortion correction on the image based on the user's request to improve the accuracy of performing tilt-shift distortion correction on the image.

[0019] In another possible implementation, the metadata further includes a trust identifier, the trust identifier indicating whether the metadata is trustworthy. , Me Based on line segment filtering policies determined based on the data Filtering,If the trust identifier indicates that the metadata is untrustworthy, it is not possible to obtain the line segment to be corrected from the line segments included in the image to be processed. , Processing Based on the image content of the target image Filtering , obtain a correction target line segment from the line segments included in the processing target image, and if the image content includes the line segment angle distribution feature, and the trust identifier indicates that the metadata is trustworthy, , Processing Based on at least one of the image content of the target image, the IMU information, and the field of view Filtering and obtaining a correction target line segment from a line segment included in the processing target image. In this way, before the line segment filtering is performed based on the IMU information and the field of view, it is first pre-determined whether the IMU information and the field of view are reliable, and a line segment filtering policy for the line segment filtering is determined based on the reliability of the IMU information and the field of view. Also, the accuracy of the line segment filtering result is improved, and the accuracy of the tilt-shift distortion correction for the image is improved.

[0020] Another possible implementation is , Processing Based on the metadata of the processed image Filtering Obtaining the to-be-corrected line segment from the line segments included in the to-be-processed image includes determining the to-be-corrected line segment based on the metadata-based filtered line segment and the image content-based filtered line segment. In this manner, the to-be-corrected line segment is determined with reference to the results of two different line segment filtering policies. This further improves the accuracy of performing the line segment filtering based on the IMU information and the field of view, and further improves the accuracy of performing the tilt-shift distortion correction on the image.

[0021] In another possible implementation, correcting the line segments to be corrected and the feature regions of the image to be processed to obtain a processed image includes: determining whether the number of the line segments to be corrected is greater than a quantity threshold and whether the length of the line segments to be corrected is greater than a length threshold; if the number of the line segments to be corrected is greater than the quantity threshold and the length of the line segments to be corrected is greater than the length threshold, correcting the line segments to be corrected to obtain a processed image; if the number of the line segments to be corrected is less than or equal to the quantity threshold or the length of the line segments to be corrected is less than or equal to the length threshold, determining whether the line segments to be corrected should be corrected based on the scenario features of a first scenario, the scenario features including architectural features and character features. In this way, when the line segments in the image are corrected, the scenario features and the user's intention are fully referenced, so that the corrected image is more in line with the distribution features of the line segments in the implementation scenario. Therefore, the accuracy of performing tilt-shift distortion correction on the image is effectively improved.

[0022] In another possible implementation, correcting the line segment to be corrected and the feature region of the processing image to obtain a processed image includes: constructing a straight line constraint to correct the line segment to be corrected, constructing a content-based homography constraint, a shape constraint, and a regular constraint to correct the background region, constructing a portrait constraint to correct the portrait region, and constructing an edge constraint to correct the edge region to obtain the processed image. In this way, constraints such as the straight line constraint, the content-based homography constraint, the shape constraint, the regular constraint, the portrait constraint, and the edge constraint are optimized together, so that the correction result is more reasonable and beautiful, the field of view is wider, and the composition is more beautiful.

[0023] If the line segments that need to be corrected are not obtained from the line segments contained in the target image by filtering based on the metadata of the target image, the target image is corrected based on the feature regions of the target image, which can effectively compensate for the line segment filtering error and further improve the accuracy of performing tilt-shift distortion correction on the image.

[0024] Another possible implementation is , Processing Based on the metadata of the processed image Filtering Obtaining the correction target line segment from the line segments included in the processing target image includes detecting the processing target image according to a line detection method to obtain a line segment set, the line segment set including M tilted line segments, where M is a positive integer.

[0025] In another possible implementation, after correcting the line segments to obtain a processed image, the method includes updating metadata (including the IMU information) of the processed image, for example deleting or modifying the IMU information, so that the accuracy of performing the column segment filtering is improved by using the IMU information.

[0026] According to a second aspect there is provided an image processing apparatus, the apparatus comprising a module configured to perform the image processing method of the first aspect or any one of its possible designs.

[0027] According to a third aspect, an image capture device is provided. The image capture device includes at least one processor and a memory. The memory is configured to store a group of computer instructions. When the processor functions as the image capture device of the first aspect or any one of the possible implementations of the first aspect and executes the group of computer instructions, the processor performs the operation steps of the image processing method of the first aspect or any one of the possible implementations of the first aspect.

[0028] According to a fourth aspect, a computing device is provided. The computing device includes at least one processor and a memory. The memory is configured to store a group of computer instructions. When the processor functions as an image capture device in the first aspect or any one of the possible implementations of the first aspect and executes the group of computer instructions, the processor performs the operation steps of the image processing method in the first aspect or any one of the possible implementations of the first aspect.

[0029] According to a fifth aspect, a computer-readable storage medium is provided, comprising computer software instructions which, when executed on a computing device, are capable of causing the computing device to perform the operational steps of the method of the first aspect or any one of the possible implementations of the first aspect.

[0030] According to a sixth aspect, there is provided a computer program product, which when executed on a computer, can cause the computing device to perform the operation steps of the method of the first aspect or any one of the possible implementations of the first aspect.

[0031] In the present application, based on the implementations provided in the above aspects, the implementations may be further combined to yield further implementations. [Brief description of the drawings]

[0032] [Figure 1] 1A-1C are schematic diagrams of a tilt-shift distorted image and a corrected image according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram of a tilt-shift distortion correction scenario according to an embodiment of the present application. [Diagram 3] 1 is a schematic diagram of a structure of an imaging device according to an embodiment of the present application; [Figure 4] 1 is a schematic diagram of an image processing method according to an embodiment of the present application; [Diagram 5] FIG. 2 is a schematic diagram of a metadata structure according to an embodiment of the present application; [Figure 6] FIG. 1 is a schematic diagram of performing line segment filtering based on IMU information and field of view according to an embodiment of the present application; [Figure 7] FIG. 2 is a schematic diagram of a visualization of line segment filtering according to an embodiment of the present application; [Figure 8A] FIG. 4 is a schematic diagram of another image processing method according to an embodiment of the present application. [Figure 8B] FIG. 4 is a schematic diagram of another image processing method according to an embodiment of the present application. [Figure 9] FIG. 2 is a schematic diagram of performing line segment filtering based on image content according to an embodiment of the present application; [Figure 10] FIG. 2 is a schematic diagram of performing line segment filtering based on field of view and image content according to an embodiment of the present application; [Figure 11] FIG. 4 is a schematic diagram of another image processing method according to an embodiment of the present application. [Figure 12A] FIG. 13 is a schematic diagram of yet another image processing method according to an embodiment of the present application. [Figure 12B] FIG. 13 is a schematic diagram of yet another image processing method according to an embodiment of the present application. [Figure 13(a)] FIG. 2 is a schematic diagram of a line segment selection interface according to an embodiment of the present application; [Figure 13(b)] FIG. 2 is a schematic diagram of a line segment selection interface according to an embodiment of the present application; [Figure 14A] FIG. 13 is a schematic diagram of yet another image processing method according to an embodiment of the present application. [Figure 14B] FIG. 13 is a schematic diagram of yet another image processing method according to an embodiment of the present application. [Figure 15(a)] FIG. 2 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 15(b)] FIG. 2 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 15(c)] FIG. 2 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 15(d)]FIG. 2 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 15(e)] FIG. 2 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 15(f)] FIG. 2 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 16(a)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(b)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(c)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(d)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(e)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(f)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(g)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 17] 1 is a schematic diagram of a structure of an image processing device according to an embodiment of the present application; [Figure 18] 1 is a schematic diagram of a structure of an imaging device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] An embodiment of the present application provides an image processing method, in particular, a method for performing tilt-shift distortion correction on an image. Specifically, a line segment to be corrected is obtained from a line segment included in the image to be processed by filtering based on metadata of the image to be processed. The line segment to be corrected may be a line segment that is tilted in the image to be processed and is at least one of a vertical line segment and a horizontal line segment in an actual scenario. The line segment to be corrected and the feature region of the image to be processed are corrected so that the line segment in the processed image better follows the distortion characteristics of the horizontal line segment and the vertical line segment in the actual scenario captured by the image capture device. Compared with performing a global correction on the image to be processed, performing a targeted correction on the tilted line segment in the image to be processed effectively improves the accuracy of performing tilt-shift distortion correction on the image.

[0034] Tilt-shift distortion refers to the phenomenon whereby originally parallel lines in a target plane of a photographed scenario are tilted or distorted in the imaging plane because the imaging plane of the imaging device is rotated about at least one of the x-, y-, and z-axes relative to the target plane.

[0035] For example, Figure 1 of (a) is an image taken by rotating the camera along the z-axis, i.e. tilting the camera horizontally, where the image includes horizontally and vertically tilted line segments relative to the horizontal and vertical line segments in the captured scenario. of (b) is an image obtained by tilt-shift distortion correction, in which the horizontal and vertical tilt line segments have been corrected, so that the image obtained by tilt-shift distortion correction better follows the distribution characteristics of the horizontal and vertical line segments in the real scenario captured by the imaging device. of(c) is an image taken by rotating the camera along the z-axis, i.e., an image taken by the camera from a low angle, in which the image contains vertically inclined line segments relative to the vertical line segments in the captured scenario. of (d) is an image obtained by tilt-shift distortion correction, in which the vertical inclined line segments are corrected, so that the image obtained by tilt-shift distortion correction better follows the distribution characteristics of the horizontal and vertical line segments in the real scenario captured by the imaging device.

[0036] In an embodiment of the present application, the device that performs tilt-shift distortion correction on the image is equipped with powerful image processing computation capabilities, which may be an image capture device, a server, a cloud device, an edge device (e.g., a box carrying a chip with processing capabilities), etc.

[0037] For example, Figure 2 of As shown in (a), the image capture device obtains the processing target image locally or from the cloud, and applies tilt-shift to the processing target image by using a processor (e.g., a graphics processing unit (GPU)) of the image capture device. distortion Perform the correction.

[0038] As another example, FIG. of As shown in (b), the image capture device uploads the image to be processed, and a server, cloud device, or edge device performs tilt-shift distortion correction on the image to be processed.

[0039] The image capture device may be an embedded module. The image capture device includes a camera, an inertial measurement unit (IMU), a communication module, and a processor. For example, the image capture device may be a terminal device, such as a mobile phone terminal, a tablet computer, a notebook computer, a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, a camera, etc.

[0040] The following uses an example in which the photographing device is an intelligent terminal for explanation. For example, FIG. 3 is a schematic diagram of the structure of a photographing device according to an embodiment of the present application. The photographing device 300 includes a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a power management module 340, an antenna, a wireless communication module 360, an audio module 370, a loudspeaker 370A, a sound box interface 370B, a microphone 370C, a sensor module 380, a button 390, an indicator 391, a display screen 392, a camera 393, and the like. The sensor module 380 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, an image sensor, a distance sensor, an optical proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, and an ambient light sensor. The image sensor is used to convert the light image on the photosensitive surface into an electrical signal proportional to the light image by using the photoelectric conversion function of the photoelectric element.

[0041] It may be understood that the structure shown in this embodiment does not constitute a specific limitation for the imager. In some other embodiments, the imager may include more or fewer components than shown, combine some components, split some components, or have a different component arrangement. The components shown may be implemented by hardware, software, or a combination of software and hardware.

[0042] The processor 310 may include one or more processing units. For example, the processor 310 may include an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent components or may be integrated into one or more processors. The NPU is a neural-network (NN) computing processor. The NPU quickly processes input information based on the structure of a biological neural network, for example, based on the transfer mode between neurons in the human brain, and the NPU may further perform self-learning. The NPU may implement applications such as intelligent perception of the photographing device, for example, image recognition, face recognition, voice recognition, and text understanding.

[0043] In this embodiment, the processor 310 , Processing Based on the line segment filtering policy determined based on the metadata of the processed image, Filtering, and obtain a correction target line segment from a line segment included in the processing target image. The line segment filtering policy supports a method of performing line segment filtering on the line segments included in the processing target image. The correction target line segment includes at least one line segment in the processing target image having a slope characteristic. Metadata is also called intermediate data or relay data. Metadata is data about data, and is mainly information describing data properties. Metadata is used to support functions such as storage location indication, history data, resource search, file recording, etc. In this embodiment of the present application, metadata is any label embedded in the processing target image file, and is used to describe image data property information, such as lens parameters, exposure parameters, or sensor information. Generally, a camera automatically adds metadata to an image when taking a picture. For example, metadata includes Inertial Measurement Unit (IMU) information and field of view. IMU information includes, but is not limited to, three-axis attitude angle and acceleration. The field of view represents the maximum range that a camera can observe, usually expressed in degrees: the wider the field of view, the wider the observation range.

[0044] In this embodiment of the present application, the inertial measurement unit includes a gyroscope sensor and an acceleration sensor.

[0045] The gyroscope sensor may be configured to determine the orientation of the image capture device 300 movement. In some embodiments, the angular velocity of the image capture device 300 around three axes (e.g., x, y, and z) may be determined by using the gyroscope sensor. The gyroscope sensor may be used for image stabilization during image capture. For example, when the shutter is pressed, the gyroscope sensor detects the angle at which the image capture device 300 is shaken, and based on the angle, calculates the distance that the lens module needs to compensate so that the lens can counteract the image capture device 300 shake with an opposite movement to implement image stabilization. The gyroscope sensor may also be used in navigation scenarios and motion-sensing game scenarios.

[0046] The acceleration sensor may detect the magnitude of acceleration of the image capture device 300 in various directions (typically on three axes). When the image capture device 300 is stationary, the acceleration sensor may detect the magnitude and direction of gravity. The acceleration sensor may be further configured to determine the orientation of the image capture device, and is used in applications such as switching between landscape and portrait modes and a pedometer.

[0047] The controller can be the nerve center and command center of the imaging device, and can generate motion control signals according to the instruction operation code and time sequence signal to complete the control of reading and executing instructions.

[0048] A memory may further be disposed in the processor 310 and configured to store instructions and data. In some embodiments, the memory in the processor 310 is a cache. The memory may store instructions or data that are just used or are used periodically by the processor 310. When the processor 310 needs to use the instructions or data again, the processor may directly retrieve the instructions or data from the memory. This avoids repeated accesses of the processor 310 and reduces the latency of the processor 310. Thus, the system efficiency is improved.

[0049] In some embodiments, the processor 310 may include one or more interfaces, which may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a USB interface, and / or the like.

[0050] The power management module 340 is configured to connect to a power source. The power management module 340 may be further connected to the processor 310, the internal memory 321, the display screen 392, the camera 393, the wireless communication module 360, etc. The power management module 340 receives input from the power source and provides power to the processor 310, the internal memory 321, the display screen 392, the camera 393, the wireless communication module 360, etc. In some embodiments, the power management module 340 may also be located within the processor 310.

[0051] The wireless communication function of the image capture device may be implemented by using an antenna, a wireless communication module 360, etc. The wireless communication module 360 ​​may provide wireless communication solutions including wireless local area network (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC) technology, and infrared (IR) technology, which are applied to the image capture device.

[0052] The wireless communication module 360 ​​may be one or more components incorporating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves through an antenna, performs frequency modulation and filtering processing on the electromagnetic wave signal, and transmits the processed signal to the processor 310. The wireless communication module 360 ​​may further receive a signal to be transmitted from the processor 310, performs frequency modulation and amplification on the signal to be transmitted, and converts the processed signal into an electromagnetic wave that is radiated by the antenna. In some embodiments, an antenna of the image capture device is coupled to the wireless communication module 360, so that the image capture device can communicate with networks and other devices by using wireless communication technology.

[0053] The image capture device implements the display function by using a GPU, a display screen 392, an application processor, etc. The GPU is a microprocessor for image processing and is connected to the display screen 392 and the application processor. The GPU is configured to perform mathematical and geometric calculations and render images. The processor 310 may include one or more GPUs and execute program instructions to generate or modify display information.

[0054] The display screen 392 is configured to display images, videos, and the like. In this embodiment of the present application, the display screen 392 is configured to display the to-be-processed image and the processed image. The display screen 392 includes a display panel. The display panel may be a liquid crystal display (LCD) screen, an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (OCL ... - emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), mini LED, micro LED, quantum dot light-emitting diode - Emitting Diode , QLED), etc.

[0055] The image capture device may implement image capture functionality by using an ISP, a camera 393, a video codec, a GPU, a display screen 392, an application processor, etc. The ISP is configured to process data fed back by the camera 393. In some embodiments, the ISP may be located in the camera 393.

[0056] The camera 393 is configured to capture still images or videos. An optical image of an object is generated by a lens and projected onto a photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then sends the electronic signal to the ISP to convert the electrical signal into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, such as RGB or YUV. In some embodiments, the image capture device may include 1 to N cameras 393, where N is a positive integer greater than 1. The position of the camera 393 in the image capture device is not limited in this embodiment of the present application.

[0057] Alternatively, the capture device may not include a camera, i.e., the camera 393 is not located on the capture device (e.g., a television set). The capture device may be externally connected to the camera 393 by using an interface (e.g., a USB interface 330). The external camera 393 may be secured to an edge, e.g., the top edge, of the display screen 392 of the capture device by using an external fastener.

[0058] The digital signal processor is configured to process the digital signals, and may process other digital signals in addition to the digital image signal. For example, if the imager selects a frequency, the digital signal processor is configured to perform a Fourier transform on the frequency energy, etc. The video codec is configured to compress or decompress the digital video. The imager may support one or more video codecs. Thus, the imager may play or record video in multiple encoding formats, e.g., moving picture experts group (MPEG)-1, MPEG-2, MPEG-3, and MPEG-4.

[0059] The external memory interface 320 can be configured to connect to an external memory card, such as a microSD card, to expand the storage capacity of the image capture device. The external memory card communicates with the processor 310 through the external memory interface 320 to implement data storage functions. For example, the external memory card saves files such as music, videos, images, etc. to the external memory card.

[0060] The internal memory 321 may be configured to store computer executable program code, which includes instructions. The processor 310 executes various functional applications and data processing of the imaging device by executing the instructions stored in the internal memory 321. The internal memory 321 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required by at least one function (e.g., audio playback function or image playback function), and the like. The data storage area may store data (e.g., audio data) generated during use of the imaging device. The internal memory 321 may also include a high-speed random access memory and may further include a volatile memory, such as at least one magnetic disk storage device, a flash memory device, or a universal flash storage (UFS).

[0061] The image capture device may implement audio functions by using an audio module 370, a loudspeaker 370A, a microphone 370C, a sound box interface 370B, an application processor, etc. For example, the audio functions include playing music, recording, etc. In this application, the microphone 370C may be configured to receive voice commands sent by a user to the image capture device. The loudspeaker 370A may be configured to feed back the image capture device's decision commands to the user.

[0062] The audio module 370 is configured to convert digital audio information to analog audio signals for output, and further configured to convert analog audio input to digital audio signals. The audio module 370 may further be configured to encode and decode audio signals. In some embodiments, the audio module 370 may be located in the processor 310, or some functional modules in the audio module 370 are located in the processor 310. The loudspeaker 370A, also called a "horn," is configured to convert audio electrical signals into acoustic signals. The microphone 370C, also called a "microphone" or "mic," is configured to convert acoustic signals into electrical signals.

[0063] The sound box interface 370B is configured to connect to a wired speaker box. The sound box interface 370B may be a USB interface 330, or may be a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular standard (CTA) standard (CTA). telecommunication The interface may be a CTIA (Computer Telecommunications Industry Association of the USA) standard interface.

[0064] The buttons 390 include a power button, a volume button, etc. The buttons 390 may be mechanical buttons or touch buttons. The image capture device may receive the button input and generate button signal inputs related to user settings and function control of the image capture device.

[0065] The indicator 391 may be an indicator light and may be used to indicate that the imager is in a powered on state, a standby state, a powered off state, etc. For example, when the indicator light is off, it indicates that the imager is in a powered off state. When the indicator light is green or blue, it indicates that the imager is in a powered on state. When the indicator light is red, it indicates that the imager is in a standby state.

[0066] It may be understood that the structure shown in this embodiment of the present application does not constitute a specific limitation on the imaging device. The imaging device may have more or less components than those shown in Fig. 3, may combine two or more components, or may have a different component configuration. For example, the imaging device may further include a component such as a sound box. The components shown in Fig. 3 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing circuits or application specific integrated circuits.

[0067] The methods in the following embodiments can all be implemented in a photographing device having the above hardware structure. In the following embodiments, an example in which the photographing device is a smartphone is used to describe the methods in the embodiments of the present application.

[0068] The following describes in detail the image processing method provided in the embodiment of the present application with reference to Fig. 4-16(a), Fig. 16(b), Fig. 16(c), Fig. 16(d), Fig. 16(e), Fig. 16(f), and Fig. 16(g). Fig. 4 is a schematic flowchart of the image processing method according to the embodiment of the present application. An example in which the photographing device 300 performs tilt-shift distortion correction on an image is used here for description. As shown in Fig. 4, the method includes the following steps:

[0069] Step 410: The image capture device 300 captures an image to be processed.

[0070] The photographing device 300 can perform real-time photographing by using a camera, and can use the image acquired by photographing as an image to be processed. , alternative Alternatively, the image to be processed may be acquired from a gallery stored in the memory. , alternative The source of the image to be processed is not limited in this embodiment of the present application.

[0071] The processed image captured by the image capture device 300 may be an unedited, real-time captured image, or an edited image. For example, the processed image may have been edited by a third-party image application or may have been edited using image editing algorithms carried by the image capture device 300. Use Image editing includes geometric transformation operations such as cropping, rotation, and rectification.

[0072] Optionally, the image to be processed may alternatively be a pre-processed image, where pre-processing includes black level correction, noise reduction, auto-exposure, auto-white balance, and image distortion.

[0073] The image to be processed in this embodiment of the present application may be obtained by tilting the image capture device 300 when the image capture device captures a first scenario (e.g., the first scenario is captured from a low angle or a high angle, or the first scenario is captured when the image capture device 300 is tilted horizontally). The image to be processed includes at least one of a vertical line segment and a horizontal line segment in the first scenario. For example, the image to be processed includes a line segment that is tilted in the image to be processed and is at least one of a vertical line segment and a horizontal line segment in the first scenario. A vertical line segment that is in the first scenario and tilted in the image to be processed may be referred to as a vertical tilted line segment. A horizontal line segment that is in the first scenario and tilted in the image to be processed may be referred to as a horizontal tilted line segment. A vertical tilted line segment is a tilted line segment that is within a vertical line segment tilt threshold with respect to a line segment perpendicular to the horizon. It may be It can be seen that a horizontally slanted line segment is a line segment that is slanted within a horizontal line segment slant threshold relative to a line segment that is parallel to the horizon. It may be The horizontal line segment tilt threshold and the horizontal line segment tilt threshold may both be 5°.

[0074] For example, Figure 1 of (a) is an image taken by tilting the camera horizontally, and the image includes horizontally tilted line segments and vertically tilted line segments. of (c) is an image taken by a camera from a low angle, in which the top of the building becomes gradually smaller and the image contains vertically sloping line segments. In contrast, in an image taken by a camera from a high angle, the bottom of the building becomes gradually smaller and the image contains vertically sloping line segments.

[0075] Step 420: The image capture device 300 detects line segments in the target image according to a line detection method.

[0076] The line detection method may be Hough line detection or LSD line detection. The image capture device 300 detects the target image according to the line detection method and obtains M line segments contained in the target image, where M is a positive integer. The M line segments include at least one of a horizontal line segment, a vertical line segment, a horizontal inclined line segment, and a vertical inclined line segment.

[0077] Step 430: The imaging device 300 , Processing Based on the field of view and IMU information contained in the metadata of the target image, Filtering A line segment to be corrected is obtained from the line segments included in the image to be processed.

[0078] Since the field of view and IMU information describe the characteristic information of the image capture device 300 that captures the target image, the image capture device 300 can accurately locate the position of the line segment that needs to be corrected based on the field of view and IMU information, select the line segment to be corrected from the line segments included in the target image, and correct the feature area of ​​the line segment to be corrected and the target image, so that the line segment in the processed image better follows the feature distribution of the horizontal line segment and the vertical line segment in the first scenario captured by the image capture device. For example, as shown in FIG. 5, the metadata includes original data and newly added data. The original data includes lens parameters and exposure parameters. The newly added data includes at least one of IMU information and field of view. The IMU information may be the tilt angle of the device that captures the target image. For example, the IMU information includes a horizontal tilt angle and a pitch tilt angle. For example, the target image is captured by the image capture device 300, and the IMU information may be the IMU information of the image capture device 300.

[0079] The method procedure set forth in FIG. 6 describes a specific operation process included in step 430 of FIG. 4 as shown in FIG.

[0080] Step 431: The imaging device 300 performs global adjustment on the M line segments included in the processing target image according to the IMU information and the field of view, to obtain M adjusted line segments.

[0081] The M line segments acquired by the image capture device 300 by detecting the processing target image according to the line detection method are not all vertical or horizontal line segments in the first scenario. When the M line segments are filtered based on the line segment slope threshold, erroneous filtering is likely to occur. For example, the line segments of portraits or clothes are not vertical or horizontal line segments in real scenarios. Therefore, the image capture device 300 performs a global adjustment on the M line segments included in the processing target image based on the IMU information and the field of view to obtain M adjusted line segments, so that the M adjusted line segments follow the line segment angle distribution characteristics of the first scenario. That is, the M adjusted line segments better follow the vertical line segment characteristics and horizontal line segment characteristics in the first scenario.

[0082] In some embodiments, the imager 300 performs a perspective transformation on the M line segments to obtain M adjusted line segments, ie, M perspective transformed line segments.

[0083] The image capture device 300 detects a set of line segments in the image to be processed according to a line detection method.

number

number

number

[0084] H represents the perspective transformation matrix. The perspective transformation coordinates of coordinates (x, y) are x'=U / W and y'=V / W.

[0085] Step 432: The imaging apparatus 300 obtains N line segments to be corrected from the M adjusted line segments based on the line segment slope threshold through filtering.

[0086] The M adjusted line segments are twist Line segments with large inclination angles, relative to 90° perpendicular line segments twist Line segments with small inclination angles, relative to the 0° horizontal line segment twist For line segments with large inclination angles and 0° horizontal line segments twist Includes line segments with small slope angles. twistThe line segments with large tilt angles may not be vertical or horizontal line segments in the first scenario and may not need to be corrected. Therefore, the image capture device 300 can select N line segments to be corrected from the M adjusted line segments based on the line segment tilt threshold, where the N line segments to be corrected better follow the line segment angle distribution characteristic of the first scenario, and the image capture device 300 can correct the N line segments to be corrected. This improves the accuracy of performing tilt-shift distortion correction on the processing target image. N is a positive integer, and N is less than or equal to M. If N is equal to M, it indicates that all of the M adjusted line segments are line segments to be corrected, and if N is less than M, it indicates that some line segments of the M adjusted line segments are line segments to be corrected. The line segment tilt threshold includes a horizontal line segment tilt threshold and a vertical line segment tilt threshold.

[0087] In some embodiments, the image capture device 300 calculates the angles of the line segments based on the start point coordinates and the end point coordinates of the M adjusted line segments to obtain an angle set of the M adjusted line segments.

number

[0088] For example, FIG. 7 is a schematic diagram of a visualization of line segment filtering according to an embodiment of the present application. of (a) shows all the line segments obtained by detecting the target image according to the line detection method, including the line segments of the portrait, the clothes, and the chair. of (b) is a line segment that follows the actual angle distribution in the actual scenario after perspective transformation. of (c) is a correction target line segment obtained by filtering a line segment whose angle is close to 90° or 0° after perspective transformation.

[0089] It should be understood that the line segments to be corrected include some or all of the line segments that are tilted in the image to be processed and are at least one of the vertical line segments and the horizontal line segments in the first scenario. For example, the image to be processed includes a building image and a portrait with tilt-shift distortion, and the line segments to be corrected include at least one of the vertical line segments and the horizontal line segments in the building that are tilted in the image, and the line segments to be corrected do not include the line segments in the portrait that are tilted. Only the tilted line segments in the building image are corrected, and the effect of longer legs is maintained in the portrait, creating a more beautiful image composition.

[0090] Step 440: The photographing apparatus 300 corrects the correction target line segment and the feature region of the processing target image to obtain a processed image.

[0091] The image capture device 300 corrects image data for the target image, where the image data includes pixels of the target image.

[0092] For example, the image capture device 300 may construct constraints according to regions of the target image, jointly optimize the constraints to obtain a global grid correction displacement, and perform an image affine transformation on the target image to obtain a processed image. The regions may include straight line regions, edge regions, background regions, and portrait regions.

[0093] The straight line constraints are used to construct horizontal and vertical constraints for the vertical and horizontal line segments contained in the corrected line segment obtained through filtering in the first scenario.

[0094] The content-based homography constraint is used to construct homography constraints for regions with rich edge textures in an image. Since the image capture device 300 may generate false negatives on straight lines, some straight lines may not be corrected if the horizontal and vertical constraints are not constrained for the undetected straight lines. In this case, the grid vertex set with rich texture regions is constrained, and the entire transformation is performed based on the homography mapping. The content-based homography constraint may be a constraint for background regions. The homography constraint is shown in the following equation (2).

number

[0095] λ H represents the weight, 0<λH<1.0. The coordinate vector obtained by the perspective transformation is (u,v)=(U,V) / W, where x i,j and y i,jdenotes the original coordinates, and i and j denote the grid point corresponding to the ith row and jth column.

[0096] The portrait constraint is used to construct a geometric transformation constraint of the portrait region based on the position of the portrait. For example, the grid point coordinates of the portrait region are constrained to satisfy a spherical transformation relationship, so that the perspective distortion, such as edge stretch of the portrait, is optimized based on correcting the tilt-shift distortion of the background and the portrait. For example, if the portrait is taken from a high angle, the head-to-body ratio of the portrait is unbalanced, that is, the head image is large and the leg image is short. As another example, if the portrait is taken from a low angle, the head-to-body ratio of the portrait is unbalanced, that is, the head image is small and the leg image is long. As another example, if the portrait is taken with an ultra-wide angle, the head image or the body image at the edge of the image is stretched and distorted. The portrait constraint is constrained according to the portrait distortion phenomenon, and the distortion in the portrait image is corrected based on the portrait constraint.

[0097] The edge constraint is used to constrain the outward and inward displacement of pixels corresponding to edge regions of the field of view. To reduce field loss, pixels at the top and bottom edges of the image are restricted from moving outward. The edge constraint is shown in the following equation (3).

number

[0098] H is the length of the image and W is the width of the image. i,j (u i,j <0 or u i,j >W) and v i,j (v i,j <0 or v i,j >H) represents the x and y coordinates of the pixel after displacement, and λ B_in represents the weight, 0<λ B_in <1.0.

[0099] Furthermore, to avoid stretching of the image edge texture, the pixels at the top, bottom, left, and right ends of the image are restricted from moving inward. The edge constraint is shown in the following equation (4).

Equation

[0100] u i,j (0 < u i,j < W) and v i,j (0 < v i,j < H) represent the x and y coordinates of the pixel after displacement, and λ B_out represents the weight, where 0 < λ B_out < 1.0.

[0101] The shape constraint is used to construct a shape maintenance constraint to avoid image texture distortion. The transformation of the grid points satisfies the constraints of local similarity transformation or equiangular transformation. The shape constraint is shown in the following equation (5).

Equation

[0102] λ S represents the weight, where 0 < λ S < 1.0.

[0103] The second - order differential approximation is calculated using the finite - difference method, and the smoothness constraint is given using the Frobenius norm. The smoothness constraint can refer to the shape constraint. The smoothness constraint is shown in the following equation (6).

Equation

[0104] λ R represents the weight, where 0 < λ R < 1.0.

[0105] Straight - line constraint, content shape maintenance constraintsConstraints such as (content-based homography constraint), field of view edge constraint (edge ​​constraint), portrait constraint, shape constraint, and smoothness constraint are jointly optimized, so that the corrected image is reasonable and beautiful, and the field of view is wide.

[0106] Therefore, performing precise corrections on line segments within the target image compared to performing a global correction on the target image effectively improves the accuracy of performing tilt-shift distortion correction on the image.

[0107] In some other embodiments, the image capture device 300 may further formulate a line segment filtering policy based on the constraints included in the metadata to filter the line segments to be corrected. The line segment filtering policy indicates how to perform line segment filtering on the line segments included in the processed image. Thus, different line segment filtering policies may be adapted based on the content included in the metadata to improve the flexibility and accuracy of segment filtering.

[0108] FIG. 8A and FIG. 8 8B is a schematic flow chart of an image processing method according to an embodiment of the present application. An example in which the image capture device 300 performs tilt-shift distortion correction on an image is used here for description. As shown in FIG. 8A and FIG. 8B, the method includes the following steps:

[0109] Step 810: The photographing device 300 acquires an image to be processed.

[0110] Step 820: The Imager 300 detects line segments in the target image according to a line detection method.

[0111] For a detailed description of step 810 and step 820, please refer to the above description of step 410 and step 420. The details will not be described again.

[0112] Step 830: The imaging device 300 , Me Based on line segment filtering policies determined based on the data Filtering A line segment to be corrected is obtained from the line segments included in the image to be processed.

[0113] It should be understood that the line segments to be corrected include some or all of the line segments that are tilted in the image to be processed and are at least one of the vertical line segments and the horizontal line segments in the first scenario. The first scenario is a scenario in which the image to be processed is taken. For example, the line segments to be corrected include the vertical line segments in the first scenario. As another example, the line segments to be corrected include the horizontal line segments in the first scenario. As another example, the line segments to be corrected include the vertical line segments and the horizontal line segments in the first scenario. For example, the image to be processed includes a building image and a portrait in which tilt-shift distortion occurs, and the line segments to be corrected include at least one of the vertical line segments and the horizontal line segments in the building that are tilted in the image, and the line segments to be corrected do not include the tilted line segments in the portrait. Only the tilted line segments in the building image are corrected, and the effect of longer legs is maintained in the portrait, creating a more beautiful image composition.

[0114] The image capturing apparatus 300 may determine different line segment filtering policies based on different content included in the metadata. For example, step 830 includes steps 831 to 834.

[0115] Step 831: The image capture device 300 determines whether the metadata includes IMU information and field of view.

[0116] If the metadata includes the IMU information and the field of view, the line segment filtering policy instructs to perform line segment filtering based on the IMU information and the field of view, and the image capture device 300 executes step 832. Specifically, the image capture device 300 obtains a correction target line segment from the line segments included in the processing target image based on the IMU information and the field of view through filtering. For a solution in which the image capture device 300 performs line segment filtering based on the IMU information and the field of view, please refer to the detailed description in FIG. 6.

[0117] If the metadata does not include the field of view, the line segment filtering policy dictates that line segment filtering be performed based on the image content of the target image, and the Image Capture Apparatus 300 performs step 833. Specifically, the Image Capture Apparatus 300 , Processing Based on the image content of the target image Filtering , obtain the correction target line segment from the line segments included in the processing target image, where the image content includes the line segment angle distribution feature. For the solution in which the image capture device 300 performs line segment filtering based on the image content of the processing target image, please refer to the detailed description in FIG. 9 below.

[0118] Metadata includes field of view Muga , if not, the line segment filtering policy instructs to perform line segment filtering based on the image content and field of view of the target image, and the image capture device 300 executes step 834. Specifically, the image capture device 300 obtains a correction line segment from the line segment included in the target image through filtering based on the field of view and the image content of the target image. For a solution in which the image capture device 300 performs line segment filtering based on the field of view and the image content of the target image, please refer to the detailed description in FIG. 10 below.

[0119] FIG. 9 is a schematic diagram of a line segment filtering method based on image content according to an embodiment of the present application.

[0120] The image capture device 300 may filter the horizontally inclined line segments and the vertically inclined line segments among the M line segments based on the line segment filtering model. For example, the image capture device 300 inputs the processing target image and the M line segment identifiers to the line segment filtering model to obtain a set of horizontally inclined line segments and a set of vertical line segments.

[0121] In some other embodiments, the image capture device 300 may detect vertically slanted line segments and horizontally slanted line segments among the M line segments according to a vanishing point detection method. and Includes 920.

[0122] Step 910: The imaging device 300 performs pre-filtering based on the angles of the M line segments to obtain a set of pre-filtered vertical inclined line segments.

[0123] The imaging device 300 determines whether the angles of the M line segments are within a vertical line segment slope threshold range [θ l e ,θ h e ] (e.g., [-75°, 75°]). If the line segment angle is within the vertical line segment slope threshold range, it indicates that the line segment angle is close to 90°. If the line segment angle is not within the vertical line segment slope threshold range, it indicates that the line segment angle is far from 90°, and the imager 300 may select a pre-filtered set of possible vertical slope line segments.

number

number

[0124] Step 920: The photographing device 300 obtains a set of vertical inclined line segments from the pre-filtered set of vertical inclined line segments through a vanishing point detection method by filtering.

[0125] The line segments in the vertical slanted line segment set intersect at a specific point, and the K equations in equation (2) above are combined to obtain the vanishing point (x vanish ,y vanish The pre-filtered line segments are then solved by using the singular value decomposition (SVD) method to obtain K pre-filtered line segments. The vanishing point is the intersection point of parallel lines. When two perpendicular parallel lines are viewed or photographed from a low angle, the two perpendicular parallel lines intersect at a point far away, i.e., the vanishing point, also called the vanishing point. The image capture device 300 then calculates the distances from the vanishing point of the perpendicular line segment to the K pre-filtered line segments.

number

[0126] Compared with the line segment filtering method based on image content, in the line segment filtering method based on field of view and image content, the vertical tilt threshold range corresponding to different field of view sizes can be adapted based on field of view. Under the same vertical tilt angle and the same horizontal tilt angle, the tilt angle range of the vertical line segments in the image changes with different fields of view. The wider the field of view, the larger the tilt of the vertical lines at the edge of the image. For example, as shown in FIG. 10, the first field of view represented by the dashed box is smaller than the second field of view represented by the solid box, and the line segment L1, the line segment L2, the line segment L3, and the line segment L4 are parallel to each other and perpendicular to the ground in the real scenario. However, the four vertical line segments are tilted because the four vertical line segments are photographed from a low angle, and when the four vertical line segments are photographed from the same low angle, the tilt angles of L3 and L4 are larger than the tilt angles of L1 and L2. Therefore, the vertical line segment tilt threshold range corresponding to different field of view sizes is adapted based on field of view to improve the accuracy of preliminary line segment filtering. The imager 300 may linearly adapt the vertical line segment tilt threshold range to correspond to the field of view: the wider the field of view, the wider the vertical line segment tilt threshold range.

[0127] Step 840: The image capturing apparatus 300 corrects the correction target line segment and the feature region of the processing target image to obtain a processed image.

[0128] For a detailed description of step 840, please refer to the description of step 440. The details will not be described again.

[0129] Thus, the image capture device 300 can adapt to different line segment filtering policies based on the content contained in the metadata to increase the flexibility and accuracy of the line segment filtering performed on the line segments contained in the images being processed.

[0130] The target image in this embodiment of the present application may be an image edited by a third-party image application. Therefore, the IMU information and field of view collected when the image was taken cannot accurately describe the image. In some embodiments, the metadata may further include at least one of a trust identifier and an edit identifier. The trust identifier indicates whether the metadata is trustworthy. For example, if the trust identifier has a value of 1, it indicates that the metadata is untrustworthy, and if the trust identifier has a value of 0, it indicates that the metadata is trustworthy. The trust identifier may be provided by a front-end IMU module. Optionally, if geometric transformation edit operations such as cropping / rotating / straightening are performed on the target image by a third-party image application, the IMU information and field of view are considered untrustworthy. The edit identifier indicates whether the target image is processed by an image application.

[0131] 11 , before performing step 831, the image capture device 300 may further perform step 850, i.e., determine whether the metadata is trustworthy. Determining whether the metadata is trustworthy may be replaced by determining whether the image to be processed has been processed by an image application.

[0132] If the trust identifier indicates that the metadata is not trustworthy or the image to be processed is to be processed by an image application, step 833 is performed, i.e., the image capture device 300 , Processing Based on the image content of the target image Filtering A line segment to be corrected is obtained from the line segments included in the image to be processed.

[0133] If the trusted identifier indicates that the metadata is trusted or the image to be processed is not processed by an image application, step 832 is performed, that is, the image capture device 300 obtains a correction target line segment from the line segments included in the image to be processed based on the IMU information and the field of view by filtering. Alternatively, step 834 is performed, that is, the image capture device 300 , visual Based on the image content of the image to be processed Filtering A line segment to be corrected is obtained from the line segments included in the image to be processed.

[0134] In this way, before the line segment filtering is performed based on the IMU information and the field of view, it is first predetermined whether the IMU information and the field of view are reliable, and a line segment filtering policy for the line segment filtering is determined based on the reliability of the IMU information and the field of view. Thus, the line segment filtering rate is improved, and the accuracy of correcting the distorted line segments in the image is improved.

[0135] In order to further improve the line segment filtering result based on image content, this embodiment of the present application provides a user interaction straight line recommendation policy. For example, as shown in Figures 12A and 12B, after performing step 833 or step 834, the imaging device 300 may further perform step 860, that is, prompt the user to select a line segment to be corrected from the recommended line segments obtained by filtering based on the image content of the processing target image. For example, step 860 may include the following detailed steps:

[0136] Step 861: The image capture device 300 determines whether the horizontal direction is recognized as stable.

[0137] If the horizontal orientation is not stably recognized, step 862 is executed, i.e., the interface displays a recommended horizontal line segment. If the user does not select a recommended horizontal line segment, step 863 is executed, i.e., prompts the user to manually pick the horizontal tilted line segment that needs to be corrected.

[0138] If the horizontal direction is not known to be stable, step 864 is executed, ie, determines whether the vertical direction is known to be stable, and a horizontal tilted line segment is output.

[0139] If the vertical orientation is not stably recognized, step 865 is performed, i.e., the interface displays a recommended vertical line segment. If the user does not select a recommended vertical line segment, step 866 is performed, i.e., prompts the user to manually pick the vertical tilt line segment that needs to be corrected.

[0140] If the vertical orientation is known to be stable, then vertically slanted line segments are output.

[0141] If the image capture device 300 determines that both the horizontal and vertical directions are stably recognized, step 840 is executed, that is, the image capture device 300 corrects the obtained correction target line segment by performing line segment filtering based on the image content, corrects the feature region of the processing target image, and obtains a processed image. If neither the horizontal nor the vertical direction is stably recognized, the image capture device 300 highlights and displays the recommended horizontally inclined line segment or the recommended vertically inclined line segment in the interface, identifies the line segment selected by the user, and corrects the line segment selected by the user.

[0142] Optionally, the photographing device 300 may determine whether the horizontal and vertical directions are stably recognized based on the number and distribution of the vertical and horizontal line segments obtained by performing line segment filtering based on the image content. For example, if the number of line segments obtained by filtering is very small or the distribution of the line segments is random, it can be considered that the horizontal and vertical directions are not stably recognized.

[0143] For example, Figures 13(a) and 13(b) are schematic diagrams of a line segment selection interface according to an embodiment of the present application. As shown in Figure 13(a), if the image capture device 300 determines that the horizontal or vertical direction is not stably recognized, the interface displays a recommended line segment. As shown in Figure 13(b), if the image capture device 300 determines that the horizontal or vertical direction is not stably recognized and the user does not select a recommended line segment, the interface prompts the user to manually pick the tilted line segment that needs to be corrected.

[0144] In another possible implementation, when filtering the to-be-processed image based on two different line segment filtering policies, the image capture device 300 does not need to execute the process of determining the line segment filtering policy based on metadata in FIG. 8A. In other words, the image capture device 300 does not need to execute the process of step 830. In practice, the image capture device 300 filters the to-be-processed image based on two different line segment filtering policies, obtains the union or intersection of the line segment sets filtered based on the two different line segment filtering policies, and determines the final to-be-corrected line segment. For example, the image capture device 300 determines the to-be-corrected line segment based on the line segment filtered based on the IMU information and the field of view and the line segment filtered based on the image content. The difference between this embodiment and the above embodiment is that based on filtering the line segment based on the IMU information and the field of view, the image capture device 300 further identifies the line segment that needs to be corrected based on the image content to improve the accuracy of performing the tilt-shift distortion correction. For example, line segment filtering is performed based on IMU information and field of view to obtain a line segment set LC1, line segment filtering is performed based on image content to obtain a line segment set LC2, and a final corrected line segment set LC is obtained by solving the union or intersection of LC1 and LC2. In this way, the accuracy of performing line segment filtering based on IMU information and field of view is further improved, and the accuracy of performing tilt-shift distortion correction on the image is improved.

[0145] It should be noted that the need to correct the tilted line segments in an image is related to the user's intention and the photographed object. The need for tilt correction is usually high when the photographed object is a straight line scenario with strong regularity, such as a building structure, a horizon, a poster, a window frame, a door frame, a cabinet. The need for tilt correction is usually low when the photographed object is a straight line scenario with weak regularity, such as a straight object, an animal, a food, a face, the sky. Furthermore, including a large number of tilted line segments in an image is usually the user's composition intention. For example, when the photographed object is a portrait photographed from a low angle, or when the photographed tilt angle is extremely large, including a large number of tilted line segments in an image is usually the user's composition intention.

[0146] In another possible implementation, before the image capture device 300 performs step 840, i.e., corrects the line segment to be corrected, the image capture device 300 may further analyze the user's behavior and adaptively select whether to correct the vertically inclined line segment and the horizontally inclined line segment obtained by filtering, as shown in Figures 14A and 14B for example.

[0147] Step 870: The image capture apparatus 300 determines whether the number of the line segments to be corrected is greater than a quantity threshold and whether the length of the line segments to be corrected is greater than a length threshold.

[0148] If the number of line segments to be corrected is greater than the quantity threshold and the length of the line segments to be corrected is greater than the length threshold, it indicates that the first scenario is a strongly regular straight line scenario, and step 840 is executed, i.e., correcting the line segments to be corrected to obtain a processed image.

[0149] If the number of line segments to be corrected is less than or equal to the quantity threshold, or the length of the line segments to be corrected is less than or equal to the length threshold, it indicates that the first scenario is a straight line scenario with weak regularity, no automatic correction is performed, and step 880 is executed.

[0150] Step 880: The image capture device 300 determines whether the line segment to be corrected should be corrected based on the scenario characteristics of the first scenario.

[0151] The scenario features include architectural features and character features. The image capture device 300 may determine different horizontal line segment tilt thresholds and vertical line segment tilt thresholds based on different scenario features. The image capture device 300 determines whether the tilt angle of the horizontal tilted line segment is within a horizontal line segment tilt threshold range, and corrects the horizontal tilted line segment if the tilt angle of the horizontal tilted line segment is within the horizontal line segment tilt threshold range. The image capture device 300 further determines whether the tilt angle of the vertical tilted line segment is within a vertical line segment tilt threshold range, and corrects the vertical tilted line segment if the tilt angle of the vertical tilted line segment is within the vertical line segment tilt threshold range.

[0152] For example, the images to be processed include a portrait image taken from a low angle and a building image taken from a low angle. Correcting the vertically tilted line segments in the portrait may ruin the "leg-lengthening" effect. In this case, the horizontally tilted line segments may be corrected, the vertically tilted line segments and the horizontally tilted line segments in the building may be corrected, and the vertically tilted line segments in the portrait are not corrected. It is usually the user's intention to include a large horizontal tilt angle in the subject and a large pitch tilt angle in the subject in the image. In this case, the vertically tilted line segments and the horizontally tilted line segments in the image are not corrected. For example, the image capture device 300 corrects line segments that are offset by 25° with respect to 90° and 0°.

[0153] In this way, when the tilted line segments in the image are corrected, the scenario features and user intention are fully reflected, so that the corrected image is better in accordance with the distribution features of the line segments in the actual scenario, and therefore the accuracy of performing tilt-shift distortion correction on the image is effectively improved.

[0154] In this embodiment of the present application, the filtering of tilted line segments is performed with reference to image metadata, image content analysis results, and user interaction straight line recommendation policies, and the constraints are jointly optimized to obtain the global displacement caused by the vertical tilt caused by correcting the pitch tilt angle and the global displacement caused by correcting the horizontal tilt caused by the horizontal tilt angle, and the tilt-shift distortion correction of different angles is adaptively performed with reference to the straight line detection results and user behavior analysis. Thus, the accuracy of performing tilt-shift distortion correction on images is effectively improved.

[0155] The following describes the interface operations for performing tilt-shift distortion correction on an image by using an example.

[0156] 15(a), 15(b), 15(c), 15(d), 15(e) and 15(f) are schematic diagrams of an image processing interface according to an embodiment of the present application. FIG. 15(a) is a schematic front view of a smartphone according to an embodiment of the present application. An album application (application, APP) icon 1501 is displayed on the screen of the smartphone, and a user may tap the album application icon 1501. A touch sensor included in the smartphone receives a touch operation and reports the touch operation to a processor so that the processor can launch the album application in response to the touch operation. Furthermore, in this embodiment of the present application, the smartphone may alternatively launch the album application in other ways and display a user interface of the album application on the display screen. For example, when the display screen of the smartphone is turned off, a lock screen interface is displayed, or a user interface is displayed after the smartphone is unlocked, the smartphone may launch the album application in response to a user's voice command, shortcut operation, etc., and display a user interface of the album application on the display screen. As shown in FIG. 15(b), the smartphone responds to the tap operation, and the display screen of the smartphone displays a user interface of an album application. The user interface includes photos taken at different time periods and a photo search box. The user may select a photo 1502 with tilt-shift distortion that was taken at an angle, and perform tilt-shift distortion correction on the photo 1502 with tilt-shift distortion. As shown in FIG. 15(c), the smartphone responds to the touch operation, and the display screen of the smartphone displays the photo 1502 with tilt-shift distortion and photo editing function buttons, such as a "Send" function button, an "Edit" function button, a "Favorite" function button, a "Delete" function button, and an "Other" function button. The user may tap the "Edit" function button 1503, and the display screen of the smartphone displays function options for the "Edit" function.As shown in FIG. 15(d), the smartphone responds to the tap operation, and the display screen of the smartphone displays function options for the “Edit” function. The function options for the “Edit” function include intelligence, crop, filter, and adjustment. The user can tap the “Intelligence” option button 1504, and the display screen of the smartphone displays multiple function options for the “Intelligence” option. For example, the “Intelligence” option includes auto optimize,. reflection removal, Brightening, dehaze, and architecture correction are included. As shown in FIG. 15(e), the smartphone responds to the tap operation, and the display screen of the smartphone may further display a "tilt-shift distortion" function option, a "horizontal correction" function option, a "vertical correction" function option, and the like, which are included in the "intelligence" option. The user can tap the "tilt-shift distortion" function option 1505. The smartphone performs tilt-shift distortion correction on the photo 1502 with tilt-shift distortion. As shown in FIG. 15(f), the smartphone responds to the tap operation, and the display screen displays the photo 1506 obtained after performing tilt-shift distortion correction on the photo 1502 with tilt-shift distortion. For a specific method of performing tilt-shift distortion correction on the photo 1502 with tilt-shift distortion by the smartphone, please refer to the description in the above embodiment. Details will not be described again.

[0157] In some other embodiments, the image capture device may alternatively perform tilt-shift distortion correction on the image based on a user's tilt-shift distortion operation after taking the photo. For example, FIG. 16(a), FIG. 16(b), FIG. 16(c), FIG. 16(d), FIG. 16(e), FIG. 16(f), and FIG. 16(g) are schematic diagrams of other image processing interfaces according to embodiments of the present application. FIG. 16(a) is a schematic front view of a smartphone according to an embodiment of the present application. A camera application (APP) icon 1601 is displayed on a display screen of the smartphone, and a user may tap the camera application icon 1601. A touch sensor included in the smartphone receives the touch operation and reports the touch operation to a processor so that the processor can launch the camera application in response to the touch operation. Furthermore, in this embodiment of the present application, the smartphone may alternatively launch the camera application in other ways and display a user interface of the camera application on the display screen. For example, when the smartphone's display screen is turned off, a lock screen interface is displayed, or the smartphone is unlocked and a user interface is displayed, the smartphone may launch a camera application in response to a user's voice command, a shortcut operation, or the like, and display the user interface of the camera application on the display screen. As shown in FIG. 16(b), the smartphone responds to a tap operation, and the smartphone's display screen displays the user interface of the camera application. The user interface includes mode options, such as, for example, a "short video" mode option, a "record" mode option, a "photo" mode option, a "portrait" mode option, and a "panorama" mode option. The user interface further includes function buttons, such as, for example, an "image preview" function button, a "photo" function button, and a "front / rear camera switching" function button. The user may first select the "photo capture" mode 1602, and then tap the "photo" function button 1603.The smartphone automatically activates the camera in response to the touch operation, and the camera captures an image of a scenario taken from a low angle. Furthermore, as shown in FIG. 16(c), the user may tap the "Image Preview" function button 1604 to view the captured image including the target object, i.e., the photo 1605 with tilt-shift distortion. As shown in FIG. 16(d), the smartphone responds to the tap operation, and the display screen of the smartphone displays the photo 1605 with tilt-shift distortion and photo editing function buttons, such as a "Send" function button, an "Edit" function button, a "Favorite" function button, a "Delete" function button, and an "Other" function button. The user can tap the "Edit" function button 1606, and the display screen of the smartphone displays function options of the "Edit" function. As shown in FIG. 16(e), the smartphone responds to the tap operation, and the display screen of the smartphone displays function options of the "Edit" function. The function options of the "Edit" function include intelligence, crop, filter, and adjustment. The user can tap the "Intelligence" option button 1607, and the display screen of the smartphone displays multiple function options for the "Intelligence" option. For example, the "Intelligence" option includes: Automatic optimization, The functions include reflection removal, brightening, haze removal, and architecture correction. As shown in FIG. 16(f), the smartphone responds to the tap operation, and the display screen of the smartphone may further display a "tilt-shift distortion" function option, a "horizontal correction" function option, a "vertical correction" function option, and the like, which are included in the "intelligence" option. The user can tap the "tilt-shift distortion" function option 1608. The smartphone performs tilt-shift distortion correction on the photo 1605 with tilt-shift distortion. As shown in FIG. 16(g), the smartphone responds to the tap operation, and the display screen displays the photo 1609 obtained after performing tilt-shift distortion correction on the photo 1605 with tilt-shift distortion. For a specific method of performing tilt-shift distortion correction on the photo 1605 with tilt-shift distortion by the smartphone, please refer to the description in the above embodiment. Details will not be described again.

[0158] In some other embodiments, the image capture device may automatically perform tilt-shift distortion correction on the image after the image capture device captures the photo. For example, as shown in FIG. 16(d), the smartphone responds to a tap operation, the smartphone's display screen displays a user interface of a camera application, the smartphone automatically launches the camera in response to a tap operation of a "Photo" function button 1603, the camera captures an image of a scenario taken from a low angle, and the smartphone performs tilt-shift distortion correction on the tilt-shift distorted photo 1605 taken from a low angle to obtain a tilt-shift distortion corrected photo 1609. As shown in FIG. 16(g), the smartphone's display screen displays the captured photo 1609 after performing tilt-shift distortion correction on the tilt-shift distorted photo 1605. Steps in Figure 16(c) to Figure 16(f) is omitted. This improves the user experience of using the smartphone by the user.

[0159] It should be noted that the display manner of the “tilt-shift distortion” function option, the “horizontal correction” function option, and the “vertical correction” function option is not limited in this embodiment of the present application. The function options displayed on the display screen of the photographing device may also be called controls or function controls.

[0160] To implement the functions in the above embodiments, the photographing device includes a corresponding hardware structure and / or a corresponding software module for performing each function. Those skilled in the art should easily recognize that the present application can be implemented by using hardware or a combination of hardware and computer software, in combination with the units and method steps in the examples described in the embodiments disclosed herein. Whether the functions are performed by hardware or by hardware driven by computer software depends on the specific application scenario and design constraints of the technical solution.

[0161] The above describes in detail the image processing method of this embodiment with reference to Figures 1 to 16(a), 16(b), 16(c), 16(d), 16(e), 16(f), and 16(g), and below describes the image processing device of this embodiment with reference to Figure 17.

[0162] 17 is a schematic diagram of a possible structure of an image processing device according to an embodiment. The image processing device can be configured to implement the function of the photographing device in the above method embodiment. Therefore, the image processing device can also implement the advantageous effects of the above method embodiment. In this embodiment, the image processing device can be the photographing device 300 shown in FIG. 3, or can be a module (e.g., a chip) applied to a server.

[0163] As shown in Figure 17, the image processing device 1700 includes a communication module 1710, a line segment filtering module 1720, a correction module 1730, and a storage module 1740. The image processing device 1700 is configured to implement the functions of the imager 300 in the method embodiments shown in Figures 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B.

[0164] The communication module 1710 is configured to acquire a processing target image, the processing target image including at least one of a vertical line segment and a horizontal line segment in a first scenario, the first scenario being a scenario in which the image capture device acquires the processing target image. For example, the communication module 1710 is configured to perform step 410. As another example, the communication module 1710 is configured to perform step 810.

[0165] The line segment filtering module 1720 , Processing Based on the metadata of the processed image Filtering, to obtain a correction target line segment from line segments included in the processing target image, and the metadata includes at least one of a field of view of the image capture device and inertial measurement unit (IMU) information. For example, the line segment filtering module 1720 is configured to perform steps 420 and 430. As another example, the line segment filtering module 1720 is configured to perform steps 820, 830, 850, 860, 870, and 880.

[0166] The correction module 1730 is configured to correct the line segments to be corrected and the characteristic regions of the processing image to obtain a processed image, where the characteristic regions include a background region, a portrait region, and an edge region. For example, the correction module 1730 is configured to perform step 440. As another example, the correction module 1730 is configured to perform step 840.

[0167] Optionally, the image processing device 1700 may upload the processed images to a server, edge device, or cloud device, or store the processed images in local memory.

[0168] The storage module 1740 is configured to store metadata indicating characteristic information of the target image, such as IMU information and field of view, and to aid in determining a line segment filtering policy based on the metadata of the target image.

[0169] Optionally, the image processing apparatus 1700 may further include a display module 1750, which is configured to display the to-be-processed image and the processed image.

[0170] The image processing device 1700 in this embodiment of the present application may be implemented by using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Alternatively, the image processing device 1700 and the modules of the image processing device 1700 may be software modules, in which case the image processing device may implement the image processing methods shown in Figures 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B by using software. stomach.

[0171] The image processing device 1700 according to this embodiment of the present application can accordingly execute the methods described in the embodiments of the present application. Furthermore, the above and other operations and / or functions of the units of the image processing device 1700 are used to implement the corresponding procedures of the methods of Figures 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B, respectively. For the sake of brevity, the details will not be described again here.

[0172] 18 is a schematic diagram of the structure of an image capture device 1800 according to an embodiment. As shown, the image capture device 1800 includes a processor 1810, a bus 1820, a memory 1830, a communication interface 1840, a memory unit 1850 (which may also be referred to as a main memory unit), and a camera 1860. Optionally, the image capture device 1800 may further include a display 1870, which is configured to display the image to be processed and the processed image. The processor 1810, the memory 1830, the memory unit 1850, the communication interface 1840, the camera 1860, and the display 1870 are connected via a bus 1820.

[0173] It should be understood that in this embodiment, the processor 1810 may be a CPU. Alternatively, the processor 1810 may be another general-purpose processor or a digital signal processing (DSP), ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0174] Alternatively, the processor may be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits configured to control program execution of the solutions of the present application.

[0175] The communication interface 1840 is configured to implement communications between the image capture device 1800 and peripheral devices or components. In this embodiment, when the image capture device 1800 is configured to implement the functionality of the image capture device 300 shown in Figures 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B, the communication interface 1840 is configured to receive images to be processed.

[0176] The bus 1820 may include paths used to transfer information between the above components (e.g., the processor 1810, the memory unit 1850, and the memory 1830). In addition to a data bus, the bus 1820 may further include a power bus, a control bus, a status signal bus, and the like. However, for clarity of description, the various buses are depicted in the figures as the bus 1820. The bus 1820 may be any of a variety of buses, including Peripheral Component Interconnect Express (PCIe) bus, extended industry standard architecture (EISA) bus, unified bus (unified bus, Ubus or UB), Computer Express Link (CPL), and the like. r express link (CXL), cache coherent interconnect for accelerators (CCIX), etc. The bus 1820 may be classified as an address bus, a data bus, a control bus, etc.

[0177] In an example, the imager 1800 may include multiple processors. The processor may be a multi-core (multi-CPU) processor. Here, the processor may be one or more devices, circuits, and / or computing units configured to process data (e.g., computer program instructions). In this embodiment, when the imager 1800 is configured to implement the functions of the imager 300 shown in FIGS. 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B, the processor 1810 accesses metadata stored in the memory 1830. , Processing Based on at least one of the IMU information and the field of view included in the metadata of the processed image Filtering , a line segment to be corrected contained in the image to be processed is obtained, and the line segment to be corrected and a feature region of the image to be processed are corrected to obtain a processed image, where the feature regions include a background region, a portrait region, and an edge region.

[0178] 18, it is noted that the imaging device 1800 includes only one processor 1810 and one memory 1830 as an example, where the processor 1810 and the memory 1830 are each configured to indicate a certain type of component or device. In a specific embodiment, the number of each type of component or device may be determined based on the service request.

[0179] The memory unit 1850 may correspond to a storage medium configured to store information such as metadata in the method embodiments described above. The memory unit 1850 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM) and may be used as an external cache. By way of example, but not by way of limitation, numerous forms of RAM may be used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (synchronous DRAM), double data rate synchronous dynamic random access memory (DRAM), and the like. rate There are various types of random access memory (RAM), including: SDRAM (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM).

[0180] The memory 1830 may correspond to a storage medium configured to store information such as computer instructions in the method embodiments described above, for example a magnetic disk such as a mechanical hard disk, or a solid-state disk.

[0181] The imager 1800 may be a general-purpose device or a dedicated device. For example, the imager 1800 may be an edge device (e.g., a box carrying a chip with processing capabilities), etc. Optionally, the imager 1800 may alternatively be a server or other device with computing capabilities.

[0182] It should be understood that the imaging device 1800 according to the present embodiment may correspond to the image processing device 1700 according to the present embodiment, and may correspond to a corresponding main body for performing any method described in Figures 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B. Furthermore, the above and other operations and / or functions of the modules of the image processing device 1700 are used to implement the corresponding procedures of the methods in Figures 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B, respectively. Details will not be described again here.

[0183] The method steps of the present embodiment may be implemented by hardware or may be implemented by a processor by executing a software processor. The software instructions may include corresponding software modules. The software modules may be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable hard disk, a CD-ROM, or any other form of storage medium well known in the art. For example, the storage medium is coupled to the processor, such that the processor can read information from and write information to the storage medium. Indeed, the storage medium may be a component of the processor. The processor and the storage medium may be located in an ASIC. Furthermore, the ASIC may be located in the imaging device. Indeed, the processor and the storage medium may be present in the imaging device as discrete components.

[0184] All or part of the above embodiments may be implemented by using software, hardware, firmware, or any combination thereof. When software is used to implement the embodiments, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer programs and instructions. When the computer programs or instructions are loaded and executed in a computer, all or part of the procedures or functions of the embodiments of the present application are executed. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer programs or instructions may be stored in a computer-readable storage medium, or may be transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer programs or instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device such as a server or data center incorporating one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape, an optical medium, such as a digital video disc (DVD), or a semiconductor medium, such as a solid state drive (SSD). The above description is merely a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications or replacements that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should fall within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the scope of protection of the claims.

Claims

1. 1. An image processing method, comprising: acquiring an image to be processed, the image to be processed including at least one of a vertical line segment and a horizontal line segment in a first scenario, the first scenario being a scenario in which an image capture device acquires the image to be processed; obtaining a correction target line segment from the line segments included in the processing target image based on metadata of the processing target image by filtering, the metadata including at least one of a field of view of the image capture device and an inertial measurement unit (IMU) information; correcting the line segment to be corrected and a feature region of the processing target image to obtain a processed image, the feature region including a background region, a portrait region, and an edge region; The method according to claim 1,

2. the target image has line segments that are tilted in the target image and that are at least one of the vertical line segments and the horizontal line segments in the first scenario; The method of claim 1.

3. the line segment to be corrected includes a part or an entirety of a line segment that is tilted in the image to be processed and is at least one of the vertical line segment and the horizontal line segment in the first scenario; The method of claim 2.

4. The imaging device displays tilt-shift distortion control, and before the filtering to obtain a correction target line segment from the line segments included in the processing target image based on metadata of the processing target image, the method further comprises: and receiving a user performed manipulation of the tilt-shift distortion control.

4. The method according to any one of claims 1 to 3.

5. The above-mentioned filtering is performed to obtain a correction target line segment from the line segments included in the processing target image based on metadata of the processing target image. and obtaining the correction target line segment from the line segments included in the processing target image based on a line segment filtering policy determined based on the metadata, the line segment filtering policy instructing how to perform line segment filtering on the line segments included in the processing target image.

5. The method according to any one of claims 1 to 4.

6. the metadata includes the field of view and the IMU information, and the line segment filtering policy directs performing line segment filtering based on the field of view and the IMU information; The above-mentioned filtering step of obtaining the correction target line segment from the line segments included in the processing target image based on the line segment filtering policy determined based on the metadata includes: According to the field of view and the IMU information, perform a global adjustment on M line segments included in the processing target image to obtain M adjusted line segments, wherein the M adjusted line segments satisfy a line segment angle distribution characteristic of the first scenario; and Obtaining N line segments to be corrected from the M adjusted line segments through filtering based on a line segment tilt threshold, the line segment tilt threshold including a horizontal line segment tilt threshold and a vertical line segment tilt threshold, M and N are both positive integers, and N is less than or equal to M; having The method according to claim 5.

7. The metadata does not include the field of view, and the filtering step of obtaining the correction target line segment from the line segments included in the processing target image based on a line segment filtering policy determined based on the metadata includes: and obtaining the correction target line segment from the line segments included in the processing target image based on an image content of the processing target image by filtering, the image content including a line segment angle distribution feature. The method according to claim 5.

8. The metadata includes the field of view and does not include the IMU information, and the filtering step of obtaining the correction target line segment from the line segments included in the processing target image based on a line segment filtering policy determined based on the metadata includes: and obtaining the correction target line segment from the line segments included in the processing target image based on the field of view and an image content of the processing target image by filtering, the image content including a line segment angle distribution feature. The method according to claim 5.

9. The method comprises: and prompting the user to select the line segment to be corrected from recommended line segments obtained by filtering based on the image content of the image to be processed.

9. The method according to claim 7 or 8.

10. The metadata further includes a trust identifier, the trust identifier indicating whether the metadata is trustworthy, and the filtering step of obtaining the correction target line segment from the line segments included in the processing target image based on a line segment filtering policy determined based on the metadata includes: If the trusted identifier indicates that the metadata is untrustworthy, obtaining the correction target line segment from the line segments included in the processing target image by filtering based on the image content of the processing target image, the image content including the line segment angle distribution feature; if the trusted identifier indicates that the metadata is trusted, obtaining the correction target line segment from the line segments included in the processing target image based on at least one of the image content, the IMU information, and the field of view of the processing target image by filtering; having 9. The method according to any one of claims 5 to 8.

11. The above-mentioned filtering is performed to obtain a correction target line segment from the line segments included in the processing target image based on metadata of the processing target image. determining the line segment to be corrected based on the metadata filtered line segment and the image content filtered line segment.

5. The method according to any one of claims 1 to 4.

12. The above-mentioned correction target line segment and the feature region of the processing target image are corrected to obtain a processed image, determining whether a number of the line segments to be corrected is greater than a quantity threshold and whether a length of the line segments to be corrected is greater than a length threshold; if the number of the line segments to be corrected is greater than the quantity threshold and the length of the line segments to be corrected is greater than the length threshold, correcting the line segments to be corrected to obtain the processed image; if the number of the line segments to be corrected is less than or equal to the quantity threshold, or the length of the line segments to be corrected is less than or equal to the length threshold, determining whether to correct the line segments to be corrected based on scenario features of the first scenario, the scenario features including architectural features and character features; having 12. The method according to any one of claims 1 to 11.

13. The above-mentioned correction target line segment and the feature region of the processing target image are corrected to obtain a processed image, constructing a line constraint to correct the line segment to be corrected, constructing a content-based homography constraint, a shape constraint, and a regular constraint to correct the background region, constructing a portrait constraint to correct the portrait region, and constructing an edge constraint to correct the edge region to obtain the processed image.

13. The method according to any one of claims 1 to 12.

14. An image processing device, a communication module configured to acquire a processing target image, the processing target image including at least one of a vertical line segment and a horizontal line segment in a first scenario, the first scenario being a scenario in which an image capture device acquires the processing target image; a line segment filtering module configured to obtain a correction target line segment from line segments included in the processing target image based on metadata of the processing target image by filtering, the metadata including at least one of a field of view of the image capture device and an inertial measurement unit (IMU) information; a correction module configured to correct the correction target line segment and a feature region of the processing target image to obtain a processed image, the feature region including a background region, a portrait region, and an edge region; An apparatus having

15. the target image has line segments that are tilted in the target image and that are at least one of the vertical line segments and the horizontal line segments in the first scenario; 15. The apparatus of claim 14.

16. the line segment to be corrected includes a part or an entirety of a line segment that is tilted in the image to be processed and is at least one of the vertical line segment and the horizontal line segment in the first scenario; 16. The apparatus of claim 15.

17. The apparatus further comprises a display module; the display module is configured to display a tilt-shift distortion control and receive a user-performed operation on the tilt-shift distortion control.

17. Apparatus according to any one of claims 14 to 16.

18. When the correction target line segment is obtained from the line segments included in the processing target image based on the metadata of the processing target image by filtering, the line segment filtering module: and wherein said filtering is particularly configured to obtain said corrected line segments from said line segments included in said target image based on a line segment filtering policy determined based on said metadata, said line segment filtering policy instructing how to perform line segment filtering on said line segments included in said target image.

18. Apparatus according to any one of claims 14 to 17.

19. the metadata includes the field of view and the IMU information, and the line segment filtering policy directs performing line segment filtering based on the field of view and the IMU information; When the correction target line segment is obtained from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, the line segment filtering module: According to the field of view and the IMU information, perform a global adjustment on M line segments included in the processing target image to obtain M adjusted line segments, and the M adjusted line segments satisfy the line segment angle distribution characteristic of the first scenario; Obtain N line segments to be corrected from the M adjusted line segments through filtering based on a line segment tilt threshold, the line segment tilt threshold including a horizontal line segment tilt threshold and a vertical line segment tilt threshold, M and N are both positive integers, and N is less than or equal to M; Specifically configured to 20. The apparatus of claim 18.

20. When the metadata does not include the field of view, and the correction target line segment is obtained from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, the line segment filtering module: The method is particularly configured to obtain the correction target line segment from the line segments included in the processing target image based on an image content of the processing target image by filtering, the image content including a line segment angle distribution feature; 20. The apparatus of claim 18.

21. When the metadata includes the field of view and does not include the IMU information, and the correction target line segment is obtained from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, the line segment filtering module: The method is particularly configured to obtain the correction target line segment from the line segments included in the processing target image based on the field of view and the image content of the processing target image by filtering, the image content including a line segment angle distribution feature.

20. The apparatus of claim 18.

22. The apparatus further comprises a display module; the display module is configured to prompt the user to select the line segment to be corrected from recommended line segments obtained by filtering based on the image content of the image to be processed.

22. Apparatus according to claim 20 or 21.

23. The metadata further includes a trust identifier, and the trust identifier indicates whether the metadata is trustworthy. When the line segment to be corrected is obtained from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, the line segment filtering module: if the trusted identifier indicates that the metadata is untrusted, filtering is performed to obtain the correction target line segment from the line segments included in the processing target image based on the image content of the processing target image, the image content including the line segment angle distribution feature; If the trusted identifier indicates that the metadata is trusted, filtering is performed to obtain the correction target line segment from the line segments included in the processing target image based on at least one of the image content, the IMU information, and the field of view of the processing target image. Specifically configured to 22. Apparatus according to any one of claims 18 to 21.

24. When the correction target line segment is obtained from the line segments included in the processing target image based on the metadata of the processing target image by filtering, the line segment filtering module: and determining the line segment to be corrected based on the metadata filtered line segment and the image content filtered line segment.

18. Apparatus according to any one of claims 14 to 17.

25. When correcting the correction target line segment and the feature region of the processing target image to obtain the processed image, the correction module determining whether a number of the line segments to be corrected is greater than a quantity threshold and whether a length of the line segments to be corrected is greater than a length threshold; if the number of the line segments to be corrected is greater than the quantity threshold and the length of the line segments to be corrected is greater than the length threshold, correcting the line segments to be corrected to obtain the processed image; if the number of the line segments to be corrected is less than or equal to the quantity threshold, or the length of the line segments to be corrected is less than or equal to the length threshold, determining whether to correct the line segments to be corrected based on scenario features of the first scenario, the scenario features including architectural features and character features; Specifically configured to 25. Apparatus according to any one of claims 14 to 24.

26. When correcting the correction target line segment and the feature region of the processing target image to obtain the processed image, the correction module to obtain the processed image, the method is particularly configured to: construct a straight line constraint for correcting the line segment to be corrected; construct a content-based homography constraint, a shape constraint, and a regular constraint for correcting the background region; construct a portrait constraint for correcting the portrait region; and construct an edge constraint for correcting the edge region.

26. Apparatus according to any one of claims 14 to 25.

27. An imaging device having at least one processor, a memory, a camera, and an inertial measurement unit (IMU), the camera is configured to capture an image; The IMU is configured to acquire IMU information of the imaging device; the memory is configured to store computer programs and instructions; The processor is configured to call the computer program and the instructions to cause the camera and the IMU to perform the method of any one of claims 1 to 13. Filming equipment.

28. at least one processor and a memory; the memory is configured to store computer programs and instructions; The processor is configured to call the computer program and the instructions to perform the method according to any one of claims 1 to 13. Image processing system.

Images