Image processing method, apparatus, and device

Abstract

An image processing method and apparatus and a device are provided, which relate to the field of image processing. After obtaining an image to be processed, a line segment to be corrected is obtained from the line segment included in the image to be processed by filtering based on at least one of IMU information and field of view included in the metadata. The image to be processed includes at least one of vertical line segments and horizontal line segments in a first scenario captured by a photographing device. The line segment to be corrected and the feature region of the image to be processed are corrected to obtain a processed image, and the feature region includes a background region, a portrait region and an edge region. Since the IMU information and field of view describe the characteristic information of the photographing device that captures the image to be processed, the line segment to be corrected can be accurately located based on the IMU information and field of view, and the line segment to be corrected and the feature region of the image to be processed are corrected so that the line segment in the processed image better follows the distribution characteristics of the horizontal line segment and the vertical line segment in the first scenario captured by the photographing device. This effectively improves the accuracy of correcting the tilt-shift distortion phenomenon of the image.

Description

[Technical Field]

[0001] The present application relates to the field of image processing, and in particular to image processing methods and apparatus and devices. [Background technology]

[0002] Generally, when a user uses a camera to take a photograph, tilt-shift distortion (e.g., tilting or convergence) occurs in the image on straight lines in the photographed scenario due to non-stationary reasons such as the tilt of the camera. As a result, the image composition becomes unattractive. For example, in an image of a building taken from a low angle, the boundaries of the exterior walls converge closely at the top of the building.

[0003] Currently, for images with tilt-shift distortion, a global correction is performed based on the image content. However, for non-square content with special characteristics, the above correction method may cause correction errors, so the accuracy of the correction method is low. Summary of the Invention

[0004] The present application provides an image processing method and apparatus, as well as a device, for solving the problem of correction errors that occur when a global correction based on image content is performed on an image with tilt-shift distortion, and for improving the accuracy of performing tilt-shift distortion correction on an image.

[0005] According to a first aspect, there is provided an image processing method, the method being carried out by a computing device, wherein after acquiring an image to be processed, the computing device , processing based on at least one of the field of view and inertial measurement unit (IMU) information contained in the metadata of the processed image. By filteringand obtain a correction target line segment from a line segment included in the processing target image. The processing target image includes at least one of a vertical line segment and a horizontal line segment in a first scenario of the processing target image captured by the imaging device. It can be understood that the line segment included in the processing target image may be a line segment in the first scenario that is not tilted in the image, or a line segment in the first scenario that is tilted in the image. Furthermore, the computing device can correct the correction target line segment and a feature region of the processing target image to obtain a processed image, where the feature region includes a background region, a portrait region, and an edge region.

[0006] In this way, since the IMU information and the field of view describe the characteristic information of the image capture device that captures the target image, the line segments that need to be corrected can be accurately located based on the IMU information and the field of view, the line segments to be corrected are selected from the line segments included in the target image, and the line segments to be corrected and the feature regions of the target image are corrected so that the line segments in the processed image better follow the distribution characteristics of the horizontal and vertical line segments in the first scenario captured by the image capture device. Compared with performing a global correction on the target image, performing a precise correction on the line segments in the target image effectively improves the accuracy of performing tilt-shift distortion correction on the image.

[0007] For example, IMU information includes the horizontal and pitch tilt angles of the camera that captures the image to be processed. The field of view represents the maximum range that the camera can observe. The field of view is visually expressed in degrees. The wider the field of view, the larger the observation range.

[0008] If the image capturing device captures the first scenario from a low or high angle and a vertical line segment in the first scenario is tilted in the image, the image to be processed includes a line segment that is tilted in the image to be processed and is a vertical line segment in the first scenario.

[0009] If the image capturing device captures the first scenario horizontally and the vertical and horizontal line segments in the first scenario are tilted in the image, the image to be processed includes line segments that are tilted in the image to be processed and are vertical and horizontal line segments in the first scenario.

[0010] It should be noted that the computing device that performs the image processing in this embodiment of the present application may be a photographing device (e.g., a smartphone) that has a photographing function, or may be another device that has an image processing function (e.g., a server, a cloud device, an edge device, etc.).

[0011] The line segments to be corrected include at least one of the vertical line segments and the horizontal line segments in the first scenario. It can be understood that the line segments to be corrected include some or all of the line segments that are tilted in the image to be processed and are at least one of the vertical line segments and the horizontal line segments in the first scenario. For example, the image to be processed includes a building image and a portrait with tilt-shift distortion, and the line segments to be corrected include at least one of the vertical line segments and the horizontal line segments in the building that are tilted in the image, and the line segments to be corrected do not include the tilted line segments in the portrait. Only the tilted line segments in the building image are corrected, maintaining the effect of longer legs in the portrait and creating a more beautiful image composition.

[0012] In a possible implementation, the image capture device displays tilt-shift distortion controls, and the method further comprises: after the image capture device receives a user-performed operation on the tilt-shift distortion controls, the image capture device , processing Based on the metadata of the processed image By filteringThe method further includes obtaining a correction target line segment from the line segments included in the processing target image, and correcting the correction target line segment and the characteristic region of the processing target image to obtain a processed image. The tilt-shift distortion control is performed on the image in a human-computer interactive manner, thereby reducing the complexity of the operation steps for a user to perform tilt-shift distortion correction on the image and increasing the flexibility to perform tilt-shift distortion control on the image. Therefore, the user experience of processing the image is improved.

[0013] In another possible implementation, the method further includes displaying the processed image after correcting the target line segments and the feature regions of the target image to obtain the processed image, so that a user can more intuitively see the image on which the tilt-shift distortion correction has been performed, which improves the user experience of processing the image.

[0014] In another possible implementation, the line segments to be corrected are filtered based on a line segment filtering policy formulated based on the content contained in the metadata. The line segment filtering policy indicates how to perform line segment filtering on the line segments contained in the processed image. For example, , processing Based on the metadata of the processed image By filtering It is possible to obtain the line segment to be corrected from the line segments included in the image to be processed. , Me Based on line segment filtering policies determined based on the data By filtering , and deriving the correction target line segments from the line segments contained in the processing target image. Thus, various line segment filtering policies are adapted based on the content contained in the metadata to improve the flexibility and accuracy of performing line segment filtering on the line segments contained in the processing target image.

[0015] Example 1: Metadata includes field of view and IMU information. The line segment filtering policy instructs to perform line segment filtering based on the field of view and IMU information. Me Based on line segment filtering policies determined based on the data By filtering Obtaining a correction target line segment from the line segments included in the processing target image is to perform a global adjustment on M line segments included in the processing target image based on the field of view and IMU information to obtain M adjusted line segments, and the M adjusted line segments satisfy the line segment angle distribution characteristics of the first scenario; ,La Based on in-segment slope threshold By filtering , and obtaining N line segments to be corrected from the M adjusted line segments. The range of the line segment tilt threshold includes a horizontal line segment tilt threshold and a vertical line segment tilt threshold, where M and N are both positive integers and N is less than or equal to M. In this way, since the IMU information indicates the tilt angle of the image capture device, the image capture device performs filtering on tilted line segments in the image to be processed based on the IMU information and field of view. This helps to improve the accuracy of the line segments, and further improves the accuracy of performing tilt-shift distortion correction on the image.

[0016] Example 2: Metadata does not include field of view Me Based on line segment filtering policies determined based on the data By filtering It is possible to obtain the line segment to be corrected from the line segments included in the image to be processed. , processing Based on the image content of the target image By filtering The method includes obtaining the line segments to be corrected from the line segments included in the image to be processed, where the image content includes the line segment angle distribution feature. In this way, the image capture device performs accurate line segment filtering based on the tilt angles of the line segments in the image content, thereby further improving the accuracy of performing tilt-shift distortion correction on the image.

[0017] Example 3: Metadata includes field of view Muga , does not include IMU information Me Based on line segment filtering policies determined based on the data By filtering It is possible to obtain the line segment to be corrected from the line segments included in the image to be processed. , visual Based on the field and the image content of the image to be processed By filtering The method includes obtaining a correction target line segment from line segments included in the processing target image, where the image content includes a line segment angle distribution feature. The inclination angle of the line segment varies within different field of view ranges, so the wider the field of view, the larger the inclination angle of the edge line segment in the image. The image capture device performs appropriate line segment filtering based on the field of view and the inclination angle of the line segment in the image content, thereby further improving the accuracy of performing tilt-shift distortion correction on the image.

[0018] In another possible implementation, the method further includes prompting a user to select a line segment to be corrected from the recommended line segments obtained by filtering based on the image content of the image to be processed. Sometimes, a user may capture an image from a low or high angle for compositional purposes. In this case, the user may not want to correct a tilted line segment in the image. In this way, the image capture device provides a user interaction line recommendation policy to prompt the user whether to perform line segment correction, and performs tilt-shift distortion correction on the image based on the user's request, thereby improving the accuracy of performing tilt-shift distortion correction on the image.

[0019] In another possible implementation, the metadata further includes a trust identifier, which indicates whether the metadata is trustworthy. , Me Based on line segment filtering policies determined based on the data By filtering, ... , processing Based on the image content of the target image By filtering , a correction target line segment is obtained from the line segments included in the processing target image, and the image content includes the line segment angle distribution feature, and if the trust identifier indicates that the metadata is trustworthy, , processing Based on at least one of the image content of the target image, the IMU information, and the field of view. By filtering and obtaining a correction target line segment from line segments included in the processing target image. In this way, before line segment filtering is performed based on the IMU information and the field of view, it is first determined in advance whether the IMU information and the field of view are reliable, and a line segment filtering policy for line segment filtering is determined based on the reliability of the IMU information and the field of view. Furthermore, the accuracy of the line segment filtering result is improved, and the accuracy of tilt-shift distortion correction for the image is improved.

[0020] Another possible implementation is , processing Based on the metadata of the processed image By filtering Obtaining the line segment to be corrected from the line segments included in the image to be processed includes determining the line segment to be corrected based on the line segment filtered based on the metadata and the line segment filtered based on the image content. In this manner, the line segment to be corrected is determined with reference to the results of two different line segment filtering policies. This further improves the accuracy of performing line segment filtering based on IMU information and field of view, and further improves the accuracy of performing tilt-shift distortion correction on the image.

[0021] In another possible implementation, correcting the line segments to be corrected and the feature regions of the image to be processed to obtain a processed image includes determining whether the number of line segments to be corrected is greater than a quantity threshold and whether the length of the line segments to be corrected is greater than a length threshold; if the number of line segments to be corrected is greater than the quantity threshold and the length of the line segments to be corrected is greater than the length threshold, correcting the line segments to be corrected to obtain a processed image; if the number of line segments to be corrected is less than or equal to the quantity threshold or the length of the line segments to be corrected is less than or equal to the length threshold, determining whether to correct the line segments to be corrected based on scenario features of a first scenario, where the scenario features include architectural features and character features. In this way, when the line segments in the image are corrected, the scenario features and the user's intention are fully taken into consideration, so that the corrected image better follows the distribution characteristics of the line segments in that scenario. Therefore, the accuracy of performing tilt-shift distortion correction on the image is effectively improved.

[0022] In another possible implementation, correcting the line segments to be corrected and the feature regions of the image to be processed to obtain a processed image includes: constructing a straight line constraint to correct the line segments to be corrected, constructing a content-based homography constraint, a shape constraint, and a regular constraint to correct the background region, constructing a portrait constraint to correct the portrait region, and constructing an edge constraint to correct the edge region, to obtain the processed image. In this way, constraints such as the straight line constraint, the content-based homography constraint, the shape constraint, the regular constraint, the portrait constraint, and the edge constraint are jointly optimized, so that the correction result is more reasonable and beautiful, the field of view is wider, and the composition is more beautiful.

[0023] If the line segments to be corrected cannot be obtained from the line segments contained in the target image by filtering based on the metadata of the target image, the target image is corrected based on the feature regions of the target image, which can effectively compensate for the line segment filtering error and further improve the accuracy of performing tilt-shift distortion correction on the image.

[0024] Another possible implementation is , processing Based on the metadata of the processed image By filtering Obtaining the line segments to be corrected from the line segments included in the image to be processed includes detecting the image to be processed according to a line detection method to obtain a set of line segments, where the set of line segments includes M inclined line segments, where M is a positive integer.

[0025] In another possible implementation, after correcting the line segments to obtain the processed image, the method includes updating the metadata (including the IMU information) of the processed image, for example, deleting or modifying the IMU information, so that the accuracy of performing the line segment filtering is improved by using the IMU information.

[0026] According to a second aspect, there is provided an image processing apparatus, the apparatus comprising a module configured to perform the image processing method of the first aspect or any one of the possible designs of the first aspect.

[0027] According to a third aspect, there is provided an image capture device, the image capture device including at least one processor and a memory, the memory being configured to store a group of computer instructions, when the processor functions as the image capture device of the first aspect or any one of the possible implementations of the first aspect and executes the group of computer instructions, the processor performs the operation steps of the image processing method of the first aspect or any one of the possible implementations of the first aspect.

[0028] According to a fourth aspect, there is provided a computing device. The computing device includes at least one processor and a memory. The memory is configured to store a group of computer instructions. When the processor functions as an image capture device in the first aspect or any one of the possible implementations of the first aspect and executes the group of computer instructions, the processor performs the operation steps of the image processing method in the first aspect or any one of the possible implementations of the first aspect.

[0029] According to a fifth aspect, there is provided a computer-readable storage medium, comprising computer software instructions that, when executed on a computing device, enable the computing device to perform the operational steps of the method of the first aspect or any one of the possible implementations of the first aspect.

[0030] According to a sixth aspect, there is provided a computer program product which, when executed on a computer, enables the computing device to perform the operational steps of the method of the first aspect or any one of the possible implementations of the first aspect.

[0031] In this application, based on the implementations provided in the above aspects, the implementations may be further combined to yield further implementations. [Brief explanation of the drawings]

[0032] [Figure 1] 1A and 1B are schematic diagrams of a tilt-shift distorted image and a corrected image according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram of a tilt-shift distortion correction scenario according to an embodiment of the present application. [Figure 3] 1 is a schematic diagram of a structure of an imaging device according to an embodiment of the present application; [Figure 4] 1 is a schematic diagram of an image processing method according to an embodiment of the present application; [Figure 5] FIG. 2 is a schematic diagram of a metadata structure according to an embodiment of the present application; [Figure 6] FIG. 1 is a schematic diagram of performing line segment filtering based on IMU information and field of view according to an embodiment of the present application; [Figure 7] FIG. 1 is a schematic diagram of a visualization of line segment filtering according to an embodiment of the present application; [Figure 8A] FIG. 10 is a schematic diagram of another image processing method according to an embodiment of the present application. [Figure 8B] FIG. 10 is a schematic diagram of another image processing method according to an embodiment of the present application. [Figure 9] FIG. 2 is a schematic diagram of performing line segment filtering based on image content according to an embodiment of the present application; [Figure 10] FIG. 1 is a schematic diagram of performing line segment filtering based on field of view and image content, according to an embodiment of the present application; [Figure 11] FIG. 10 is a schematic diagram of another image processing method according to an embodiment of the present application. [Figure 12A] FIG. 10 is a schematic diagram of yet another image processing method according to an embodiment of the present application. [Figure 12B] FIG. 10 is a schematic diagram of yet another image processing method according to an embodiment of the present application. [Figure 13(a)] FIG. 2 is a schematic diagram of a line segment selection interface according to an embodiment of the present application; [Figure 13(b)] FIG. 2 is a schematic diagram of a line segment selection interface according to an embodiment of the present application; [Figure 14A] FIG. 10 is a schematic diagram of yet another image processing method according to an embodiment of the present application. [Figure 14B] FIG. 10 is a schematic diagram of yet another image processing method according to an embodiment of the present application. [Figure 15(a)] FIG. 1 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 15(b)] FIG. 1 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 15(c)] FIG. 1 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 15(d)]FIG. 1 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 15(e)] FIG. 1 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 15(f)] FIG. 1 is a schematic diagram of an image processing interface according to an embodiment of the present application. [Figure 16(a)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(b)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(c)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(d)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(e)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(f)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 16(g)] FIG. 2 is a schematic diagram of another image processing interface according to an embodiment of the present application; [Figure 17] 1 is a schematic diagram illustrating the structure of an image processing device according to an embodiment of the present application. [Figure 18] 1 is a schematic diagram of a structure of an imaging device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0033] An embodiment of the present application provides an image processing method, and in particular, a method for performing tilt-shift distortion correction on an image. Specifically, a line segment to be corrected is obtained from a line segment included in the image to be processed by filtering based on metadata of the image to be processed. The line segment to be corrected may be a line segment that is tilted in the image to be processed and is at least one of a vertical line segment and a horizontal line segment in a real scenario. The line segment to be corrected and a feature region of the image to be processed are corrected so that the line segment in the processed image better follows the distortion characteristics of the horizontal line segment and the vertical line segment in the real scenario captured by the imaging device. Compared with performing a global correction on the image to be processed, performing a precise correction on the tilted line segment in the image to be processed effectively improves the accuracy of performing tilt-shift distortion correction on the image.

[0034] Tilt-shift distortion refers to the phenomenon whereby originally parallel lines in the target plane of a photographed scenario are tilted or distorted in the imaging plane because the imaging plane of the imaging device is rotated about at least one of the x-axis, y-axis, and z-axis relative to the target plane.

[0035] For example, Figure 1 of (a) is an image captured by rotating the camera along the z-axis, i.e., tilting the camera horizontally. The image contains horizontally and vertically tilted line segments relative to the horizontal and vertical line segments in the captured scenario. of (b) is an image obtained by tilt-shift distortion correction. In this image, the horizontal and vertical tilt line segments have been corrected, so that the image obtained by tilt-shift distortion correction better follows the distribution characteristics of the horizontal and vertical line segments in the real scenario captured by the imaging device. of(c) is an image taken by rotating the camera along the z-axis, i.e., an image taken by the camera from a low angle, and the image contains vertically tilted line segments relative to the vertical line segments in the captured scenario. of (d) is an image obtained by tilt-shift distortion correction, in which the vertically inclined line segments are corrected, so that the image obtained by tilt-shift distortion correction better follows the distribution characteristics of horizontal and vertical line segments in the real scenario captured by the imaging device.

[0036] In an embodiment of the present application, the device that performs tilt-shift distortion correction on the image is equipped with powerful image processing computing capabilities, and the device may be an image capture device, a server, a cloud device, an edge device (e.g., a box carrying a chip with processing capabilities), etc.

[0037] For example, Figure 2 of As shown in (a), the image capture device acquires the image to be processed locally or from the cloud, and applies tilt-shift to the image to be processed by using a processor (e.g., a graphics processing unit (GPU)) of the image capture device. distortion Perform the correction.

[0038] As another example, see Figure 2 of As shown in (b), the image capture device uploads the image to be processed, and the server, cloud device, or edge device performs tilt-shift distortion correction on the image to be processed.

[0039] The image capture device may be an embedded module, including a camera, an inertial measurement unit (IMU), a communication module, and a processor. For example, the image capture device may be a terminal device, such as a mobile phone terminal, a tablet computer, a notebook computer, a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, a camera, or the like.

[0040] For the purposes of explanation, the following uses an example in which the photographing device is an intelligent terminal. For example, FIG. 3 is a schematic diagram of the structure of a photographing device according to an embodiment of the present application. The photographing device 300 includes a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a power management module 340, an antenna, a wireless communication module 360, an audio module 370, a loudspeaker 370A, a sound box interface 370B, a microphone 370C, a sensor module 380, a button 390, an indicator 391, a display screen 392, a camera 393, etc. The sensor module 380 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, an image sensor, a distance sensor, an optical proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, and an ambient light sensor. The image sensor converts a light image on a photosensitive surface into an electrical signal proportional to the light image by using the photoelectric conversion function of a photoelectric element.

[0041] It can be understood that the structure shown in this embodiment does not constitute a specific limitation on the image capture device. In some other embodiments, the image capture device may include more or fewer components than shown, combine some components, separate some components, or have a different component arrangement. The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0042] The processor 310 may include one or more processing units. For example, the processor 310 may include an application processor (AP), a modem processor, a GPU, an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent components or may be integrated into one or more processors. The NPU is a neural-network (NN) computing processor. The NPU rapidly processes input information based on the structure of biological neural networks, e.g., the transfer mode between neurons in the human brain. The NPU may also perform continuous self-learning. The NPU can implement applications such as intelligent perception of the imaging device, such as image recognition, face recognition, speech recognition, and text understanding.

[0043] In this embodiment, processor 310 , processing Based on the line segment filtering policy determined based on the metadata of the processed image, By filteringThe line segment filtering policy is configured to obtain a correction target line segment from line segments included in the processing target image. The line segment filtering policy supports a method for performing line segment filtering on line segments included in the processing target image. The correction target line segment includes at least one line segment in the processing target image that has a slope characteristic. Metadata is also called intermediate data or relay data. Metadata is data about data and is mainly information describing data properties. Metadata is used to support functions such as storage location indication, history data, resource search, and file recording. In this embodiment of the present application, metadata is any label embedded in the processing target image file and is used to describe image data characteristic information, such as lens parameters, exposure parameters, or sensor information. Generally, a camera automatically adds metadata to an image when taking a picture. For example, the metadata includes inertial measurement unit (IMU) information and field of view. IMU information includes, but is not limited to, three-axis attitude angles and acceleration. The field of view represents the maximum range that a camera can observe, usually expressed in degrees. The wider the field of view, the wider the observation range.

[0044] In this embodiment of the present application, the inertial measurement unit includes a gyroscope sensor and an acceleration sensor.

[0045] The gyroscope sensor may be configured to determine the orientation of the image capture device 300 during motion. In some embodiments, the angular velocity of the image capture device 300 around three axes (e.g., x, y, and z) may be determined using the gyroscope sensor. The gyroscope sensor may be used for image stabilization during image capture. For example, when the shutter is pressed, the gyroscope sensor detects the angle at which the image capture device 300 is shaken, and based on the angle, calculates the distance the lens module needs to compensate so that the lens can offset the shake of the image capture device 300 with an opposite motion to implement image stabilization. The gyroscope sensor may also be used in navigation scenarios and motion-sensing gaming scenarios.

[0046] The acceleration sensor may detect the magnitude of acceleration of the image capture device 300 in various directions (typically on three axes). When the image capture device 300 is stationary, the acceleration sensor may detect the magnitude and direction of gravity. The acceleration sensor may also be configured to determine the orientation of the image capture device, for use in applications such as switching between landscape and portrait modes and as a pedometer.

[0047] The controller can be the nerve center and command center of the imaging device, and can generate operation control signals according to the instruction operation code and time sequence signal to complete the control of instruction reading and instruction execution.

[0048] A memory may also be disposed in the processor 310 and configured to store instructions and data. In some embodiments, the memory in the processor 310 is a cache. The memory may store instructions or data that are currently being used or are periodically used by the processor 310. When the processor 310 needs to use the instructions or data again, the processor may retrieve the instructions or data directly from the memory. This avoids repeated accesses by the processor 310 and reduces the latency of the processor 310. Therefore, system efficiency is improved.

[0049] In some embodiments, the processor 310 may include one or more interfaces, which may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a USB interface, and / or the like.

[0050] The power management module 340 is configured to connect to a power source. The power management module 340 may be further connected to the processor 310, the internal memory 321, the display screen 392, the camera 393, the wireless communication module 360, etc. The power management module 340 receives input from the power source and provides power to the processor 310, the internal memory 321, the display screen 392, the camera 393, the wireless communication module 360, etc. In some embodiments, the power management module 340 may also be located within the processor 310.

[0051] The wireless communication function of the image capture device may be implemented by using an antenna, a wireless communication module 360, etc. The wireless communication module 360 ​​may provide wireless communication solutions, including wireless local area networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC) technology, and infrared (IR) technology, that are applied to the image capture device.

[0052] The wireless communication module 360 ​​may be one or more components incorporating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves through an antenna, performs frequency modulation and filtering on the electromagnetic wave signals, and transmits the processed signals to the processor 310. The wireless communication module 360 ​​may also receive signals to be transmitted from the processor 310, perform frequency modulation and amplification on the signals to be transmitted, and convert the processed signals into electromagnetic waves that are radiated by the antenna. In some embodiments, an antenna of the image capture device is coupled to the wireless communication module 360, thereby allowing the image capture device to communicate with networks and other devices using wireless communication technology.

[0053] The image capture device implements display functionality by using a GPU, a display screen 392, an application processor, etc. The GPU is a microprocessor for image processing and is connected to the display screen 392 and the application processor. The GPU is configured to perform mathematical and geometric calculations and render images. The processor 310 may include one or more GPUs and execute program instructions to generate or modify display information.

[0054] The display screen 392 is configured to display images, videos, etc. In this embodiment of the present application, the display screen 392 is configured to display the image to be processed and the processed image. The display screen 392 includes a display panel. The display panel may be a liquid crystal display (LCD) screen, an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (ALD), or a combination of these. - emitting diode (AMOLED), flexible light-emitting diode (FLED), mini LED, micro LED, quantum dot light-emitting diode (QD) - emitting diode , QLED), etc.

[0055] The image capture device may implement the image capture function by using an ISP, a camera 393, a video codec, a GPU, a display screen 392, an application processor, etc. The ISP is configured to process data fed back by the camera 393. In some embodiments, the ISP may be located in the camera 393.

[0056] The camera 393 is configured to capture still images or video. An optical image of an object is generated by a lens and projected onto a photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which then sends the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, such as RGB or YUV. In some embodiments, the image capture device may include 1 to N cameras 393, where N is a positive integer greater than 1. The location of the cameras 393 in the image capture device is not limited to this embodiment of the present application.

[0057] Alternatively, the image capture device may not include a camera, i.e., the camera 393 is not located on the image capture device (e.g., a television set). The image capture device may be externally connected to the camera 393 by using an interface (e.g., a USB interface 330). The external camera 393 may be secured to an edge, e.g., the top edge, of the image capture device's display screen 392 by using an external fastener.

[0058] The digital signal processor is configured to process digital signals, and may process other digital signals in addition to digital image signals. For example, if the image capture device selects frequencies, the digital signal processor may be configured to perform a Fourier transform on the frequency energy, etc. The video codec is configured to compress or decompress digital video. The image capture device may support one or more video codecs. In this way, the image capture device may play or record video in multiple encoding formats, such as Moving Picture Experts Group (MPEG)-1, MPEG-2, MPEG-3, and MPEG-4.

[0059] The external memory interface 320 may be configured to connect to an external memory card, such as a microSD card, to expand the storage capacity of the image capture device. The external memory card communicates with the processor 310 through the external memory interface 320 to implement data storage functions. For example, the external memory card may save files such as music, videos, and images to the external memory card.

[0060] The internal memory 321 may be configured to store computer-executable program code, which includes instructions. The processor 310 executes the instructions stored in the internal memory 321 to perform various functional applications and data processing of the imaging device. The internal memory 321 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required by at least one function (e.g., audio playback function or image playback function), etc. The data storage area may store data generated during use of the imaging device (e.g., audio data). The internal memory 321 may also include high-speed random access memory and may further include volatile memory, such as at least one magnetic disk storage device, flash memory device, or universal flash storage (UFS).

[0061] The image capture device may implement audio functions by using an audio module 370, a loudspeaker 370A, a microphone 370C, a sound box interface 370B, an application processor, etc. For example, audio functions include playing music, recording, etc. In this application, the microphone 370C may be configured to receive voice commands sent by a user to the image capture device. The loudspeaker 370A may be configured to provide feedback of image capture device decisions to the user.

[0062] The audio module 370 is configured to convert digital audio information into analog audio signals for output, and further configured to convert analog audio input into digital audio signals. The audio module 370 may be further configured to encode and decode audio signals. In some embodiments, the audio module 370 may be located in the processor 310, or alternatively, some functional modules within the audio module 370 are located in the processor 310. The loudspeaker 370A, also known as a "horn," is configured to convert audio electrical signals into acoustic signals. The microphone 370C, also known as a "microphone" or "mic," is configured to convert acoustic signals into electrical signals.

[0063] The sound box interface 370B is configured to connect to a wired speaker box. The sound box interface 370B may be a USB interface 330, or may be a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular phone interface. telecommunication It may be a CTIA (Communication Technology Industry Association of the USA) standard interface.

[0064] The buttons 390 include a power button, a volume button, etc. The buttons 390 may be mechanical buttons or touch buttons. The image capture device may receive the button input and generate button signal inputs related to user settings and function control of the image capture device.

[0065] The indicator 391 may be an indicator light and can be used to indicate that the image capture device is in a power-on state, a standby state, a power-off state, etc. For example, when the indicator light is off, it indicates that the image capture device is in a power-off state. When the indicator light is green or blue, it indicates that the image capture device is in a power-on state. When the indicator light is red, it indicates that the image capture device is in a standby state.

[0066] It can be understood that the structure shown in this embodiment of the present application does not constitute a specific limitation on the imaging device. The imaging device may have more or fewer components than those shown in FIG. 3, may combine two or more components, or may have a different component configuration. For example, the imaging device may further include a component such as a sound box. The components shown in FIG. 3 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing circuits or application-specific integrated circuits.

[0067] The methods in the following embodiments can all be implemented by a photographing device having the above hardware structure. In the following embodiments, an example in which the photographing device is a smartphone is used to describe the methods in the embodiments of the present application.

[0068] The image processing method provided in the embodiment of the present application will be described in detail below with reference to Figures 4 to 16(a), 16(b), 16(c), 16(d), 16(e), 16(f), and 16(g). Figure 4 is a schematic flowchart of the image processing method according to the embodiment of the present application. An example in which the image capture device 300 performs tilt-shift distortion correction on an image is used here for explanation. As shown in Figure 4, the method includes the following steps:

[0069] Step 410: The photographing device 300 acquires an image to be processed.

[0070] The photographing device 300 can perform real-time photographing by using a camera, and can use the image acquired by photographing as the image to be processed. , alternative Alternatively, the image to be processed may be acquired from a gallery stored in the memory. , alternative The source of the image to be processed is not limited in this embodiment of the present application.

[0071] The processed image captured by the image capture device 300 may be an unedited, real-time captured image, or an edited image. For example, the processed image may have been edited by a third-party image application or by an image editing algorithm carried by the image capture device 300. Use Image editing includes geometric transformation operations such as cropping, rotation, and rectification.

[0072] Optionally, the image to be processed may alternatively be a pre-processed image, where pre-processing includes black level correction, noise reduction, auto-exposure, auto-white balance, and image distortion.

[0073] The image to be processed in this embodiment of the present application may be acquired by tilting the image capture device 300 when the image capture device captures a first scenario (e.g., the first scenario is captured from a low angle or a high angle, or the first scenario is captured when the image capture device 300 is tilted horizontally). The image to be processed includes at least one of a vertical line segment and a horizontal line segment in the first scenario. For example, the image to be processed includes a line segment that is tilted in the image to be processed and is at least one of a vertical line segment and a horizontal line segment in the first scenario. A vertical line segment that is in the first scenario and tilted in the image to be processed may be referred to as a vertical tilt line segment. A horizontal line segment that is in the first scenario and tilted in the image to be processed may be referred to as a horizontal tilt line segment. A vertical tilt line segment is a tilted line segment that is within a vertical line segment tilt threshold relative to a line segment that is perpendicular to the horizon. It may be It can be seen that a horizontally tilted line segment is a line segment that is tilted within a horizontal line segment tilt threshold relative to a line segment that is parallel to the horizon. It may be The horizontal line segment tilt threshold and the horizontal line segment tilt threshold may both be 5°.

[0074] For example, Figure 1 of (a) is an image taken by tilting the camera horizontally, and the image includes horizontally tilted line segments and vertically tilted line segments. of (c) is an image taken by a camera from a low angle. In this image, the top of the building gradually becomes smaller and the image contains vertically sloping line segments. In contrast, in an image taken by a camera from a high angle, the bottom of the building gradually becomes smaller and the image contains vertically sloping line segments.

[0075] Step 420: The image capture device 300 detects line segments in the image to be processed according to a line detection method.

[0076] The line detection method may be Hough line detection or LSD line detection. The image capture device 300 detects the processing target image according to the line detection method and obtains M line segments contained in the processing target image, where M is a positive integer. The M line segments include at least one of horizontal line segments, vertical line segments, horizontally inclined line segments, and vertically inclined line segments.

[0077] Step 430: The imaging device 300 , processing Based on the field of view and IMU information contained in the metadata of the target image By filtering A line segment to be corrected is obtained from the line segments included in the image to be processed.

[0078] The field of view and IMU information describe the characteristics of the image capture device 300 capturing the target image. Based on the field of view and IMU information, the image capture device 300 can accurately locate the position of the line segment that needs to be corrected, select the line segment to be corrected from the line segments included in the target image, and correct the characteristic regions of the line segment to be corrected and the target image. Therefore, the line segments in the processed image better conform to the characteristic distribution of the horizontal and vertical line segments in the first scenario captured by the image capture device. For example, as shown in FIG. 5 , the metadata includes original data and newly added data. The original data includes lens parameters and exposure parameters. The newly added data includes at least one of IMU information and field of view. The IMU information may be the tilt angle of the device capturing the target image. For example, the IMU information may include the horizontal tilt angle and the pitch tilt angle. For example, the target image is captured by the image capture device 300, and the IMU information may be the IMU information of the image capture device 300.

[0079] The method procedure described in FIG. 6 describes the specific operation process included in step 430 of FIG. 4 as shown in FIG.

[0080] Step 431: The image capturing device 300 performs global adjustment on the M line segments included in the image to be processed according to the IMU information and the field of view, to obtain M adjusted line segments.

[0081] The M line segments acquired by the image capture device 300 by detecting the image to be processed according to the line detection method are not all vertical or horizontal line segments in the first scenario. When the M line segments are filtered based on the line segment slope threshold, incorrect filtering is likely to occur. For example, line segments of portraits or clothing are not vertical or horizontal line segments in real scenarios. Therefore, the image capture device 300 performs a global adjustment on the M line segments included in the image to be processed based on the IMU information and field of view to obtain M adjusted line segments, so that the M adjusted line segments follow the line segment angle distribution characteristics of the first scenario. That is, the M adjusted line segments better follow the vertical and horizontal line segment characteristics in the first scenario.

[0082] In some embodiments, the imager 300 performs a perspective transformation on the M line segments to obtain M adjusted line segments, i.e., M perspective transformed line segments.

[0083] The image capture device 300 detects a set of line segments in the image to be processed according to a line detection method.

number

number

number

[0084] H represents the perspective transformation matrix. The perspective transformation coordinates of coordinates (x, y) are x'=U / W and y'=V / W.

[0085] Step 432: The imaging device 300 obtains N line segments to be corrected from the M adjusted line segments by filtering based on the line segment slope threshold.

[0086] The M adjusted line segments are twist Line segments with large inclination angles, 90° relative to vertical line segments twist Line segments with small inclination angles, 0° relative to horizontal line segments twist For line segments with large inclination angles and 0° horizontal line segments twist Contains line segments with small slope angles. twistLine segments with large tilt angles may not be vertical or horizontal line segments in the first scenario and may not need to be corrected. Therefore, the image capture device 300 can select N line segments to be corrected from the M adjusted line segments based on the line segment tilt threshold, where the N line segments to be corrected better conform to the line segment angle distribution characteristics of the first scenario, and the image capture device 300 can correct the N line segments to be corrected. This improves the accuracy of performing tilt-shift distortion correction on the image to be processed. N is a positive integer and is less than or equal to M. If N is equal to M, it indicates that all of the M adjusted line segments are line segments to be corrected. If N is less than M, it indicates that some of the M adjusted line segments are line segments to be corrected. The line segment tilt threshold includes a horizontal line segment tilt threshold and a vertical line segment tilt threshold.

[0087] In some embodiments, the image capture device 300 calculates the angles of the line segments based on the coordinates of the start points and the coordinates of the end points of the M adjusted line segments to generate an angle set of the M adjusted line segments.

number

[0088] For example, Figure 7 is a schematic diagram of a visualization of line segment filtering according to an embodiment of the present application. of (a) shows all the line segments obtained by detecting the target image according to the line detection method, including the line segments of the portrait, the clothes, and the chair. of (b) Line segments that follow the actual angle distribution in the actual scenario after perspective transformation. of (c) is a line segment to be corrected obtained by filtering a line segment whose angle is close to 90° or 0° after perspective transformation.

[0089] It should be understood that the line segments to be corrected include some or all of the line segments that are tilted in the image to be processed and are at least one of vertical and horizontal line segments in the first scenario. For example, the image to be processed includes a building image and a portrait with tilt-shift distortion, and the line segments to be corrected include at least one of vertical and horizontal line segments in the building that are tilted in the image, but do not include the tilted line segments in the portrait. Only the tilted line segments in the building image are corrected, maintaining the effect of longer legs in the portrait and creating a more beautiful image composition.

[0090] Step 440: The photographing device 300 corrects the line segment to be corrected and the feature region of the image to be processed, and obtains a processed image.

[0091] The image capture device 300 corrects image data of the target image, where the image data includes pixels of the target image.

[0092] For example, the image capture device 300 may construct constraints according to regions of the target image, jointly optimize the constraints to obtain a global grid correction displacement, and perform an image affine transformation on the target image to obtain a processed image, which may include a line region, an edge region, a background region, and a portrait region.

[0093] The straight line constraints are used to construct horizontal and vertical constraints for the vertical and horizontal line segments included in the line segments to be corrected that are in the first scenario and obtained through filtering.

[0094] The content-based homography constraint is used to construct a homography constraint for regions with rich edge texture in an image. Since the image capture device 300 may generate false positives on straight lines, if the horizontal and vertical constraints are not applied to the undetected straight lines, some straight lines may not be corrected. In this case, the grid vertex set with the rich texture region is constrained, and the entire transformation is performed based on the homography mapping. The content-based homography constraint may be a constraint for the background region. The homography constraint is shown in the following equation (2):

number

[0095] λ H represents the weight, and 0<λH<1.0. The coordinate vector obtained by perspective transformation is (u,v)=(U,V) / W, where x i,j and y i,jdenotes the original coordinates, and i and j denote the grid point corresponding to the ith row and jth column.

[0096] The portrait constraint is used to construct a geometric transformation constraint for the portrait region based on the position of the portrait. For example, the grid point coordinates of the portrait region are constrained to satisfy a spherical transformation relationship, so that perspective distortions such as edge stretching of the portrait are optimized based on correcting tilt-shift distortions of the background and the portrait. For example, if a portrait is taken from a high angle, the head-to-body ratio of the portrait is unbalanced, that is, the head image is large and the leg image is short. As another example, if a portrait is taken from a low angle, the head-to-body ratio of the portrait is unbalanced, that is, the head image is small and the leg image is long. As another example, if a portrait is taken with an ultra-wide angle, the head image or the body image at the edge of the image is stretched and distorted. The portrait constraint is constrained according to the portrait distortion phenomenon, and the distortion in the portrait image is corrected based on the portrait constraint.

[0097] Edge constraints are used to constrain the outward and inward displacement of pixels corresponding to edge regions of the field of view. To reduce field loss, pixels at the top and bottom edges of the image are restricted from moving outward. The edge constraint is shown in Equation (3) below.

number

[0098] H is the length of the image and W is the width of the image. i,j (u i,j <0 or u i,j >W) and v i,j (v i,j <0 or v i,j >H) represent the x and y coordinates of the pixel after displacement, and λ B_in represents the weight, and 0<λ B_in <1.0.

[0099] Furthermore, to avoid stretching of the image edge texture, the pixels at the top, bottom, left, and right ends of the image are restricted from moving inward. The edge constraint is shown in the following formula (4).

Equation

[0100] u i,j (0 < u i,j < W) and v i,j (0 < v i,j < H) represent the x and y coordinates of the pixel after displacement, and λ B_out represents the weight, and 0 < λ B_out < 1.0.

[0101] The shape constraint is used to construct a shape maintenance constraint to avoid image texture distortion. The transformation of the grid points satisfies the constraints of local similarity transformation or equiangular transformation. The shape constraint is shown in the following formula (5).

Equation

[0102] λ S represents the weight, and 0 < λ S < 1.0.)

[0103] The second - order differential approximation value is calculated using the finite - difference method, and the smoothness constraint is given using the Frobenius norm. The smoothness constraint can refer to the shape constraint. The smoothness constraint is shown in the following formula (6).

Equation

[0104] λ R represents the weight, and 0 < λ R < 1.0.

[0105] Straight - line constraint, content shape maintenance constraintsConstraints such as (content-based homography constraint), field-of-view edge constraint (edge ​​constraint), portrait constraint, shape constraint, and smoothness constraint are jointly optimized, so that the corrected image is reasonable and beautiful, and the field of view is wide.

[0106] Therefore, performing precise corrections on line segments within the target image compared to performing a global correction on the target image effectively improves the accuracy of performing tilt-shift distortion correction on the image.

[0107] In some other embodiments, the image capture device 300 may further formulate a line segment filtering policy based on constraints contained in the metadata to filter the line segments to be corrected. The line segment filtering policy instructs how to perform line segment filtering on the line segments contained in the image to be processed. Thus, various line segment filtering policies may be adapted based on the content contained in the metadata to improve the flexibility and accuracy of segment filtering.

[0108] Fig. 8A and Fig. 8 8A and 8B are schematic flowcharts of an image processing method according to an embodiment of the present application. An example in which the image capture device 300 performs tilt-shift distortion correction on an image is used here for explanation. As shown in FIG. 8A and FIG. 8B, the method includes the following steps:

[0109] Step 810: The photographing device 300 acquires an image to be processed.

[0110] Step 820: The image capture device 300 detects line segments in the image to be processed according to a line detection method.

[0111] For a detailed description of step 810 and step 820, please refer to the above description of step 410 and step 420. The details will not be described again.

[0112] Step 830: The imaging device 300 , Me Based on line segment filtering policies determined based on the data By filtering A line segment to be corrected is obtained from the line segments included in the image to be processed.

[0113] It should be understood that the line segments to be corrected include some or all of the line segments that are tilted in the image to be processed and are at least one of vertical and horizontal line segments in a first scenario. The first scenario is a scenario in which the image to be processed is captured. For example, the line segments to be corrected include vertical line segments in the first scenario. As another example, the line segments to be corrected include horizontal line segments in the first scenario. As another example, the line segments to be corrected include vertical and horizontal line segments in the first scenario. For example, the image to be processed includes a building image and a portrait experiencing tilt-shift distortion, and the line segments to be corrected include at least one of vertical and horizontal line segments in the building that are tilted in the image, and the line segments to be corrected do not include tilted line segments in the portrait. Only the tilted line segments in the building image are corrected, maintaining the effect of longer legs in the portrait and creating a more beautiful image composition.

[0114] The image capturing apparatus 300 may determine different line segment filtering policies based on different content included in the metadata. For example, step 830 includes steps 831 to 834.

[0115] Step 831: The image capture device 300 determines whether the metadata includes IMU information and field of view.

[0116] If the metadata includes the IMU information and the field of view, the line segment filtering policy instructs the image capture device 300 to perform line segment filtering based on the IMU information and the field of view, and the image capture device 300 executes step 832. Specifically, the image capture device 300 obtains the line segments to be corrected from the line segments included in the processing target image by filtering based on the IMU information and the field of view. For a solution in which the image capture device 300 performs line segment filtering based on the IMU information and the field of view, please refer to the detailed description in FIG. 6.

[0117] If the metadata does not include a field of view, the line segment filtering policy directs the image capture device 300 to perform line segment filtering based on the image content of the image being processed, and the image capture device 300 performs step 833. Specifically, the image capture device 300 , processing Based on the image content of the target image By filtering , the line segments to be corrected are obtained from the line segments included in the image to be processed, where the image content includes the line segment angle distribution feature. For a solution in which the image capture device 300 performs line segment filtering based on the image content of the image to be processed, please refer to the detailed description in FIG. 9 below.

[0118] The metadata includes the field of view. Muga If the IMU information is not included, the line segment filtering policy instructs the image capture device 300 to perform line segment filtering based on the image content and field of view of the target image, and the image capture device 300 performs step 834. Specifically, the image capture device 300 obtains the line segments to be corrected from the line segments included in the target image through filtering based on the field of view and the image content of the target image. For a solution in which the image capture device 300 performs line segment filtering based on the field of view and the image content of the target image, please refer to the detailed description in FIG. 10 below.

[0119] FIG. 9 is a schematic diagram of a line segment filtering method based on image content according to an embodiment of the present application.

[0120] The image capture device 300 may filter the horizontally slanted line segments and the vertically slanted line segments among the M line segments based on the line segment filtering model. For example, the image capture device 300 inputs the processing target image and the M line segment identifiers into the line segment filtering model to obtain a set of horizontally slanted line segments and a set of vertical line segments.

[0121] In some other embodiments, the image capture device 300 may detect vertically slanted line segments and horizontally slanted line segments among the M line segments according to a vanishing point detection method. and Including 920.

[0122] Step 910: The imaging device 300 performs pre-filtering based on the angles of the M line segments to obtain a set of pre-filtered vertical inclined line segments.

[0123] The imaging device 300 determines whether the angles of the M line segments are within a vertical line segment tilt threshold range [θ l e ,θ h e ] (e.g., [-75°, 75°]). If the line segment angle is within the vertical line segment slope threshold range, it indicates that the line segment angle is close to 90°. If the line segment angle is not within the vertical line segment slope threshold range, it indicates that the line segment angle is far from 90°, and the image capture device 300 determines whether the line segment angle is within the vertical line segment slope threshold range.

number

number

[0124] Step 920: The photographing device 300 obtains a set of vertical inclined line segments from the pre-filtered set of vertical inclined line segments through filtering using a vanishing point detection method.

[0125] The line segments in the vertical slanted line segment set intersect at a specific point, and the K equations in equation (2) above are combined to find the vanishing point (x vanish ,y vanish The vanishing point is the intersection of parallel lines. When two perpendicular parallel lines are viewed or photographed from a low angle, the two perpendicular parallel lines intersect at a point far away, i.e., the vanishing point, also called the vanishing point. The image capture device 300 then calculates the distances from the vanishing point of the perpendicular line segment to the K pre-filtered line segments.

number

[0126] Compared with the line segment filtering method based on image content, the line segment filtering method based on field of view and image content allows the vertical tilt threshold range corresponding to different field of view sizes to be adapted based on field of view. Under the same vertical tilt angle and the same horizontal tilt angle, the tilt angle range of vertical line segments in an image varies with different field of view. The wider the field of view, the greater the tilt of the vertical lines at the edge of the image. For example, as shown in FIG. 10 , the first field of view represented by the dashed box is smaller than the second field of view represented by the solid box. Line segments L1, L2, L3, and L4 are parallel to each other and perpendicular to the ground in a real scenario. However, the four vertical line segments are tilted because they are photographed from a low angle. If the four vertical line segments are photographed from the same low angle, the tilt angles of L3 and L4 will be larger than those of L1 and L2. Therefore, the vertical line segment tilt threshold range corresponding to different field of view sizes is adapted based on field of view to improve the accuracy of preliminary line segment filtering. The image capture device 300 may linearly adapt the vertical line segment tilt threshold range to correspond to the field of view: the wider the field of view, the wider the vertical line segment tilt threshold range.

[0127] Step 840: The photographing apparatus 300 corrects the line segment to be corrected and the feature region of the image to be processed, and obtains a processed image.

[0128] For a detailed description of step 840, please refer to the description of step 440. The details will not be described again.

[0129] Therefore, the image capture device 300 can adapt to different line segment filtering policies based on the content contained in the metadata, thereby improving the flexibility and accuracy of the line segment filtering performed on the line segments contained in the image being processed.

[0130] The image to be processed in this embodiment of the present application may be an image edited by a third-party image application. Therefore, the IMU information and field of view collected when the image was captured may not accurately describe the image. In some embodiments, the metadata may further include at least one of a trust identifier and an edit identifier. The trust identifier indicates whether the metadata is trustworthy. For example, if the trust identifier has a value of 1, it indicates that the metadata is untrustworthy, and if the trust identifier has a value of 0, it indicates that the metadata is trustworthy. The trust identifier may be provided by a front-end IMU module. Optionally, if geometric transformation edit operations such as cropping / rotation / rectification are performed on the image to be processed by a third-party image application, the IMU information and field of view are considered untrustworthy. The edit identifier indicates whether the image to be processed has been processed by an image application.

[0131] 11 , before performing step 831, the image capture device 300 may further perform step 850, i.e., determine whether the metadata is trustworthy. Determining whether the metadata is trustworthy may be replaced by determining whether the image to be processed has been processed by an image application.

[0132] If the trusted identifier indicates that the metadata is not trusted or the image to be processed is processed by an image application, step 833 is performed, i.e., the image capture device 300 , processing Based on the image content of the target image By filtering A line segment to be corrected is obtained from the line segments included in the image to be processed.

[0133] If the trusted identifier indicates that the metadata is trusted or the image to be processed is not processed by an image application, step 832 is performed, that is, the image capture device 300 obtains the line segments to be corrected from the line segments included in the image to be processed based on the IMU information and the field of view by filtering. Alternatively, step 834 is performed, that is, the image capture device 300 , visual Based on the image content of the image and the By filtering A line segment to be corrected is obtained from the line segments included in the image to be processed.

[0134] In this way, before the line segment filtering is performed based on the IMU information and the field of view, it is first predetermined whether the IMU information and the field of view are reliable, and a line segment filtering policy for the line segment filtering is determined based on the reliability of the IMU information and the field of view, thereby improving the line segment filtering rate and the accuracy of correcting distorted line segments in an image.

[0135] To further improve the line segment filtering results based on image content, this embodiment of the present application provides a user interaction line recommendation policy. For example, as shown in Figures 12A and 12B, after performing step 833 or step 834, the imaging device 300 may further perform step 860, that is, prompt the user to select a line segment to be corrected from the recommended line segments obtained by filtering based on the image content of the image to be processed. For example, step 860 may include the following detailed steps:

[0136] Step 861: The image capturing device 300 determines whether the horizontal direction is stably recognized.

[0137] If the horizontal direction is not stably recognized, step 862 is executed, i.e., the interface displays a recommended horizontal line segment. If the user does not select a recommended horizontal line segment, step 863 is executed, i.e., prompts the user to manually pick the horizontal tilt line segment that needs to be corrected.

[0138] If the horizontal direction is not stably recognized, step 864 is executed, ie, it is determined whether the vertical direction is stably recognized, and a horizontal tilt line segment is output.

[0139] If the vertical direction is not stably recognized, step 865 is executed, i.e., the interface displays a recommended vertical line segment. If the user does not select a recommended vertical line segment, step 866 is executed, i.e., prompts the user to manually pick the vertical tilt line segment that needs to be corrected.

[0140] If the vertical direction is known to be stable, then vertically slanted line segments are output.

[0141] If the image capture device 300 determines that both the horizontal and vertical directions are stably recognized, step 840 is executed, that is, the image capture device 300 corrects the obtained correction target line segment by performing line segment filtering based on the image content, corrects the feature region of the processing target image, and obtains a processed image. If neither the horizontal nor the vertical direction is stably recognized, the image capture device 300 highlights and displays a recommended horizontally inclined line segment or a recommended vertically inclined line segment in the interface, identifies the line segment selected by the user, and corrects the line segment selected by the user.

[0142] Optionally, the image capture device 300 may determine whether the horizontal and vertical directions are stably recognized based on the number and distribution of vertical and horizontal line segments obtained by performing line segment filtering based on the image content. For example, if the number of line segments obtained by filtering is very small or the distribution of the line segments is random, it can be considered that the horizontal and vertical directions are not stably recognized.

[0143] 13(a) and 13(b) are schematic diagrams of a line segment selection interface according to an embodiment of the present application. As shown in FIG. 13(a), if the image capture device 300 determines that the horizontal or vertical direction is not stably recognized, the interface displays a recommended line segment. As shown in FIG. 13(b), if the image capture device 300 determines that the horizontal or vertical direction is not stably recognized and the user does not select a recommended line segment, the interface prompts the user to manually select the tilted line segment that needs to be corrected.

[0144] In another possible implementation, when filtering the image to be processed based on two different line segment filtering policies, the image capture device 300 does not need to execute the process of determining the line segment filtering policy based on metadata in FIG. 8A . In other words, the image capture device 300 does not need to execute the process of step 830. In practice, the image capture device 300 filters the image to be processed based on two different line segment filtering policies, obtains the union or intersection of the line segment sets filtered based on the two different line segment filtering policies, and determines the final line segment to be corrected. For example, the image capture device 300 determines the line segment to be corrected based on the line segment filtered based on IMU information and field of view and the line segment filtered based on image content. The difference between this embodiment and the above embodiment is that, based on filtering the line segments based on IMU information and field of view, the image capture device 300 further identifies the line segment to be corrected based on the image content, thereby improving the accuracy of performing tilt-shift distortion correction. For example, line segment filtering is performed based on IMU information and field of view to obtain a line segment set LC1, line segment filtering is performed based on image content to obtain a line segment set LC2, and the final line segment set LC to be corrected is obtained by solving the union or intersection of LC1 and LC2. In this way, the accuracy of performing line segment filtering based on IMU information and field of view is further improved, and the accuracy of performing tilt-shift distortion correction on the image is improved.

[0145] It should be noted that the need to correct tilted line segments in an image is related to the user's intention and the photographed object. The need for tilt correction is usually high when the photographed subject is a straight line scenario with strong regularity, such as a building structure, a horizon, a poster, a window frame, a door frame, or a cabinet. The need for tilt correction is usually low when the photographed subject is a straight line scenario with weak regularity, such as a vertical object, an animal, food, a face, or the sky. Furthermore, including a large number of tilted line segments in an image is usually the user's intention. For example, when the photographed subject is a portrait photographed from a low angle or when the photographed tilt angle is extremely large, including a large number of tilted line segments in the image is usually the user's composition intention.

[0146] In another possible implementation, before the image capture device 300 performs step 840, i.e., corrects the line segments to be corrected, the image capture device 300 may further analyze the user's behavior and adaptively select whether to correct the vertically inclined line segments and the horizontally inclined line segments obtained by filtering, as shown in Figures 14A and 14B for example.

[0147] Step 870: The imaging apparatus 300 determines whether the number of the line segments to be corrected is greater than the quantity threshold and whether the length of the line segments to be corrected is greater than the length threshold.

[0148] If the number of the line segments to be corrected is greater than the quantity threshold and the length of the line segments to be corrected is greater than the length threshold, it indicates that the first scenario is a strongly regular straight line scenario, and step 840 is executed, i.e., correcting the line segments to be corrected to obtain a processed image.

[0149] If the number of line segments to be corrected is less than or equal to the quantity threshold, or the length of the line segments to be corrected is less than or equal to the length threshold, it indicates that the first scenario is a straight line scenario with weak regularity, and no automatic correction is performed, and step 880 is executed.

[0150] Step 880: The image capture device 300 determines whether the line segment to be corrected should be corrected based on the scenario characteristics of the first scenario.

[0151] The scenario features include architectural features and character features. The image capture device 300 can determine different horizontal line segment tilt thresholds and vertical line segment tilt thresholds based on different scenario features. The image capture device 300 determines whether the tilt angle of the horizontally tilted line segment is within a horizontal line segment tilt threshold range, and corrects the horizontally tilted line segment if the tilt angle of the horizontally tilted line segment is within the horizontal line segment tilt threshold range. The image capture device 300 also determines whether the tilt angle of the vertically tilted line segment is within a vertical line segment tilt threshold range, and corrects the vertically tilted line segment if the tilt angle of the vertically tilted line segment is within the vertical line segment tilt threshold range.

[0152] For example, the images to be processed include a portrait image taken from a low angle and an image of a building taken from a low angle. Correcting vertically tilted line segments in the portrait may ruin the "leg-lengthening" effect. In this case, horizontally tilted line segments may be corrected, and vertically tilted line segments and horizontally tilted line segments in the building may be corrected, but vertically tilted line segments in the portrait are not corrected. It is usually the user's intention to include large horizontal tilt angles and large pitch tilt angles in the image. In this case, vertically tilted line segments and horizontally tilted line segments in the image are not corrected. For example, the image capture device 300 corrects line segments that are offset by 25° from 90° and 0°.

[0153] In this way, when the tilted line segments in the image are corrected, the scenario characteristics and user intention are fully reflected, so that the corrected image better follows the distribution characteristics of the line segments in the actual scenario, and therefore the accuracy of performing tilt-shift distortion correction on the image is effectively improved.

[0154] In this embodiment of the present application, the filtering of tilted line segments is performed with reference to image metadata, image content analysis results, and user interaction line recommendation policies, and the constraints are jointly optimized to obtain the global displacement caused by the vertical tilt caused by correcting the pitch tilt angle and the global displacement caused by correcting the horizontal tilt caused by the horizontal tilt angle, and the tilt-shift distortion correction of different angles is adaptively performed with reference to the line detection results and user behavior analysis, thus effectively improving the accuracy of performing tilt-shift distortion correction on images.

[0155] The following describes the interface operations to perform tilt-shift distortion correction on an image by using an example.

[0156] 15(a), 15(b), 15(c), 15(d), 15(e), and 15(f) are schematic diagrams of an image processing interface according to an embodiment of the present application. FIG. 15(a) is a schematic front view of a smartphone according to an embodiment of the present application. An album application (APP) icon 1501 is displayed on the screen of the smartphone, and a user may tap the album application icon 1501. A touch sensor included in the smartphone receives the touch operation and reports the touch operation to a processor, enabling the processor to launch the album application in response to the touch operation. Furthermore, in this embodiment of the present application, the smartphone may alternatively launch the album application in other ways and display the user interface of the album application on the display screen. For example, when the display screen of the smartphone is turned off, a lock screen interface is displayed, or a user interface is displayed after the smartphone is unlocked, the smartphone may launch the album application and display the user interface of the album application on the display screen in response to a user voice command, a shortcut operation, or the like. As shown in FIG. 15(b), the smartphone responds to the tap operation, and the smartphone's display screen displays a user interface for an album application. The user interface includes photos taken at different time periods and a photo search box. The user can select a photo 1502 taken at an angle and have tilt-shift distortion and perform tilt-shift distortion correction on the tilt-shift distortion photo 1502. As shown in FIG. 15(c), the smartphone responds to the touch operation, and the smartphone's display screen displays the tilt-shift distortion photo 1502 and photo editing function buttons, such as a "Send" function button, an "Edit" function button, a "Favorite" function button, a "Delete" function button, and an "More" function button. The user can tap the "Edit" function button 1503, and the smartphone's display screen displays function options for the "Edit" function.As shown in FIG. 15(d), the smartphone responds to the tap operation, and the display screen of the smartphone displays function options for the "Edit" function. The function options for the "Edit" function include Intelligence, Crop, Filter, and Adjust. The user can tap the "Intelligence" option button 1504, and the display screen of the smartphone displays multiple function options for the "Intelligence" option. For example, the "Intelligence" option includes Auto Optimize, reflection removal, These functions include brightening, defogging, and architectural correction. As shown in FIG. 15(e), the smartphone responds to a tap operation, and the display screen of the smartphone may further display a "Tilt-Shift Distortion" function option, a "Horizontal Correction" function option, a "Vertical Correction" function option, and the like, which are included in the "Intelligence" option. The user can tap the "Tilt-Shift Distortion" function option 1505. The smartphone performs tilt-shift distortion correction on the tilt-shift-distorted photo 1502. As shown in FIG. 15(f), the smartphone responds to a tap operation, and the display screen displays a photo 1506 obtained after performing tilt-shift distortion correction on the tilt-shift-distorted photo 1502. For a specific method of performing tilt-shift distortion correction on the tilt-shift-distorted photo 1502 by the smartphone, please refer to the description in the above embodiment. Details will not be described again.

[0157] In some other embodiments, the image capture device may alternatively perform tilt-shift distortion correction on the image based on a user's tilt-shift distortion operation after capturing the photo. For example, FIGS. 16(a), 16(b), 16(c), 16(d), 16(e), 16(f), and 16(g) are schematic diagrams of other image processing interfaces according to embodiments of the present application. FIG. 16(a) is a schematic front view of a smartphone according to an embodiment of the present application. A camera application (APP) icon 1601 is displayed on the smartphone's display screen, and a user may tap the camera application icon 1601. A touch sensor included in the smartphone receives the touch operation and reports the touch operation to a processor, enabling the processor to launch the camera application in response to the touch operation. Furthermore, in this embodiment of the present application, the smartphone may alternatively launch the camera application in other ways and display the user interface of the camera application on the display screen. For example, when the smartphone's display screen is turned off, a lock screen interface is displayed, or a user interface is displayed after the smartphone is unlocked, the smartphone may launch a camera application in response to a user's voice command, a shortcut operation, or the like, and display the user interface of the camera application on the display screen. As shown in FIG. 16(b), the smartphone responds to a tap operation, and the smartphone's display screen displays the user interface of the camera application. The user interface includes mode options such as a "short video" mode option, a "record" mode option, a "photo" mode option, a "portrait" mode option, and a "panorama" mode option. The user interface further includes function buttons such as an "image preview" function button, a "photo" function button, and a "front / rear camera switch" function button. The user may first select the "photo capture" mode 1602 and then tap the "photo" function button 1603.In response to the touch operation, the smartphone automatically activates the camera, and the camera captures an image of a scenario taken from a low angle. Furthermore, as shown in FIG. 16(c), the user may tap an “Image Preview” function button 1604 to view the captured image including the target object, i.e., a photo 1605 with tilt-shift distortion. As shown in FIG. 16(d), the smartphone responds to the tap operation, and the smartphone's display screen displays the photo 1605 with tilt-shift distortion and photo editing function buttons, such as a “Send” function button, an “Edit” function button, a “Favorite” function button, a “Delete” function button, and an “More” function button. The user can tap the “Edit” function button 1606, and the smartphone's display screen displays function options for the “Edit” function. As shown in FIG. 16(e), the smartphone responds to the tap operation, and the smartphone's display screen displays function options for the “Edit” function. The function options for the “Edit” function include intelligence, cropping, filters, and adjustments. The user can tap the "Intelligence" option button 1607, and the display screen of the smartphone displays multiple function options for the "Intelligence" option. For example, the "Intelligence" option includes: Automatic optimization, These functions include reflection removal, brightening, haze removal, and architecture correction. As shown in FIG. 16(f), the smartphone responds to a tap operation, and the display screen of the smartphone may further display a "Tilt-Shift Distortion" function option, a "Horizontal Correction" function option, a "Vertical Correction" function option, and the like, which are included in the "Intelligence" option. The user can tap the "Tilt-Shift Distortion" function option 1608. The smartphone performs tilt-shift distortion correction on the tilt-shift-distorted photo 1605. As shown in FIG. 16(g), the smartphone responds to a tap operation, and the display screen displays a photo 1609 obtained after performing tilt-shift distortion correction on the tilt-shift-distorted photo 1605. For a specific method of performing tilt-shift distortion correction on the tilt-shift-distorted photo 1605 by the smartphone, please refer to the description in the above embodiment. Details will not be described again.

[0158] In some other embodiments, the image capture device may automatically perform tilt-shift distortion correction on an image after the image capture device captures a photo. For example, as shown in FIG. 16(d), a smartphone responds to a tap operation, and the smartphone's display screen displays a user interface of a camera application. The smartphone automatically launches the camera in response to a tap of the "Photo" function button 1603. The camera captures an image of a scenario taken from a low angle. The smartphone performs tilt-shift distortion correction on the tilt-shift-distorted photo 1605 taken from a low angle to obtain a tilt-shift-distorted-corrected photo 1609. As shown in FIG. 16(g), the smartphone's display screen displays the captured photo 1609 after performing tilt-shift distortion correction on the tilt-shift-distorted photo 1605. Steps in Figure 16(c) to Figure 16(f) is omitted, which improves the user experience when using the smartphone.

[0159] It should be noted that the display methods of the "tilt-shift distortion" function option, the "horizontal correction" function option, and the "vertical correction" function option are not limited in this embodiment of the present application. The function options displayed on the display screen of the image capture device may also be referred to as controls or function controls.

[0160] To implement the functions in the above embodiments, the image capture device includes a corresponding hardware structure and / or a corresponding software module for performing each function. Those skilled in the art will easily recognize that the present application can be implemented by using hardware or a combination of hardware and computer software, in combination with the units and method steps in the examples described in the embodiments disclosed herein. Whether the functions are performed by hardware or hardware driven by computer software depends on the specific application scenario and design constraints of the technical solution.

[0161] The above describes in detail the image processing method according to this embodiment with reference to Figures 1 to 16(a), 16(b), 16(c), 16(d), 16(e), 16(f), and 16(g), and below describes the image processing device according to this embodiment with reference to Figure 17.

[0162] 17 is a schematic diagram of a possible structure of an image processing device according to an embodiment. The image processing device may be configured to implement the functions of the image capture device in the above method embodiment. Therefore, the image processing device can also implement the advantageous effects of the above method embodiment. In this embodiment, the image processing device may be the image capture device 300 shown in FIG. 3, or may be a module (e.g., a chip) applied to a server.

[0163] As shown in Figure 17, image processing device 1700 includes a communication module 1710, a line segment filtering module 1720, a correction module 1730, and a storage module 1740. Image processing device 1700 is configured to implement the functionality of image capture device 300 in the method embodiments shown in Figures 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B.

[0164] The communication module 1710 is configured to acquire a processing target image, the processing target image including at least one of a vertical line segment and a horizontal line segment in a first scenario, the first scenario being a scenario in which the image capture device acquires the processing target image. For example, the communication module 1710 is configured to perform step 410. As another example, the communication module 1710 is configured to perform step 810.

[0165] The line segment filtering module 1720 , processing Based on the metadata of the processed image By filtering, and obtains a correction target line segment from line segments included in the processing target image, where the metadata includes at least one of a field of view of the image capture device and inertial measurement unit (IMU) information. For example, the line segment filtering module 1720 is configured to perform step 420 and step 430. As another example, the line segment filtering module 1720 is configured to perform step 820, step 830, step 850, step 860, step 870, and step 880.

[0166] The correction module 1730 is configured to correct the line segments to be corrected and characteristic regions of the processing image to obtain a processed image, where the characteristic regions include a background region, a portrait region, and an edge region. For example, the correction module 1730 is configured to perform step 440. As another example, the correction module 1730 is configured to perform step 840.

[0167] Optionally, the image processing device 1700 may upload the processed images to a server, edge device, or cloud device, or store the processed images in local memory.

[0168] The storage module 1740 is configured to store metadata, where the metadata indicates characteristic information of the image to be processed, such as IMU information and field of view, and to aid in determining a line segment filtering policy based on the metadata of the image to be processed.

[0169] Optionally, the image processing device 1700 may further include a display module 1750, which is configured to display the image to be processed and the processed image.

[0170] The image processing device 1700 in this embodiment of the present application may be implemented by using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Alternatively, the image processing device 1700 and the modules of the image processing device 1700 may be software modules, in which case the image processing device may implement the image processing methods shown in FIGS. 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B by using software. stomach.

[0171] The image processing device 1700 according to this embodiment of the present application can accordingly perform the methods described in the embodiments of the present application. Furthermore, the above and other operations and / or functions of the units of the image processing device 1700 are used to implement the corresponding procedures of the methods of Figures 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B, respectively. For the sake of brevity, the details will not be described again here.

[0172] 18 is a schematic diagram of the structure of an image capture device 1800 according to an embodiment. As shown, the image capture device 1800 includes a processor 1810, a bus 1820, a memory 1830, a communication interface 1840, a memory unit 1850 (which may also be referred to as a main memory unit), and a camera 1860. Optionally, the image capture device 1800 may further include a display 1870, which is configured to display the image to be processed and the processed image. The processor 1810, the memory 1830, the memory unit 1850, the communication interface 1840, the camera 1860, and the display 1870 are connected via the bus 1820.

[0173] It should be understood that in this embodiment, processor 1810 may be a CPU. Alternatively, processor 1810 may be another general-purpose processor or digital signal processing (DSP), an ASIC, an FPGA or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0174] Alternatively, the processor may be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits configured to control program execution of the solutions of the present application.

[0175] The communication interface 1840 is configured to implement communications between the image capture device 1800 and peripheral devices or components. In this embodiment, when the image capture device 1800 is configured to implement the functionality of the image capture device 300 shown in Figures 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B, the communication interface 1840 is configured to receive images to be processed.

[0176] The bus 1820 may include paths used to transfer information between the above components (e.g., the processor 1810, the memory unit 1850, and the memory 1830). In addition to a data bus, the bus 1820 may further include a power bus, a control bus, a status signal bus, etc. However, for clarity of description, various buses are depicted in the figure as the bus 1820. The bus 1820 may be a Peripheral Component Interconnect Express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (unified bus, Ubus, or UB), a Computer Express Link (CPL), or a similar bus. r express link (CXL), cache coherent interconnect for accelerators (CCIX), etc. The bus 1820 may be categorized into an address bus, a data bus, a control bus, etc.

[0177] In an example, the imager 1800 may include multiple processors. The processor may be a multi-core (multi-CPU) processor. Here, a processor may be one or more devices, circuits, and / or computing units configured to process data (e.g., computer program instructions). In this embodiment, when the imager 1800 is configured to implement the functionality of the imager 300 shown in FIGS. 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B, the processor 1810 accesses metadata stored in the memory 1830. , processing Based on at least one of the IMU information and field of view included in the metadata of the processed image. By filtering , a line segment to be corrected contained in the image to be processed is obtained, and the line segment to be corrected and a feature region of the image to be processed are corrected to obtain a processed image, where the feature region includes a background region, a portrait region, and an edge region.

[0178] 18, it is noted that the image capture device 1800 includes only one processor 1810 and one memory 1830 as an example, where the processor 1810 and the memory 1830 are each configured to indicate a certain type of component or device. In a specific embodiment, the number of each type of component or device may be determined based on service requirements.

[0179] The memory unit 1850 may correspond to a storage medium configured to store information such as metadata in the method embodiments described above. The memory unit 1850 may be volatile or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM) and may be used as an external cache. By way of example, but not limitation, numerous forms of RAM may be used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DRAM), and so on. rate There are various types of RAM: SDRAM (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (synchlink DRAM, SLDRAM), and direct rambus random access memory (direct rambus RAM, DR RAM).

[0180] The memory 1830 may correspond to a storage medium configured to store information such as computer instructions in the method embodiments described above, for example, a magnetic disk such as a mechanical hard disk, or a solid-state disk.

[0181] Imager 1800 may be a general-purpose device or a dedicated device. For example, imager 1800 may be an edge device (e.g., a box carrying a chip with processing capabilities). Optionally, imager 1800 may alternatively be a server or other device with computing capabilities.

[0182] It should be understood that the imaging device 1800 according to this embodiment may correspond to the image processing device 1700 according to the embodiment, and may correspond to a corresponding main body that performs any of the methods described in Figures 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B. Furthermore, the above and other operations and / or functions of the modules of the image processing device 1700 are used to implement corresponding procedures of the methods in Figures 4, 6, 8A and 8B, 11, 12A and 12B, and 14A and 14B, respectively. Details will not be described again here.

[0183] The method steps of the present embodiment may be implemented by hardware or by a processor executing software. The software instructions may include corresponding software modules. The software modules may be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable hard disk, a CD-ROM, or any other form of storage medium well known in the art. For example, the storage medium may be coupled to the processor, thereby enabling the processor to read information from and write information to the storage medium. Indeed, the storage medium may be a component of the processor. The processor and the storage medium may be located in an ASIC. Furthermore, the ASIC may be located in the imaging device. Indeed, the processor and the storage medium may reside as discrete components in the imaging device.

[0184] All or part of the above embodiments may be implemented using software, hardware, firmware, or any combination thereof. When software is used to implement an embodiment, all or part of the embodiment may be implemented in the form of a computer program product. The computer program product includes one or more computer programs and instructions. When the computer program or instructions are loaded and executed on a computer, all or part of the procedures or functions of the embodiments of the present application are executed. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer program or instructions may be stored in a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless method. The computer-readable storage medium may be any available medium accessible by a computer or a data storage device, such as a server or data center, incorporating one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video disc (DVD); or a semiconductor medium, such as a solid state drive (SSD). The above description is merely a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should fall within the scope of protection of the present application. Therefore, the scope of protection of the present application should be governed by the scope of protection of the claims.

Claims

1. 1. An image processing method, comprising: acquiring an image to be processed, the image to be processed being a captured image of a first scenario, the first scenario including at least one of a vertical line segment and a horizontal line segment; obtaining a correction target line segment from line segments included in the image to be processed by filtering based on metadata of the image to be processed, the metadata including at least one of field of view and inertial measurement unit (IMU) information of an image capture device used to acquire the captured image, the IMU information indicating a tilt angle of the image capture device, the image to be processed including a line segment that is tilted in the image to be processed and is at least one of the vertical line segment and the horizontal line segment in the first scenario, and the correction target line segment including a part or all of the line segment that is tilted in the image to be processed and is at least one of the vertical line segment and the horizontal line segment in the first scenario; correcting the line segment to be corrected and a feature region of the image to be processed to obtain a processed image, the feature region including a background region, a portrait region, and an edge region; A method having the following.

2. the image capture device displays tilt-shift distortion controls; Before obtaining the correction target line segment from the line segments included in the processing target image by filtering based on the metadata of the processing target image, the method includes: and receiving a user-performed operation on the tilt-shift distortion control. The method of claim 1.

3. Obtaining the correction target line segment from the line segments included in the processing target image by filtering based on the metadata of the processing target image, obtaining the correction target line segment from the line segments included in the processing target image by filtering based on a line segment filtering policy determined based on the metadata, wherein the line segment filtering policy indicates a method of performing line segment filtering on the line segments included in the processing target image. The method of claim 1.

4. the metadata includes the field of view and the IMU information, and the line segment filtering policy instructs performing line segment filtering based on the field of view and the IMU information; acquiring the correction target line segment from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, According to the field of view and the IMU information, perform a global adjustment on M line segments included in the processing target image to obtain M adjusted line segments, wherein the M adjusted line segments satisfy the line segment angle distribution characteristics of the first scenario; Obtaining N line segments to be corrected from the M adjusted line segments by filtering based on a line segment slope threshold, wherein the line segment slope threshold includes a horizontal line segment slope threshold and a vertical line segment slope threshold, M and N are both positive integers, and N is less than or equal to M; having The method of claim 3.

5. the metadata does not include the field of view; acquiring the correction target line segment from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, obtaining the correction target line segment from the line segments included in the processing target image by filtering based on image content of the processing target image, wherein the image content includes a line segment angle distribution feature; The method of claim 3.

6. the metadata includes the field of view but does not include the IMU information; acquiring the correction target line segment from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, obtaining the correction target line segment from the line segments included in the processing target image by filtering based on the field of view and image content of the processing target image, wherein the image content includes a line segment angle distribution feature; The method of claim 3.

7. The method comprises: and prompting a user to select the line segment to be corrected from recommended line segments obtained by filtering based on the image content of the image to be processed. The method of claim 5.

8. the metadata further includes a trust identifier, the trust identifier indicating whether the metadata is trustworthy; acquiring the correction target line segment from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, If the trusted identifier indicates that the metadata is untrustworthy, obtaining the correction target line segment from the line segments included in the processing target image by filtering based on image content of the processing target image, wherein the image content includes a line segment angle distribution feature; If the trusted identifier indicates that the metadata is trusted, obtaining the correction target line segment from the line segments included in the processing target image by filtering based on at least one of the image content, the IMU information, and the field of view of the processing target image. having The method of claim 3.

9. Obtaining the correction target line segment from the line segments included in the processing target image by filtering based on the metadata of the processing target image, determining the line segments to be corrected based on the metadata-filtered line segments and the image content-filtered line segments. The method of claim 1.

10. Correcting the line segment to be corrected and the feature region of the image to be processed to obtain the processed image comprises: determining whether a number of the line segments to be corrected is greater than a quantity threshold and whether a length of the line segments to be corrected is greater than a length threshold; if the number of the line segments to be corrected is greater than the quantity threshold and the length of the line segments to be corrected is greater than the length threshold, correcting the line segments to be corrected to obtain the processed image; If the number of the line segments to be corrected is less than or equal to the quantity threshold, or the length of the line segments to be corrected is less than or equal to the length threshold, determining whether to correct the line segments to be corrected based on scenario features of the first scenario, wherein the scenario features include architectural features and character features; having The method of claim 1.

11. Correcting the line segment to be corrected and the feature region of the image to be processed to obtain the processed image comprises: constructing a constraint for the correction based on the line segment to be corrected and the characteristic region of the image to be processed, and performing the correction in accordance with the constraint; The method of claim 1.

12. The constraints include a straight line constraint for correcting the line segment to be corrected, a content-based homography constraint, a shape constraint, and a regularity constraint for correcting the background region, a portrait constraint for correcting the portrait region, and an edge constraint for correcting the edge region. The method of claim 11.

13. An image processing device, a communication module configured to acquire an image to be processed, the image to be processed being a captured image of a first scenario, the first scenario including at least one of a vertical line segment and a horizontal line segment; a line segment filtering module configured to obtain a correction target line segment from line segments included in the processing target image by filtering based on metadata of the processing target image, the metadata including at least one of field of view and inertial measurement unit (IMU) information of an image capture device used to acquire the captured image, the IMU information indicating a tilt angle of the image capture device, the processing target image including line segments that are tilted in the processing target image and are at least one of the vertical line segment and the horizontal line segment in the first scenario, and the correction target line segment including some or all of the line segments that are tilted in the processing target image and are at least one of the vertical line segment and the horizontal line segment in the first scenario; a correction module configured to correct the line segment to be corrected and a feature region of the processing target image to obtain a processed image, the feature region including a background region, a portrait region, and an edge region; A device having:

14. the device further comprises a display module; the display module is configured to display a tilt-shift distortion control and receive a user-performed operation on the tilt-shift distortion control.

14. The apparatus of claim 13.

15. When the correction target line segment is acquired from the line segments included in the processing target image by filtering based on the metadata of the processing target image, the line segment filtering module and obtaining the correction target line segment from the line segments included in the processing target image by filtering based on a line segment filtering policy determined based on the metadata, the line segment filtering policy instructing how to perform line segment filtering on the line segments included in the processing target image.

14. The apparatus of claim 13.

16. the metadata includes the field of view and the IMU information, and the line segment filtering policy instructs performing line segment filtering based on the field of view and the IMU information; When acquiring the correction target line segment from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, the line segment filtering module: According to the field of view and the IMU information, perform a global adjustment on M line segments included in the processing target image to obtain M adjusted line segments, and the M adjusted line segments satisfy the line segment angle distribution characteristic of the first scenario; Obtaining N line segments to be corrected from the M adjusted line segments by filtering based on a line segment slope threshold, the line segment slope threshold including a horizontal line segment slope threshold and a vertical line segment slope threshold, where M and N are both positive integers, and N is less than or equal to M; Specifically configured to 16. The apparatus of claim 15.

17. When the metadata does not include the field of view, and the correction target line segment is acquired from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, the line segment filtering module: and obtaining the correction target line segment from the line segments included in the processing target image by filtering based on an image content of the processing target image, the image content including a line segment angle distribution feature.

16. The apparatus of claim 15.

18. When the metadata includes the field of view but does not include the IMU information, and the correction target line segment is acquired from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, the line segment filtering module: and obtaining the correction target line segment from the line segments included in the processing target image by filtering based on the field of view and an image content of the processing target image, wherein the image content includes a line segment angle distribution feature.

16. The apparatus of claim 15.

19. the device further comprises a display module; the display module is configured to prompt a user to select the line segment to be corrected from recommended line segments obtained by filtering based on the image content of the image to be processed.

18. The apparatus of claim 17.

20. The metadata further includes a trust identifier, and the trust identifier indicates whether the metadata is trustworthy. When the correction target line segment is acquired from the line segments included in the processing target image by filtering based on the line segment filtering policy determined based on the metadata, the line segment filtering module: If the trusted identifier indicates that the metadata is untrustworthy, obtaining the correction target line segment from the line segments included in the processing target image by filtering based on image content of the processing target image, wherein the image content includes a line segment angle distribution feature; If the trusted identifier indicates that the metadata is trusted, the correction target line segment is obtained from the line segments included in the processing target image by filtering based on at least one of the image content, the IMU information, and the field of view of the processing target image. Specifically configured to 16. The apparatus of claim 15.

21. When the correction target line segment is acquired from the line segments included in the processing target image by filtering based on the metadata of the processing target image, the line segment filtering module and determining the line segments to be corrected based on the metadata-based filtered line segments and the image content-based filtered line segments.

14. The apparatus of claim 13.

22. When correcting the line segment to be corrected and the feature region of the image to be processed to obtain the processed image, the correction module determining whether a number of the line segments to be corrected is greater than a quantity threshold and whether a length of the line segments to be corrected is greater than a length threshold; if the number of the line segments to be corrected is greater than the quantity threshold and the length of the line segments to be corrected is greater than the length threshold, correcting the line segments to be corrected to obtain the processed image; If the number of the line segments to be corrected is less than or equal to the quantity threshold, or the length of the line segments to be corrected is less than or equal to the length threshold, determining whether to correct the line segments to be corrected based on scenario features of the first scenario, wherein the scenario features include architecture features and character features. Specifically configured to 14. The apparatus of claim 13.

23. The correction module is particularly configured to establish constraints for the correction based on the line segment to be corrected and the feature region of the image to be processed, and to perform the correction in accordance with the constraints.

14. The apparatus of claim 13.

24. The constraints include a straight line constraint for correcting the line segment to be corrected, a content-based homography constraint, a shape constraint, and a regularity constraint for correcting the background region, a portrait constraint for correcting the portrait region, and an edge constraint for correcting the edge region.

24. The apparatus of claim 23.

25. 1. An imaging device having at least one processor, a memory, a camera, and an inertial measurement unit (IMU), the camera is configured to capture an image; the IMU is configured to acquire IMU information of the imaging device; the memory is configured to store computer programs and instructions; The processor is configured to call the computer program and the instructions to cause the camera and the IMU to perform the method of any one of claims 1 to 15. Filming equipment.

26. having at least one processor and memory; the memory is configured to store computer programs and instructions; The processor is configured to call the computer program and the instructions to perform the method of any one of claims 1. Image processing system.

Images