Image cropping method and device, electronic device and storage medium
By combining the boundary information of color images and depth images, the problem of low image cropping accuracy is solved, and the effect of accurate cropping can be achieved even under missing or interfering information.
Patent Information
- Application Number
- CN202080102636.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-14
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-07-14
AI Technical Summary
In the prior art, in the case of image cropping, especially when image information is missing or interfering information exists, the accuracy of cropping is low.
By obtaining the color image and depth image of the target object, perspective transformation is performed to extract the image boundary information, and point cloud boundary information is determined based on the point cloud data generated by the depth image, and the target boundary information is determined based on the two, and the color image is finally cropped.
Even if there are no obvious lines or interference information, the target boundary information can be accurately obtained, which improves the accuracy of image cropping and thus improves the user experience.
Smart Images

Figure CN115836322B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to an image cropping method, an image cropping device, an electronic device, and a computer-readable storage medium. Background Art
[0002] With the development of terminal devices, the image processing capabilities of terminal devices are gradually enhanced. Among them, when cropping images, a combination of traditional methods and deep learning methods can be used for automatic cropping. However, when image information is missing or there is interference information, the accuracy of image cropping is low. Summary of the invention
[0003] The purpose of the present disclosure is to provide an image cropping method, an image cropping device, an electronic device and a computer-readable storage medium, thereby overcoming the problem of low accuracy of image cropping due to limitations and defects of related technologies to a certain extent.
[0004] According to a first aspect of the present disclosure, there is provided an image cropping method, comprising:
[0005] Acquire color images and depth images of the target object;
[0006] Performing perspective transformation on the color image to obtain a perspective transformed image, and extracting image boundary information of the target object in the perspective transformed image;
[0007] Determining point cloud boundary information of the target object according to the point cloud data generated based on the depth image;
[0008] Obtaining target boundary information based on the image boundary information and the point cloud boundary information;
[0009] The color image is cropped according to the target boundary information to obtain a cropped image.
[0010] According to a second aspect of the present disclosure, there is provided an image cropping device, comprising:
[0011] An image acquisition module, used to acquire a color image and a depth image of a target object;
[0012] An image boundary information determination module, used to perform perspective transformation on the color image to obtain a perspective transformed image, and to extract image boundary information of the target object in the perspective transformed image;
[0013] A point cloud boundary information determination module, used to determine the point cloud boundary information of the target object according to the point cloud data generated based on the depth image;
[0014] A target boundary information determination module, used to obtain target boundary information based on the image boundary information and the point cloud boundary information;
[0015] The image cropping module is used to crop the color image according to the target boundary information to obtain a cropped image.
[0016] According to a third aspect of the present disclosure, there is provided an electronic device, including:
[0017] Processor; and
[0018] a memory configured to store executable instructions of the processor;
[0019] The processor is configured to perform the above-mentioned image cropping method by executing the executable instructions.
[0020] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the above-mentioned image cropping method when executed by a processor.
[0021] The exemplary embodiments of the present disclosure may have at least some or all of the following beneficial effects:
[0022] In an image cropping method provided in an example embodiment of the present disclosure, the point cloud boundary information of the target object is determined by the depth image, so that the cropping range of the target object can be obtained. And the target boundary information of the target object in the color image is combined to determine the target boundary information of the target object. In this way, even if the target object has no obvious lines or there is interference information, the target boundary information can be accurately obtained. Furthermore, by cropping the color image according to the target boundary information, the accuracy of cropping can be improved, thereby improving the user experience.
[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0025] Figure 1 A schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present disclosure is shown;
[0026] Figure 2A flow chart of an image cropping method in an embodiment of the present disclosure is shown;
[0027] Figure 3 A flowchart of perspective transformation in an embodiment of the present disclosure is shown;
[0028] Figure 4 A schematic diagram of region detection is shown;
[0029] Figure 5 A flow chart of generating point cloud boundary information in an embodiment of the present disclosure is shown;
[0030] Figure 6 A schematic diagram showing a target object and a point cloud boundary in an embodiment of the present disclosure is shown;
[0031] Figure 7 A schematic diagram showing line segment detection of an image in an embodiment of the present disclosure is shown;
[0032] Figure 8 A schematic diagram of determining a target boundary in an embodiment of the present disclosure is shown;
[0033] Fig. 9 A schematic diagram showing a cropped image in a view mode and an edit mode in an embodiment of the present disclosure is shown;
[0034] Fig.10 A schematic diagram showing a cropped image in a perspective transformation mode in an embodiment of the present disclosure is shown;
[0035] Fig.11 A schematic diagram of the structure of an image cropping device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0037] It should be noted that, in the present disclosure, the terms "including", "configured with", and "set in" are used to express an open-ended inclusion, and mean that in addition to the listed elements / components / etc., there may be additional elements / components / etc.; the terms "first", "second", etc. are only used as labels and are not limitations on the quantity or order of their objects.
[0038] See also Figure 1 , Figure 1FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Figure 1 The electronic device 100 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0039] like Figure 1 As shown, the electronic device 100 may specifically include: a processor 110, a wireless communication module 120, a mobile communication module 130, a charging management module 140, a power management module 141, a battery 142, a USB (Universal Serial Bus) interface 150, an antenna 1, an antenna 2, an internal memory 161, an external memory interface 162, a display screen 170, a sensor module 180 and a camera module 190, etc.
[0040] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0041] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a video codec, a digital signal processor, a baseband processor, and / or a neural network processor. Different processing units may be independent devices or integrated into one or more processors. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.
[0042] A memory may also be provided in the processor 110 for storing instructions and data. The memory may store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution is controlled by the processor 110. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that have just been used or are cyclically used by the processor 110. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory, thereby avoiding repeated access, reducing the waiting time of the processor 110, and thus improving the efficiency of the system.
[0043] The display screen 170 is used to display images, videos, etc.
[0044] The sensor module 180 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a fingerprint sensor, a temperature sensor, and a touch sensor, etc. Among them, the depth sensor is used to obtain the depth information of the scene. In some embodiments, the depth sensor can be set in the camera module 190.
[0045] The camera module 190 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to the image signal processor for conversion into a digital image signal. The image signal processor outputs the digital image signal to the digital signal processor for processing. The digital signal processor converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include one or more camera modules 190.
[0046] The technical solution of the embodiment of the present disclosure is described in detail below.
[0047] In image cropping technology, images can be cropped based on the boundary information of objects. However, in the case of a lack of available boundaries in the image and the presence of interfering boundaries, the accuracy of image cropping is low.
[0048] In order to solve the above problems, the present disclosure provides an image cropping method and device, an electronic device and a computer-readable storage medium, which can improve the accuracy of image cropping.
[0049] See also Figure 2 , Figure 2 A flow chart of the image cropping method in the embodiment of the present disclosure is shown, which may include the following steps:
[0050] Step S210, acquiring a color image and a depth image of the target object.
[0051] Step S220 , performing perspective transformation on the color image to obtain a perspective transformed image, and extracting image boundary information of the target object in the perspective transformed image.
[0052] Step S230: determining point cloud boundary information of the target object according to the point cloud data generated based on the depth image.
[0053] Step S240, obtaining target boundary information based on the image boundary information and the point cloud boundary information.
[0054] Step S250, cropping the color image according to the target boundary information to obtain a cropped image.
[0055] The image cropping method of the embodiment of the present disclosure determines the point cloud boundary information of the target object through the depth image, so as to obtain the cropping range of the target object. And the target boundary information of the target object is determined in combination with the image boundary information of the target object in the color image. In this way, even if the target object has no obvious lines or there is interference information, the target boundary information can be accurately obtained. Furthermore, by cropping the color image according to the target boundary information, the accuracy of cropping can be improved, thereby improving the user experience.
[0056] The image cropping method of the embodiment of the present disclosure is introduced in more detail below.
[0057] In step S210 , a color image and a depth image of the target object are acquired.
[0058] In the disclosed embodiments, the target object can be any object that the user wants to shoot, and can be a person, an animal, or a scene, etc. The electronic device can include a plurality of different camera modules, so that different images can be taken for the same target object. For example, one of the camera modules can capture a color image (such as an RGB image or a YUV image, etc.). Another camera module can capture the depth information of the target object to obtain a depth image. For example, the depth information of the target object can be obtained by a TOF (Time of Flight) sensor. Specifically, the TOF sensor emits modulated near-infrared light, and the near-infrared light is reflected after encountering an object. The distance between the TOF sensor and the object being photographed is determined by calculating the time difference or phase difference between the emission and reflection of the light, thereby obtaining the depth information.
[0059] The TOF sensor is not affected by changes in illumination and object texture, and can reduce costs while meeting accuracy requirements. Using the three-dimensional data of the TOF sensor to assist in acquiring the three-dimensional information of the target object and calculating the perspective transformation matrix can make document scanning (referring to correcting the perspective relationship) independent of image information, greatly improving the scope of application of document scanning. Of course, the depth sensor can also be a structured light sensor or a binocular sensor, etc., which is not limited in this disclosure.
[0060] In step S220, perspective transformation is performed on the color image to obtain a perspective transformed image, and image boundary information of the target object in the perspective transformed image is extracted.
[0061] It should be noted that perspective transformation is the process of projecting an image onto a new viewing plane. For example, an object that is a straight line in reality may appear as a diagonal line in an image, and the diagonal line can be converted into a straight line through perspective transformation.
[0062] See also Figure 3 , Figure 3A flowchart of perspective transformation in an embodiment of the present disclosure is shown, which may include the following steps:
[0063] Step S310: performing plane detection on the point cloud data generated based on the depth image, and determining a perspective transformation matrix according to the detected plane.
[0064] Among them, the depth image can generate point cloud data after coordinate conversion, that is, the three-dimensional coordinates in the depth image are converted into three-dimensional coordinates in the camera coordinate system to generate three-dimensional point cloud data. After that, the three-dimensional point cloud data can be plane detected. For example, the three-dimensional point cloud data can be plane detected by methods such as RANSAC (Random Sample Consensus) to obtain three-dimensional plane parameters, which can be used to represent the detected plane. After that, the perspective transformation matrix can be calculated based on the three-dimensional plane parameters. For example, the perspective transformation matrix can be calculated by the four-point method.
[0065] Step S320, performing perspective transformation on the color image through a perspective transformation matrix to obtain a perspective transformed image.
[0066] After obtaining the perspective transformation matrix, for each pixel in the color image, the position coordinates of the pixel are multiplied by the perspective transformation matrix to obtain the position coordinates after perspective transformation. The above process is performed for each pixel to obtain a perspective transformation image.
[0067] In an embodiment of the present disclosure, the image boundary information of the target object can be more accurately extracted from the perspective transformed image. In one implementation of the present disclosure, line segment detection can be performed on the perspective transformed image, for example, line segment detection can be performed on the perspective transformed image by means of Hough transform or the like. Among them, Hough transform can not only identify straight lines in an image, but also any other shapes, such as circles, ellipses, etc. When line segment information is included in the perspective transformed image, the image boundary information of the target object can be determined based on the line segment information.
[0068] Specifically, the present disclosure can also perform validity judgment and classification based on the length, angle and other information of the line segment. For example, a shorter line segment can be classified as an invalid line segment, and a longer line segment can be classified as a valid line segment, etc. The validity judgment can delete the invalid line segment information and retain the valid line segment information. Classification refers to dividing the line segment into different categories, for example, the line segment can be divided into horizontal line segments, vertical line segments, etc. The image boundary information of the target object finally extracted can contain multiple line segment information.
[0069] In another case, if the perspective transformed image does not contain line segment information, the image boundary information of the target object cannot be obtained by the above line segment detection method. At this time, the perspective transformed image can be processed based on the region detection algorithm to obtain the main region image. Specifically, for the perspective transformed image, the saliency distribution can be calculated first, and then the range box can be calculated. The region detection algorithm can adopt an attention region detection algorithm, a main region detection algorithm, or a method based on saliency region detection, among which the methods based on saliency region detection include: DeepGaze, DenseGaze, FastGaze, etc.
[0070] See also Figure 4 , Figure 4 A schematic diagram of region detection is shown. It can be seen that through region detection, a main region image can be obtained. The main region image contains a range frame, and the range frame contains the target object. Afterwards, the image boundary information of the target object can be determined based on the main region image. It can be understood that Figure 4 The range box in can be the image boundary of the target object.
[0071] It should be noted that when extracting the image boundary information of the target object in the perspective transformation image, the region detection algorithm can be used if no line segment information is detected after line segment detection, or the region detection algorithm can be used directly, which is not limited in the present disclosure. Moreover, when using the region detection algorithm, the perspective transformation image can be processed separately, or the perspective transformation image and the depth image can be processed separately and fused after processing, which can improve the accuracy of determining the image boundary information.
[0072] In step S230, point cloud boundary information of the target object is determined according to the point cloud data generated based on the depth image.
[0073] In the disclosed embodiment, in addition to determining image boundary information based on a color image, point cloud boundary information may also be determined based on a depth image, thereby determining the boundary information of the target object from more levels to improve the accuracy of the boundary information ultimately determined.
[0074] Specifically, the process of generating point cloud data according to the depth image can refer to the description in step S310. The process of determining the point cloud boundary information of the target object according to the point cloud data can refer to Figure 5 , Figure 5 A flow chart of generating point cloud boundary information in an embodiment of the present disclosure is shown, which may include the following steps:
[0075] Step S510, after performing plane detection on the point cloud data, plane point cloud data is obtained.
[0076] As mentioned above, after plane detection is performed on the point cloud data, the three-dimensional plane parameters can be obtained. According to the three-dimensional plane parameters, the point cloud belonging to the plane can be extracted to obtain plane point cloud data. It should be noted that the point cloud data belonging to the plane can be data on the plane, or point cloud data on the plane, and point cloud data whose distance from the plane is less than a distance threshold. The distance threshold can be set according to actual conditions. The smaller the distance threshold, the higher the accuracy of image cropping.
[0077] Step S520: determine the bounding box of the planar point cloud data.
[0078] Specifically, assuming that the detected plane is the target plane, a rectangular coordinate system can be established with the normal of the target plane and the two tangents of the target plane as orthogonal bases, and the bounding box of the plane point cloud data can be calculated in this coordinate system. The bounding box is an algorithm for solving the optimal bounding space of a discrete point set. A geometric body with a slightly larger volume and simpler characteristics (called a bounding box) can be used to approximately replace complex geometric objects. Bounding boxes include: AABB bounding boxes, bounding spheres, directional bounding boxes OBBs, and fixed-direction convex hulls.
[0079] The AABB bounding box is the smallest hexahedron that contains the target object and whose sides are parallel to the coordinate axes. When the present disclosure uses the AABB bounding box, the process of calculating the bounding box is to calculate the maximum and minimum values along each axis of the coordinate system.
[0080] Step S530 , mapping the bounding box to the perspective transformed image to obtain point cloud boundary information of the target object.
[0081] In the disclosed embodiment, after the bounding box is obtained, the three-dimensional range (maximum value and minimum value) of the bounding box is directly projected onto the perspective transformation image to obtain the point cloud boundary information.
[0082] It should be noted that after obtaining the plane point cloud data, before executing step S520, the plane point cloud data can be filtered. By filtering, the noise point cloud data can be filtered out, so that the target plane point cloud data can be obtained. In one implementation of the present disclosure, the plane point cloud data can be distance filtered, for example, by counting the density of the point cloud, the point cloud with lower density is filtered out as noise. That is, the noise point cloud in the plane point cloud data that is mistakenly judged as belonging to the target plane is filtered out by distance filtering.
[0083] In another implementation of the present disclosure, the plane point cloud data may also be subjected to clustering filtering processing, for example, by clustering the plane point cloud data, the class with the largest number of plane point cloud data may be extracted. In this way, non-subject point clouds that belong to other objects but are calculated because they intersect with the target plane may be filtered out. Among them, the clustering algorithm may be a K-means clustering algorithm, a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm, etc. Of course, the plane point cloud data may also be subjected to distance filtering processing and clustering filtering processing, which may further improve the accuracy of the target plane point cloud data compared to performing distance filtering processing or clustering filtering processing alone.
[0084] Accordingly, after filtering, the bounding box of the target plane point cloud data can be determined, thereby improving the accuracy of the bounding box, and further improving the accuracy of the point cloud boundary information.
[0085] In step S240 , target boundary information is obtained based on the image boundary information and the point cloud boundary information.
[0086] In the disclosed embodiment, image boundary information is information used to describe the image boundary, and different image boundary information corresponds to different image boundaries. Point cloud boundary information is information used to describe the point cloud boundary, and different point cloud boundary information corresponds to different point cloud boundaries. By fusing the image boundary information calculated based on the color image with the point cloud boundary information calculated based on the point cloud data, more accurate target boundary information can be obtained.
[0087] Among them, the fusion method can be to use the point cloud boundary as the initial value and select the boundary closest to the outside of the point cloud boundary within a certain range from the image boundary; or to use the point cloud boundary as the initial value and select the boundary closest to the inside of the point cloud boundary within a certain range from the image boundary.
[0088] See also Figure 6 , Figure 6 A schematic diagram of the target object and the point cloud boundary in the embodiment of the present disclosure is shown. Due to the influence of sensor performance, sensor accuracy, reflection characteristics of the object, etc., there are a large number of noise point clouds in the point cloud data, so the detection algorithm based on the bounding box cannot give a very accurate object range. In other words, due to the influence of the noise point cloud, the boundary of the target object cannot be accurately determined based on the point cloud data. It can be seen that the point cloud boundary is quite different from the actual boundary of the target object.
[0089] According to the method of the embodiment of the present disclosure, for example, by detecting line segments, the following can be obtained: Figure 7The detection result shown in the figure shows that multiple straight lines can be detected. Since the detected straight line is on the inner side of the point cloud boundary, when the detected straight line and the point cloud boundary are fused, the point cloud boundary can be used as the initial value, and the boundary closest to the inner side of the point cloud boundary can be selected from multiple straight lines. The target boundary after fusion can be seen in Figure 8 .
[0090] In addition, the point cloud boundary can be used as the initial value, and the line segment closest to the point cloud boundary and with an angle less than the angle threshold can be selected from the image boundary, that is, the target line segment, and the line segment information corresponding to the target line segment can be used as the target boundary information. The angle threshold can be set according to the actual situation, for example, it can be 10°, 15°, etc. Since the image boundary can include multiple directions, for example, for a rectangular boundary, it includes four different directions. Therefore, the line segment closest to the point cloud boundary and with an angle less than the angle threshold refers to the line segment closest to the point cloud boundary in each direction and with an angle less than the angle threshold. Correspondingly, the line segment finally selected is also a line segment in multiple directions, and the line segments in multiple directions constitute the target boundary.
[0091] In an implementation of the present disclosure, after obtaining the target boundary information, the color image may not be cropped, but the target boundary information may be displayed in the color image for reference by the user, so that the user can manually crop the color image based on the target boundary information. That is, the target boundary information is displayed in the color image to assist the user in manual cropping, and the user can directly crop according to the target boundary information, or of course, may not crop according to the target boundary information.
[0092] In step S250, the color image is cropped according to the target boundary information to obtain a cropped image.
[0093] It should be noted that after obtaining the target boundary information, the cropping can also be performed automatically to obtain a cropped image. After obtaining the cropped image, the cropped image can also be displayed to the user. Fig. 9 As shown, when displaying the cropped image, it can also be displayed in view mode, and the user can click "manual crop" to enter the edit mode, and the user can further manually crop the image.
[0094] In the edit mode, users can adjust the cropping frame, rotate the image, translate the image, stretch the image, and perform perspective transformation on the image. Users can drag the four corners of the cropping frame to adjust the size of the cropping frame, or drag the border of the cropping frame to adjust the size of the cropping frame. Users can rotate the image by dragging, and the rotation angle can be displayed to assist the user in rotation. Users can also translate the image by dragging, stretch the image by two-finger touch, and semi-automatically stretch it by using the slider.
[0095] In the disclosed embodiment, the user can click "Perspective Transformation" to enter the perspective transformation mode. Fig.10 As shown, by giving four corner points, the user can manually drag each corner point to achieve perspective transformation.
[0096] When the user performs an editing operation on the cropped image, the electronic device edits the cropped image in response to the user's editing operation on the cropped image, thereby obtaining an image that better meets the user's needs.
[0097] The image cropping method of the embodiment of the present disclosure obtains plane point cloud data by plane detection on point cloud data, performs perspective transformation on color image based on plane point cloud data, obtains perspective transformation image, so that perspective transformation does not depend on image information. And calculates bounding box based on three-dimensional information of plane point cloud data, and obtains cropping range information according to the projection position of bounding box in perspective transformation image. By utilizing line segment information in color image, the result of point cloud detection can be further assisted to improve the accuracy of automatic cropping. Or by performing region detection on color image, the main body range can be obtained in the absence of line segment information, and the range calculation of automatic cropping can be assisted. Therefore, the present disclosure can accurately obtain the boundary range of target object in the case of no obvious lines or any direction information of target object, or even in the case of interference information (such as line segment stripes of object, etc.), and perform automatic cropping, which can improve the accuracy of cropping. In addition, the user can also perform manual cropping further, so as to obtain an image that better meets the needs of the user.
[0098] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0099] Corresponding to the above method embodiment, the present disclosure also provides an image cropping device, see Fig.11 , the image cropping device 1100 comprises:
[0100] An image acquisition module 1110 is used to acquire a color image and a depth image of a target object;
[0101] An image boundary information determination module 1120 is used to perform perspective transformation on the color image to obtain a perspective transformed image, and to extract image boundary information of a target object in the perspective transformed image;
[0102] The point cloud boundary information determination module 1130 is used to determine the point cloud boundary information of the target object according to the point cloud data generated based on the depth image;
[0103] The target boundary information determination module 1140 is used to obtain the target boundary information based on the image boundary information and the point cloud boundary information;
[0104] The image cropping module 1150 is used to crop the color image according to the target boundary information to obtain a cropped image.
[0105] In an exemplary embodiment of the present disclosure, an image boundary information determination module includes:
[0106] A line segment detection unit, used for performing line segment detection on the perspective transformed image;
[0107] The first image boundary information determining unit is used to determine the image boundary information of the target object according to the line segment information when the perspective transformed image contains line segment information.
[0108] In an exemplary embodiment of the present disclosure, the image boundary information determination module further includes:
[0109] The second image boundary information determination unit is used to process the perspective transformed image based on the region detection algorithm to obtain a main region image when the perspective transformed image does not contain line segment information; and determine the image boundary information of the target object according to the main region image.
[0110] In an exemplary embodiment of the present disclosure, the image boundary information determination module further includes:
[0111] The perspective transformation unit is used to perform plane detection on the point cloud data generated based on the depth image, determine the perspective transformation matrix according to the detected plane, and perform perspective transformation on the color image through the perspective transformation matrix to obtain a perspective transformed image.
[0112] In an exemplary embodiment of the present disclosure, the point cloud boundary information determination module includes:
[0113] A plane point cloud data acquisition unit, used to obtain plane point cloud data after performing plane detection on the point cloud data generated based on the depth image;
[0114] A bounding box determination unit, used to determine a bounding box of the planar point cloud data;
[0115] The mapping unit is used to map the bounding box to the perspective transformed image to obtain the point cloud boundary information of the target object.
[0116] In an exemplary embodiment of the present disclosure, the image cropping device further includes:
[0117] A plane point cloud data filtering module is used to filter the plane point cloud data after obtaining the plane point cloud data to obtain target plane point cloud data;
[0118] The bounding box determination unit is specifically used to determine the bounding box of the target plane point cloud data.
[0119] In an exemplary embodiment of the present disclosure, the plane point cloud data filtering module is specifically used to perform distance filtering processing on the plane point cloud data; or to perform clustering filtering processing on the plane point cloud data; or to perform distance filtering processing and clustering filtering processing on the plane point cloud data.
[0120] In an exemplary embodiment of the present disclosure, a target boundary information determination module includes:
[0121] A target line segment selection unit, used to select a target line segment from the line segments represented by the image boundary information, wherein the target line segment is closest to the line segment represented by the point cloud boundary information and the included angle is less than an angle threshold;
[0122] The target boundary information determining unit is used to use the line segment information corresponding to the target line segment as the target boundary information.
[0123] In an exemplary embodiment of the present disclosure, the image cropping device further includes:
[0124] The image display module is used to display the target boundary information in the color image after obtaining the target boundary information, so that the user can crop the color image based on the target boundary information.
[0125] In an exemplary embodiment of the present disclosure, the image cropping device further includes:
[0126] The image display module is further used to display the cropped image to the user after obtaining the cropped image;
[0127] The image editing module is used to edit the cropped image in response to the user's editing operation on the cropped image.
[0128] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a computer, the computer executes any one of the above-mentioned methods.
[0129] It should be noted that the computer-readable storage medium shown in the present disclosure may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency, etc., or any suitable combination of the above.
[0130] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.
[0131] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. An image cropping method, comprising: Acquire color images and depth images of the target object; Performing perspective transformation on the color image to obtain a perspective transformed image, and extracting image boundary information of the target object in the perspective transformed image; Determining point cloud boundary information of the target object according to the point cloud data generated based on the depth image; Obtaining target boundary information based on the image boundary information and the point cloud boundary information; The color image is cropped according to the target boundary information to obtain a cropped image, The step of performing perspective transformation on the color image to obtain a perspective transformed image includes: Performing plane detection on the point cloud data generated based on the depth image, and determining a perspective transformation matrix according to the detected plane; The color image is perspective transformed by using the perspective transformation matrix to obtain a perspective transformed image.
2. The method according to claim 1, wherein: Extracting image boundary information of the target object in the perspective transformed image includes: Performing line segment detection on the perspective transformed image; When the perspective transformed image includes line segment information, image boundary information of the target object is determined according to the line segment information.
3. The method according to claim 2, wherein: After performing line segment detection on the perspective transformed image, the method further includes: When the perspective transformed image does not contain line segment information, the perspective transformed image is processed based on a region detection algorithm to obtain a main body region image; Image boundary information of the target object is determined according to the main body area image.
4. The method according to claim 1, wherein: Determining point cloud boundary information of the target object according to point cloud data generated based on the depth image includes: After performing plane detection on the point cloud data, plane point cloud data is obtained; Determine a bounding box of the planar point cloud data; The bounding box is mapped to the perspective transformed image to obtain point cloud boundary information of the target object.
5. The method according to claim 4, wherein: Also includes: After obtaining the plane point cloud data, filtering the plane point cloud data to obtain target plane point cloud data; Determining the bounding box of the planar point cloud data specifically includes: Determine a bounding box of the target plane point cloud data.
6. The method according to claim 5, wherein: The filtering process of the plane point cloud data comprises: Performing distance filtering on the plane point cloud data; or Performing clustering filtering on the plane point cloud data; or The plane point cloud data is subjected to distance filtering and clustering filtering.
7. The method according to claim 3, wherein: The obtaining target boundary information based on the image boundary information and the point cloud boundary information includes: Selecting a target line segment from the line segments represented by the image boundary information, wherein the target line segment is closest to the line segment represented by the point cloud boundary information and the included angle is less than an angle threshold; The line segment information corresponding to the target line segment is used as the target boundary information.
8. The method according to claim 1, wherein: Also includes: After the target boundary information is obtained, the target boundary information is displayed in the color image, so that the user can crop the color image based on the target boundary information.
9. The method according to claim 1, wherein: Also includes: After obtaining the cropped image, displaying the cropped image to the user; In response to the user's editing operation on the cropped image, the cropped image is edited.
10. An image cropping device, comprising: An image acquisition module, used to acquire a color image and a depth image of a target object; An image boundary information determination module, used to perform perspective transformation on the color image to obtain a perspective transformed image, and to extract image boundary information of the target object in the perspective transformed image; A point cloud boundary information determination module, used to determine the point cloud boundary information of the target object according to the point cloud data generated based on the depth image; A target boundary information determination module, used to obtain target boundary information based on the image boundary information and the point cloud boundary information; An image cropping module is used to crop the color image according to the target boundary information to obtain a cropped image. Wherein, the image boundary information determination module is further used for: Performing plane detection on the point cloud data generated based on the depth image, and determining a perspective transformation matrix according to the detected plane; The color image is perspective transformed by using the perspective transformation matrix to obtain a perspective transformed image.
11. An electronic device, comprising: processor; as well as a memory configured to store executable instructions of the processor; The processor is configured to perform the method according to any one of claims 1 to 9 by executing the executable instructions.
12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Method and system for extracting characteristic information of planar geometrical shape with perspective deformation
CN103177253A
3D object modeling method based on depth sensor
CN109087388A
Stereoscopic shelf visual checking method and system based on camera image splicing
CN110120010A
A method and apparatus for detecting keypoints
CN110427917A
Method for reducing distortion after perspective transformation of image
CN110796602A
