Projection boundary determination method and device, storage medium and method

Abstract

The embodiment of the invention provides a projection boundary determination method and device, a storage medium and a method, and the method comprises the steps: obtaining a target depth image of a target object, determining a vanishing point of the target depth image in a target direction, determining a plurality of target straight lines passing through the vanishing point in the target depth image, for each target straight line, acquiring a plurality of target pixel points corresponding to a first component located on the target straight line in the target depth image, and obtaining a second component located on the target straight line according to the plurality of target pixel points; and determining projection points of a three-dimensional straight line corresponding to the target straight line in the world coordinate system on the target plane along the target direction, and determining a two-dimensional projection boundary of the target object on the target plane according to the projection points corresponding to the plurality of target straight lines respectively. According to the method provided by the embodiment of the invention, the algorithm is simple and rapid, complex model training is not needed, and the cost is relatively low.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of extended reality (XR), and in particular, to a method, device, storage medium, and system for determining a projection boundary. Background Art

[0002] In the application process of XR devices, in order to improve the gaming experience, it is usually necessary to reconstruct the user's environment.

[0003] In related technologies, a deep learning model is usually used to generate a house outline based on the captured images.

[0004] However, the inventors found that there are at least the following technical problems in the related technologies: In the above method, the training of the deep learning model is relatively complex and the input cost is relatively high. Summary of the Invention

[0005] Embodiments of the present disclosure provide a method, device, storage medium, and system for determining a projection boundary, which can simplify operations and reduce costs.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for determining a projection boundary, including:

[0007] Obtaining a target depth image of a target object; the target object includes a first component for forming a three-dimensional contour of the target object; a surface of the first component includes a straight line in a target direction in a world coordinate system;

[0008] Determining a vanishing point of the target depth image in the target direction, and determining a plurality of target straight lines passing through the vanishing point in the target depth image;

[0009] For each target straight line, obtaining a plurality of target pixel points corresponding to the first component located on the target straight line in the target depth image; and determining a projection point of a three-dimensional straight line corresponding to the target straight line in the world coordinate system on a target plane along the target direction according to the plurality of target pixel points;

[0010] Determining a two-dimensional projection boundary of the target object on the target plane according to the projection points corresponding to the plurality of target straight lines respectively.

[0011] In a second aspect, an embodiment of the present disclosure provides a device for determining a projection boundary, including:

[0012] An obtaining module, configured to obtain a target depth image of a target object; the target object includes a first component for forming a three-dimensional contour of the target object; a surface of the first component includes a straight line in a target direction in a world coordinate system;

[0013] A determination module, configured to determine a vanishing point of the target depth image in the target direction, and determine a plurality of target lines passing through the vanishing point in the target depth image;

[0014] A projection module, configured to, for each target line, obtain a plurality of target pixel points corresponding to the first component located on the target line in the target depth image; and determine a projection point of a three-dimensional line corresponding to the target line in the world coordinate system on a target plane along the target direction according to the plurality of target pixel points;

[0015] The determination module is further configured to determine a two-dimensional projection boundary of the target object on the target plane according to the projection points respectively corresponding to the plurality of target lines.

[0016] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0017] The memory stores computer-executable instructions;

[0018] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the projection boundary determination method as described in the first aspect and various possible designs of the first aspect above.

[0019] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the processor executes the computer-executable instructions, the projection boundary determination method as described in the first aspect and various possible designs of the first aspect above is implemented.

[0020] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by the processor, the projection boundary determination method as described in the first aspect and various possible designs of the first aspect above is implemented.

[0021] The projection boundary determination method, device, storage medium, and method provided in this embodiment include: obtaining a target depth image of a target object, where the target object includes a first component for forming the three-dimensional contour of the target object, and the surface of the first component includes a straight line in the target direction in the world coordinate system; determining the vanishing point of the target depth image in the target direction; determining a plurality of target straight lines passing through the vanishing point in the target depth image; for each target straight line, obtaining a plurality of target pixel points corresponding to the first component located on the target straight line in the target depth image; determining the projection points of the three-dimensional straight line corresponding to the target straight line in the world coordinate system on the target plane along the target direction according to the plurality of target pixel points; and determining the two-dimensional projection boundary of the target object on the target plane according to the projection points corresponding to the plurality of target straight lines. The method provided in the embodiments of the present disclosure determines the vanishing point of the target depth image in the projection direction, and based on multiple straight lines passing through the vanishing point, filters out multiple groups of target pixel points from the depth image. Furthermore, the projection points corresponding to different straight lines are determined, so as to determine the projection boundary of the target object based on multiple projection points. The algorithm is simple and fast, does not require complex model training, and has a low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following briefly introduces the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1a It is a schematic diagram of the application scenario of the projection boundary determination method provided in the embodiments of the present disclosure;

[0024] Figure 1b It is a schematic diagram of the structure of the XR device provided in the embodiments of the present disclosure;

[0025] Figure 1c It is a schematic diagram of the projection boundary of the target object provided in the embodiments of the present disclosure;

[0026] Figure 2 It is a schematic diagram of the process flow of the projection boundary determination method provided in the embodiments of the present disclosure I;

[0027] Figure 3 It is a schematic diagram of the process flow of the projection boundary determination method provided in the embodiments of the present disclosure Figure 2 ;

[0028] Figure 4 It is a block diagram of the structure of the projection boundary determination device provided in the embodiments of the present disclosure;

[0029] Figure 5 Schematic diagram of the hardware structure of the projection boundary determination device provided by an embodiment of the present disclosure. Detailed implementation manners

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are only a part rather than all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the protection scope of the present disclosure.

[0031] With the rapid development of extended reality technology (XR) (such as mixed reality (MR), augmented reality (AR), and virtual reality (VR)), reconstructing the scene where the user is located (such as the room where the user is located) helps to better perform games and XR interactions.

[0032] In the related art, the construction and training of a deep learning model are usually adopted to realize the reconstruction of a home scene.

[0033] However, in the above method, the deep learning model is relatively complex, its training period is long, and a relatively high cost is required.

[0034] To solve the above technical problems, the inventors of the present disclosure have found through research that XR scenes are usually home scenes such as bedrooms, living rooms, and kitchens, and the basic data for home scene reconstruction is to obtain a room contour model. To obtain the room contour, the projection boundary of a wall or the like on the ground can be determined first. To obtain the projection boundary, the vanishing point of a target depth image in the projection direction can be determined, and multiple groups of target pixel points can be filtered out from the depth image based on multiple straight lines passing through the vanishing point. Furthermore, the projection points corresponding to different straight lines can be determined, and thus the projection boundary of the target object can be determined based on multiple projection points. Based on this, an embodiment of the present application provides a projection boundary determination method.

[0035] Figure 1a Schematic diagram of the application scenario of the projection boundary determination method provided by an embodiment of the present disclosure Figure 1b Schematic diagram of the structure of the XR device provided by an embodiment of the present disclosure Figure 1c Schematic diagram of the projection boundary of the target object provided by an embodiment of the present disclosure. As Figure 1bAs shown, the XR device includes a depth camera and a controller. Among them, the depth camera is used to collect the depth image of the target object (such as the room where it is located). The controller is used to determine the projection boundary of the component for forming the three-dimensional contour of the target object based on the depth image. Optionally, the XR device may further include a grayscale camera, which is used to collect the grayscale image of the target object. Correspondingly, the controller can identify the pixel points corresponding to the target components (such as walls, doors, and windows, etc.) of the target object based on the grayscale image, and then determine the pixel points corresponding to the target components in the depth image based on the correspondence between the grayscale image and the depth image.

[0036] In the specific implementation process, as Figure 1a shown, in the room, the user wears the XR device, and the depth camera of the XR device is used to collect the depth image of the room where the user is located to obtain the target depth image of the target object; among them, the target object includes a first component for forming the three-dimensional contour of the target object, and the surface of the first component includes a straight line in the target direction in the world coordinate system. Determine the vanishing point of the target depth image in the target direction, and determine multiple target straight lines passing through the vanishing point in the target depth image. For each target straight line, obtain multiple target pixel points corresponding to the first component located on the target straight line in the target depth image. According to the multiple target pixel points, determine the projection points of the three-dimensional straight line corresponding to the target straight line in the world coordinate system on the target plane along the target direction. According to the projection points corresponding to the multiple target straight lines respectively, determine the two-dimensional projection boundary of the target object on the target plane as shown in Figure 1c shown. The projection boundary determination method provided by the embodiments of the present disclosure determines the vanishing point of the target depth image in the projection direction, and filters out multiple groups of target pixel points from the depth image based on multiple straight lines passing through the vanishing point. Furthermore, the projection points corresponding to different straight lines are determined, so as to determine the projection boundary of the target object based on multiple projection points. The algorithm is simple and fast, does not require complex model training, and has a low cost.

[0037] Optionally, when obtaining multiple target pixel points corresponding to the first component located on the target straight line in the target depth image for each target straight line, the pixel points corresponding to the first component (such as walls, doors, and windows, etc.) of the target object can be identified based on the grayscale image, and then based on the correspondence between the grayscale image and the depth image, for each target straight line, multiple target pixel points corresponding to the first component located on the target straight line are obtained in the target depth image.

[0038] It should be noted that the scene schematic diagram shown in FIG. 1 is only an example. The projection boundary determination method and the scene described in the embodiments of the present disclosure are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those of ordinary skill in the art will know that with the evolution of the system and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0039] Reference Figure 2 , Figure 2 FIG. 1 is a schematic flow chart of the projection boundary determination method provided by the embodiments of the present disclosure. The method of the embodiments of the present disclosure can be applied to a terminal device (such as an XR device) or a server. The projection boundary determination method includes:

[0040] 201. Obtain a target depth image of the target object; the target object includes a first component for forming the three-dimensional contour of the target object; the surface of the first component includes a straight line in the target direction in the world coordinate system.

[0041] Specifically, inside the target object, for example, inside a room, a depth camera (such as a depth camera set on an XR device) can be used to photograph components such as walls, curtains, doors, and windows in the room to obtain a target depth image. The measured value of each pixel point of the target depth image is the distance from the pixel point to the corresponding measurement point. In addition, depth image shooting can also be performed outside the target object. For example, shooting can be performed around a table, a computer, or a vase. It can be specifically set according to actual needs, and this embodiment does not limit this.

[0042] In the embodiments of the present disclosure, the target object can be a three-dimensional object that needs to be contour-reconstructed, such as a bedroom, a living room, etc. A room. Correspondingly, the first component can be a component such as a curtain, a door, a window, or a wall that can form the three-dimensional contour of the target object.

[0043] Exemplarily, the target direction can be the direction of gravitational acceleration, and an inertial sensor (Inertial Measurement Unit, IMU) set in the terminal device (such as an XR device) can be used to determine it.

[0044] 202. Determine the vanishing point of the target depth image in the target direction, and determine a plurality of target straight lines passing through the vanishing point in the target depth image.

[0045] Specifically, the target depth image can be one frame or multiple frames. After obtaining the target depth image, the vanishing point in the target direction can be determined. Since the parallel lines in the target direction will all pass through the vanishing point, conversely, the lines passing through the vanishing point all belong to the parallel lines in the target direction. Therefore, multiple target lines can be determined through this vanishing point, and these multiple target lines are all parallel lines in the target direction.

[0046] In an embodiment of the present disclosure, determining the vanishing point of the target depth image in the target direction may include: determining the vanishing point direction in the depth camera coordinate system according to the target direction and the attitude of the world coordinate system in the depth camera coordinate system; determining the corresponding normalized planar points according to the vanishing point direction; and determining the pixel coordinates of the vanishing point in the pixel coordinate system of the depth camera according to the normalized planar points and the internal parameters of the depth camera.

[0047] Exemplarily, assuming the target direction is the vertical direction, the calculation of the vanishing point can be performed according to the following expression.

[0048] d w =[0, 0, 1] (1)

[0049] d c =R cw *d w (2)

[0050]

[0051]

[0052] Wherein, d w represents the direction of the vanishing point, that is, the vertical direction; R cw is the attitude of the world coordinate system in the camera coordinate system, d c is the vanishing point direction in the camera coordinate system, and are the normalized planar points, and f x c x f y c y are the internal parameters of the camera.

[0053] In an embodiment of the present disclosure, determining multiple target lines passing through the vanishing point in the target depth image may include: starting from the vanishing point, constructing multiple rays so that the multiple rays pass through the pixel points in the target depth image; and determining the lines where the multiple rays are located as the multiple target lines.

[0054] Specifically, assuming that the vanishing point is directly below the target depth image, a fan-shaped region can be determined starting from the vanishing point and extending upward. The angle of the fan-shaped region is determined such that the target depth image is exactly included within the fan-shaped region. Subsequently, multiple rays can be constructed within the fan-shaped region. The multiple rays can be evenly distributed, meaning that the angles between adjacent rays are equal, or they can be unevenly distributed.

[0055] In one embodiment of the present disclosure, considering that if the number of target lines is too small, it will affect the calculation accuracy, and if it is too large, it will increase the computational load. After multiple experiments and studies, it is found that the number of target lines can be determined based on the pixel size of the target depth image. In this embodiment, determining multiple target lines passing through the vanishing point in the target depth image may include: determining the number of target lines passing through the vanishing point in the target depth image according to the pixel size of the target depth image, where the pixel size includes the number of pixels of the long side and / or the short side of the target depth image; determining multiple target lines passing through the vanishing point in the target depth image according to the number of target lines. Optionally, before determining multiple target lines passing through the vanishing point in the target depth image, it may further include: obtaining the pixel size of the target depth image; the pixel size includes the number of pixels of the long side and the short side; if the vanishing point is located outside the long side, multiplying the number of pixels of the long side by a preset ratio to obtain a preset number of target lines; if the vanishing point is located outside the short side, multiplying the number of pixels of the short side by a preset ratio to obtain a preset number of target lines; determining the preset number of target lines passing through the vanishing point in the target depth image may include: determining the preset number of target lines passing through the vanishing point in the target depth image; the angles between adjacent lines among the multiple target lines are equal.

[0056] Exemplarily, the preset ratio can be a value between one-eighth and one-fifteenth, such as one-tenth.

[0057] 203. For each target line, obtain multiple target pixel points corresponding to the first component located on the target line in the target depth image; according to the multiple target pixel points, determine the projection point of the three-dimensional line corresponding to the target line in the world coordinate system along the target direction on the target plane.

[0058] Specifically, each pixel point in the target depth image can be classified in advance, with each category representing a component. After determining multiple target lines, among the pixel points passed through by the target lines in the target depth image, the pixel points belonging to the first component are selected as target pixel points. Since the pixel points passed through by the target line are the lines in the target direction, the coordinate values of the target pixel points belonging to the same target line only vary in the target direction, and the coordinate values in the direction perpendicular to the target direction are theoretically equal. The coordinate of the projection point in the target direction is the coordinate value in the direction perpendicular to the target direction. That is to say, after this processing, multiple target pixel points passing through the target line are downsampled to a single projection point, and based on the theory that the above-mentioned coordinate values are theoretically equal, outliers can be removed during the determination of the projection point, which is equivalent to removing noise and improves the calculation accuracy.

[0059] Exemplarily, there can be multiple ways to classify each pixel point in the target depth image. In one implementable way, a color image or a grayscale image of the surrounding environment can be acquired to identify the pixel points belonging to different components in the image using a deep learning model and mark the pixel points accordingly. For example, the pixel points corresponding to the wall are marked as category a, the pixel points corresponding to the ceiling are marked as category b, and so on. Then, based on the correspondence between the grayscale image and the depth image, the pixel points in the depth image are classified and marked. Specifically, reference can be made to the introduction of the following Figure 3 illustrated embodiments. In another implementable way, each pixel point in the depth image can be classified and marked manually.

[0060] In an embodiment of the present disclosure, determining the projection point of the three-dimensional line corresponding to the target line in the world coordinate system on the target plane along the target direction according to the multiple target pixel points may include: determining the three-dimensional points respectively corresponding to the multiple target pixel points in the world coordinate system; determining the projection coordinates on the target plane along the target direction respectively corresponding to the multiple three-dimensional points; and determining the mean value of the multiple projection coordinates as the coordinate of the projection point of the target line on the target plane.

[0061] Specifically, after determining multiple target pixel points of the first component corresponding to the target line, since the coordinate values of the multiple target pixel points in the direction perpendicular to the target direction are equal, therefore, based on this, the mean value of the coordinate values of the multiple target pixel points can be calculated and determined as the two-dimensional coordinate of the projection point on the target plane. Of course, if there is an angle between the target plane and the target direction, the rule for determining the projection coordinate value can also be based on this angle, and then a result close to the theoretical value can be calculated. The above calculation method of the projection point coordinate can achieve both the denoising effect and the downsampling effect.

[0062] 204. Determine the two-dimensional projection boundary of the target object on the target plane according to the projection points corresponding to the multiple target lines respectively.

[0063] Specifically, after determining the multiple projection points of the multiple lines on the surface of the first component (such as the wall of a room) of the target object along the target direction (such as the vertical direction) on the target plane (such as the floor), the two-dimensional projection boundary can be determined based on the projection points (for example, the multiple projection points of the wall can be connected to obtain the projection boundary corresponding to the room).

[0064] In an embodiment of the present disclosure, in order to make the projection boundary more accurate and clear, a fitting method can be used to determine the projection boundary. Specifically, the step of determining the two-dimensional projection boundary of the target object on the target plane according to the projection points corresponding to the multiple target lines respectively may include: fitting the projection points corresponding to the multiple target lines respectively to obtain the two-dimensional projection boundary of the target object on the target plane.

[0065] As can be seen from the above description, the projection boundary determination method provided by the embodiments of the present disclosure determines the vanishing point of the target depth image in the projection direction, and based on multiple lines passing through the vanishing point, multiple groups of target pixel points are screened out from the depth image. Furthermore, the projection points corresponding to different lines are determined, and thus the projection boundary of the target object is determined based on the multiple projection points. The algorithm is simple and fast, without the need for complex model training, and the cost is relatively low.

[0066] Reference Figure 3 , Figure 3 is a schematic flowchart of the projection boundary determination method provided by the embodiments of the present disclosure. Figure 2 In the above embodiment, for example Figure 2 on the basis of the embodiment shown, an exemplary description of the application of the grayscale image and the determination of the height of the three-dimensional contour of the target object is given in this embodiment. The projection boundary determination method includes:

[0067] 301. Obtain the target depth image of the target object; the target object includes a first component for forming the three-dimensional contour of the target object; the surface of the first component includes lines in the target direction in the world coordinate system.

[0068] 302. Determine the vanishing point of the target depth image in the target direction, and determine multiple target lines passing through the vanishing point in the target depth image.

[0069] Steps 301 to 302 in this embodiment are similar to steps 201 to 202 in the above embodiment, and will not be elaborated here.

[0070] 303. Obtain the target grayscale image of the target object, and perform recognition on the target grayscale image to obtain a plurality of first pixel points corresponding to the first component in the target grayscale image.

[0071] Exemplarily, a grayscale camera (such as a grayscale camera set on an XR device) can be used to collect the grayscale image of the target object (such as the grayscale image in a room), and the grayscale image is input into a pre-trained deep learning model to obtain the recognition result of the grayscale image, that is, the construction category of each pixel point. Optionally, the training process of the deep learning model may include: using the method of manual annotation to annotate components such as walls, ceilings, floors, doors, and curtains in the grayscale image, and using a neural network (the network structure depends on the algorithm of the specific chip. If the computing power is sufficient, a larger network can be used to make the final effect more accurate) to train the annotation data. Among them, walls, doors, and curtains can be classified into category A, the ceiling into category B, the floor into category C, and others into category D.

[0072] 304. According to the pixel correspondence between the target depth image and the target grayscale image, determine second pixel points in the target depth image that respectively correspond to the plurality of first pixel points.

[0073] Specifically, the depth camera and the grayscale camera can be set on the terminal device (such as an XR device) with a fixed relative pose. Based on this relative pose, the pixel correspondence between the depth image obtained by the depth camera and the grayscale image obtained by the grayscale camera can be determined. Furthermore, based on this pixel correspondence and the recognition result of the pixel points of the grayscale image, the type of the component to which each pixel point in the depth image belongs can be determined, and then corresponding marking can be performed.

[0074] In an embodiment of the present disclosure, the step of determining the pixel correspondence between the target depth image and the grayscale image may include: determining the first coordinates of a plurality of third pixel points in the target depth image in the depth camera coordinate system respectively; according to the conversion relationship between the depth camera coordinate system and the world coordinate system and the first coordinates, determining the second coordinates of the plurality of three-dimensional points in the world coordinate system; according to the conversion relationship between the world coordinate system and the grayscale camera coordinate system and the second coordinates, determining the third coordinates of the plurality of three-dimensional points in the grayscale camera coordinate system; and according to the third coordinates, determining the pixel correspondence between the target depth image and the grayscale image.

[0075] Exemplarily, the XR device acquires grayscale images and depth images in a room. For the grayscale images, they are input into a deep learning model to obtain which class of component (such as class A, class B, class C, or class D) each pixel belongs to. Then, according to the pose transformation relationship between the depth camera and the grayscale camera, the pixels in each depth image are projected onto the grayscale image to obtain the class of each depth image pixel. The projection formula is as follows:

[0076]

[0077] P w =R wcg *P cg +t wcg (6)

[0078] P cd =R cdw *P w +t cdw (7)

[0079] P uvd =K d *P cd / d (8)

[0080] Wherein, P cg is the three-dimensional coordinate of the target 3D point corresponding to the target pixel point in the depth image acquired by the depth camera in the depth camera coordinate system, d1 is the measurement value of the depth camera corresponding to the target pixel point, K g is the internal parameter matrix of the depth camera, P uvg is the pixel coordinate of the target pixel point in the pixel coordinate system of the depth camera, R wcg and t wcg are the rotation and translation relationships between the depth camera coordinate system and the world coordinate system, P w is the world coordinate of the target 3D point, P cd is the coordinate of the target 3D point in the grayscale camera coordinate system (d is the z coordinate component of P cd ), R cdw and t cdw are the rotation and translation relationships between the grayscale camera coordinate system and the world coordinate system, K d is the internal parameter matrix of the grayscale camera, P uvd is the pixel coordinate of the target pixel point in the pixel coordinate system of the grayscale camera.

[0081] 305. For each target line, obtain multiple target pixel points corresponding to the first component located on the target line among the multiple second pixel points; determine the projection point of the three-dimensional line corresponding to the target line in the world coordinate system on the target plane along the target direction according to the multiple target pixel points.

[0082] 306. Determine the two-dimensional projection boundary of the target object on the target plane according to the projection points respectively corresponding to the multiple target straight lines.

[0083] In this embodiment, steps 305 to 306 are similar to steps 203 to 204 in the above embodiment, and will not be elaborated here.

[0084] 307. Determine the target height of the second component of the target object along the target direction. According to the target height and the two-dimensional projection boundary, determine the three-dimensional structure model of the target object. The plane where the surface of the second component is located is perpendicular to the target direction. The target object further includes a second component, and the plane where the surface of the second component is located is perpendicular to the target direction.

[0085] Specifically, after determining the projection boundary of the first component (such as the wall of a room) of the target object, it is also necessary to determine the target height (such as the distance between the ceiling and the floor) corresponding to the second component. Specifically, it can be calculated based on the coordinate difference in the target direction (such as the vertical direction).

[0086] In an embodiment of the present disclosure, the determining the target height of the second component along the target direction may include: obtaining a plurality of third pixel points corresponding to the second component in the target depth image; obtaining the first coordinate values of the plurality of third pixel points in the target direction; and determining the target height of the second component along the target direction according to the median of the plurality of first coordinate values.

[0087] Specifically, considering that there may be points with relatively large errors among the third pixel points corresponding to the second component, the median value can be used to determine the target height to achieve higher accuracy.

[0088] As can be seen from the above description, the projection boundary determination method provided by the embodiment of the present disclosure uses the recognition result of the grayscale image and the correspondence between the grayscale image and the depth image to mark the components to which each pixel point of the depth image belongs. Grayscale image recognition is a relatively mature algorithm, which can ensure a high accuracy rate and low implementation cost. In addition, by using the median value to determine the target height, the calculation accuracy can be improved.

[0089] Corresponding to the projection boundary determination method in the above embodiment, Figure 4 is a structural block diagram of the projection boundary determination device provided by the embodiment of the present disclosure. For the sake of convenience of description, only the parts related to the embodiment of the present disclosure are shown. Refer to Figure 4 The device includes: an acquisition module 401, a determination module 402, and a projection module 403.

[0090] Among them, an acquisition module 401 is configured to acquire a target depth image of a target object; the target object includes a first component for forming a three-dimensional contour of the target object; a surface of the first component includes a straight line in a target direction in a world coordinate system.

[0091] A determination module 402 is configured to determine a vanishing point of the target depth image in the target direction, and determine a plurality of target straight lines passing through the vanishing point in the target depth image.

[0092] A projection module 403 is configured to, for each target straight line, acquire a plurality of target pixel points corresponding to the first component located on the target straight line in the target depth image; and determine a projection point of a three-dimensional straight line corresponding to the target straight line in the world coordinate system on a target plane along the target direction according to the plurality of target pixel points.

[0093] The determination module 402 is further configured to determine a two-dimensional projection boundary of the target object on the target plane according to projection points respectively corresponding to the plurality of target straight lines.

[0094] In an embodiment of the present disclosure, the determination module 402 is specifically configured to: determine a vanishing point direction in the depth camera coordinate system according to the target direction and an attitude of the world coordinate system in the depth camera coordinate system; determine a corresponding normalized plane point according to the vanishing point direction; and determine pixel coordinates of the vanishing point in a pixel coordinate system of the depth camera according to the normalized plane point and internal parameters of the depth camera.

[0095] In an embodiment of the present disclosure, the determination module 402 is specifically configured to: construct a plurality of rays starting from the vanishing point so that the plurality of rays pass through pixel points in the target depth image; and determine straight lines where the plurality of rays are located as the plurality of target straight lines.

[0096] In an embodiment of the present disclosure, the projection module 403 is further configured to: acquire a pixel size of the target depth image; the pixel size includes the number of pixels of a long side and the number of pixels of a wide side; if the vanishing point is located outside the long side, multiply the number of pixels of the long side by a preset ratio to obtain a preset number of target straight lines; if the vanishing point is located outside the wide side, multiply the number of pixels of the wide side by a preset ratio to obtain a preset number of target straight lines; the projection module 403 is specifically configured to: determine the preset number of target straight lines passing through the vanishing point in the target depth image; an included angle between adjacent straight lines among the plurality of target straight lines is equal.

[0097] In an embodiment of the present disclosure, the projection module 403 is specifically configured to: determine the number of target straight lines passing through the vanishing point in the target depth image according to the pixel size of the target depth image, where the pixel size includes the number of pixels of the long side and / or the short side of the target depth image; determine a plurality of target straight lines passing through the vanishing point in the target depth image according to the number of the target straight lines.

[0098] In an embodiment of the present disclosure, the projection module 403 is further configured to: determine the first coordinates of the three-dimensional points corresponding to a plurality of third pixel points in the target depth image in the depth camera coordinate system; determine the second coordinates of the plurality of three-dimensional points in the world coordinate system according to the conversion relationship between the depth camera coordinate system and the world coordinate system and the first coordinates; determine the third coordinates of the plurality of three-dimensional points in the grayscale camera coordinate system according to the conversion relationship between the world coordinate system and the grayscale camera coordinate system and the second coordinates; determine the pixel correspondence between the target depth image and the grayscale image according to the third coordinates.

[0099] In an embodiment of the present disclosure, the projection module 403 is specifically configured to: determine the three-dimensional points corresponding to the plurality of target pixel points in the world coordinate system respectively; determine the projection coordinates of the plurality of three-dimensional points on the target plane along the target direction respectively; determine the mean value of the plurality of projection coordinates as the coordinates of the projection point of the target straight line on the target plane.

[0100] In an embodiment of the present disclosure, the determination module 402 is specifically configured to: fit the projection points corresponding to the plurality of target straight lines respectively to obtain the two-dimensional projection boundary of the target object on the target plane.

[0101] In an embodiment of the present disclosure, the target object further includes a second component, and the plane where the surface of the second component is located is perpendicular to the target direction; the determination module 402 is further configured to: determine the target height of the second component along the target direction; determine the three-dimensional structure model of the target object according to the target height and the two-dimensional projection boundary.

[0102] In an embodiment of the present disclosure, the determination module 402 is specifically configured to: obtain a plurality of third pixel points corresponding to the second component in the target depth image; obtain the first coordinate values of the plurality of third pixel points in the target direction; determine the target height of the second component along the target direction according to the median of the plurality of first coordinate values.

[0103] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments, and its implementation principles and technical effects are similar, which will not be elaborated here in this embodiment.

[0104] To implement the above embodiments, an embodiment of the present disclosure also provides an electronic device.

[0105] Referring to Figure 5 , which shows a schematic structural diagram of an electronic device 900 suitable for implementing the embodiments of the present disclosure. The electronic device 900 may be a terminal device or a server. Among them, the terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0106] As Figure 5 shown, the electronic device 900 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage device 908 into the random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. The input / output (I / O) interface 905 is also connected to the bus 904.

[0107] Generally, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 can allow the electronic device 900 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 5 the electronic device 900 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.

[0108] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program includes program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, the above-described functions defined in the method of the embodiment of the present disclosure are performed.

[0109] It should be noted that the above-mentioned computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. And in the present disclosure, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable signal medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0110] The above-mentioned computer-readable medium can be included in the above-mentioned electronic device; or it can exist separately without being assembled into the electronic device.

[0111] The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device is caused to execute the method shown in the above embodiment.

[0112] Computer program code for performing the operations of this disclosure may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, execute as a stand-alone software package, execute partially on the user's computer and partially on a remote computer, or execute entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0114] The units involved in the embodiments described in this disclosure may be implemented in software or in hardware. Among them, the name of the unit does not constitute a limitation to the unit itself in some cases. For example, the first acquisition unit may also be described as "the unit for acquiring at least two Internet protocol addresses".

[0115] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on a chip (SOC), complex programmable logic devices (CPLD), and so on.

[0116] In the context of the present disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, a technical solution formed by mutually replacing the above features with technical features having similar functions (but not limited to) disclosed in the present disclosure.

[0118] In addition, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although a number of specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments may also be implemented combinatorially in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0119] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for determining a projection boundary, characterized in that, Including: Obtaining a target depth image of a target object; the target object includes a first component for forming a three-dimensional contour of the target object; The surface of the first component includes a straight line in the target direction in the world coordinate system; Determining a vanishing point of the target depth image in the target direction, and determining a plurality of target straight lines passing through the vanishing point in the target depth image; For each target straight line, obtaining a plurality of target pixel points corresponding to the first component located on the target straight line in the target depth image; according to the plurality of target pixel points, determining a projection point of a three-dimensional straight line corresponding to the target straight line in the world coordinate system on a target plane along the target direction; Determining a two-dimensional projection boundary of the target object on the target plane according to the projection points respectively corresponding to the plurality of target straight lines.

2. The method according to claim 1, wherein The determining the vanishing point of the target depth image in the target direction includes: Determining a vanishing point direction in the depth camera coordinate system according to the target direction and the pose of the world coordinate system in the depth camera coordinate system; Determining a corresponding normalized planar point according to the vanishing point direction; Determining the pixel coordinates of the vanishing point in the pixel coordinate system of the depth camera according to the normalized planar point and the internal parameters of the depth camera.

3. The method according to claim 1, characterized in that, The determining a plurality of target straight lines passing through the vanishing point in the target depth image includes: Taking the vanishing point as a starting point, constructing a plurality of rays so that the plurality of rays pass through pixel points in the target depth image; Determining the straight lines where the plurality of rays are located as the plurality of target straight lines.

4. The method according to claim 1, characterized in that The determining a plurality of target straight lines passing through the vanishing point in the target depth image includes: Determining the number of target straight lines passing through the vanishing point in the target depth image according to the pixel size of the target depth image, the pixel size including the number of pixels of the long side and / or the number of pixels of the wide side of the target depth image; Determining a plurality of target straight lines passing through the vanishing point in the target depth image according to the number of the target straight lines.

5. The method according to claim 1, characterized in that, Before obtaining the plurality of target pixel points corresponding to the first component located on the target straight line in the target depth image, further including: Obtaining a target grayscale image of the target object; Identifying the target grayscale image to obtain a plurality of first pixel points corresponding to the first component in the target grayscale image; Determining second pixel points respectively corresponding to the plurality of first pixel points in the target depth image according to the pixel correspondence between the target depth image and the target grayscale image; The obtaining the plurality of target pixel points corresponding to the first component located on the target straight line in the target depth image includes: Obtaining the plurality of target pixel points corresponding to the first component located on the target straight line among the plurality of second pixel points.

6. The method according to claim 5, wherein Before determining the second pixel points respectively corresponding to the plurality of first pixel points in the target depth image according to the pixel correspondence between the target depth image and the target grayscale image, further including: Determining first coordinates of three-dimensional points respectively corresponding to a plurality of third pixel points in the target depth image in the depth camera coordinate system; Determine the second coordinates of the multiple three-dimensional points in the world coordinate system according to the conversion relationship between the depth camera coordinate system and the world coordinate system and the first coordinates; Determine the third coordinates of the multiple three-dimensional points in the grayscale camera coordinate system according to the conversion relationship between the world coordinate system and the grayscale camera coordinate system and the second coordinates; Determine the pixel correspondence between the target depth image and the grayscale image according to the third coordinates.

7. The method according to claim 1, wherein The step of determining the projection points of the three-dimensional line corresponding to the target line in the world coordinate system on the target plane along the target direction according to the multiple target pixel points includes: Determine the three-dimensional points respectively corresponding to the multiple target pixel points in the world coordinate system; Determine the projection coordinates of the multiple three-dimensional points on the target plane along the target direction respectively; Determine the mean value of the multiple projection coordinates as the coordinates of the projection points of the target line on the target plane.

8. The method according to any one of claims 1-7, characterized in that, The step of determining the two-dimensional projection boundary of the target object on the target plane according to the projection points respectively corresponding to the multiple target lines includes: Fit the projection points respectively corresponding to the multiple target lines to obtain the two-dimensional projection boundary of the target object on the target plane.

9. The method according to any one of claims 1-7, characterized in that, The target object further includes a second component, and the plane where the surface of the second component is located is perpendicular to the target direction; After determining the two-dimensional projection boundary of the target object on the target plane according to the projection points respectively corresponding to the multiple target lines, it further includes: Determine the target height of the second component along the target direction; Determine the three-dimensional structure model of the target object according to the target height and the two-dimensional projection boundary.

10. The method according to claim 9, wherein The step of determining the target height of the second component along the target direction includes: Obtain multiple third pixel points corresponding to the second component in the target depth image; Obtain the first coordinate values of the multiple third pixel points in the target direction; Determine the target height of the second component along the target direction according to the median of the multiple first coordinate values.

11. A projection boundary determination device, characterized in that, It includes: An acquisition module for acquiring a target depth image of a target object; the target object includes a first component for forming the three-dimensional contour of the target object; The surface of the first component includes a straight line in the target direction in the world coordinate system; A determination module for determining the vanishing point of the target depth image in the target direction and determining multiple target lines passing through the vanishing point in the target depth image; A projection module for, for each target line, acquiring multiple target pixel points corresponding to the first component located on the target line in the target depth image; and determining the projection points of the three-dimensional line corresponding to the target line in the world coordinate system on the target plane along the target direction according to the multiple target pixel points; The determination module is further configured to determine the two-dimensional projection boundary of the target object on the target plane according to the projection points respectively corresponding to the multiple target lines.

12. An electronic device, characterized in that, It includes: A processor and a memory; The memory stores computer execution instructions; The processor executes the computer-executable instructions stored in the memory, such that the processor executes the projection boundary determination method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the processor executes the computer-executable instructions, the projection boundary determination method according to any one of claims 1 to 10 is implemented.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the method for determining the projection boundary according to any one of claims 1 to 10 is implemented.

Images