Method and device for generating three-dimensional bounding frame identification, and computer storage medium

Abstract

The embodiment of the present application discloses a method for generating a three-dimensional bounding frame mark, the method comprising: collecting a current scene image containing a target object; determining a three-dimensional coordinate data set corresponding to the current scene image; determining a coordinate direction corresponding to the target object and a target three-dimensional coordinate data subset based on the three-dimensional coordinate data set; determining a three-dimensional coordinate range of the target object in the three-dimensional coordinate direction according to a coordinate direction and the target three-dimensional coordinate data subset; generating a three-dimensional bounding frame mark corresponding to the target object based on the three-dimensional coordinate direction and the three-dimensional coordinate range. At the same time, the present application also proposes a device for generating a three-dimensional bounding frame mark and a computer storage medium.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and in particular to a method and device for generating a three-dimensional bounding frame mark, and a computer storage medium. Background Art

[0002] In the field of augmented reality (AR), 3D object recognition is a commonly used object recognition technology. In 3D object recognition, it is necessary to first obtain relevant data (including depth map, RGB map, internal and external parameters of the scanning device, etc.) by scanning the object, and then train the data to reconstruct the feature library, which is then used for subsequent object recognition and pose estimation.

[0003] When scanning an object, the object needs to be placed on a plane with a clean background. In order to eliminate background interference and reconstruct the object more accurately, the target object needs to be framed before scanning begins to determine the three-dimensional space bounding box of the object.

[0004] However, the related technology generally adopts manual framing of objects to generate bounding boxes, which has the problem of poor object framing accuracy, resulting in large deviations in the bounding boxes of the objects. Summary of the invention

[0005] In order to solve the above technical problems, the embodiments of the present application hope to provide a method and device for generating a three-dimensional bounding frame mark, and a computer storage medium.

[0006] The technical solution of this application is implemented as follows:

[0007] The present application provides a method for generating a three-dimensional bounding box marker, the method comprising:

[0008] Acquire a current scene image containing a target object;

[0009] Determine a three-dimensional coordinate data set corresponding to the current scene image;

[0010] Determine a coordinate direction corresponding to the target object and a target three-dimensional coordinate data subset based on the three-dimensional coordinate data set;

[0011] Determining a three-dimensional coordinate range of the target object in the three-dimensional coordinate direction according to the one coordinate direction and the target three-dimensional coordinate data subset;

[0012] A three-dimensional bounding frame marker corresponding to the target object is generated based on the three-dimensional coordinate direction and the three-dimensional coordinate range.

[0013] The embodiment of the present application provides a device for generating a three-dimensional bounding frame mark, the device comprising a collection unit, a calculation unit, a determination unit and a generation unit.

[0014] The acquisition unit is used to acquire a current scene image containing a target object;

[0015] The computing unit is used to determine a three-dimensional coordinate data set corresponding to the current scene image;

[0016] The determining unit is used to determine a coordinate direction corresponding to the target object and a target three-dimensional coordinate data subset based on the three-dimensional coordinate data set;

[0017] The determining unit is further used to determine the three-dimensional coordinate range of the target object in the three-dimensional coordinate direction according to the one coordinate direction and the target three-dimensional coordinate data subset;

[0018] The generating unit is used to generate a three-dimensional bounding frame marker corresponding to the target object based on the three-dimensional coordinate direction and the three-dimensional coordinate range.

[0019] An embodiment of the present application provides a device for generating a three-dimensional bounding box mark, the device comprising a processor and a memory storing instructions executable by the processor, and when the instructions are executed by the processor, the method for generating a three-dimensional bounding box mark as described above is implemented.

[0020] An embodiment of the present application provides a computer-readable storage medium storing a program, and when the program is executed by a processor, the method for generating a three-dimensional bounding box mark as described above is implemented.

[0021] The method for generating a three-dimensional bounding frame mark provided in the embodiment of the present application is as follows: collecting a current scene image containing a target object; determining a three-dimensional coordinate data set corresponding to the current scene image; determining a coordinate direction corresponding to the target object and a target three-dimensional coordinate data subset based on the three-dimensional coordinate data set; determining a three-dimensional coordinate range of the target object in the three-dimensional coordinate direction according to a coordinate direction and the target three-dimensional coordinate data subset; and generating a three-dimensional bounding frame mark corresponding to the target object based on the three-dimensional coordinate direction and the three-dimensional coordinate range. In this way, by determining the three-dimensional coordinate data corresponding to the target object and the three-dimensional coordinate range of the target object in the three-dimensional direction from the current scene image, the object to be scanned is automatically and accurately framed from the current scene image, thereby generating a three-dimensional space bounding box of the object to be scanned. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of manually framing an object;

[0023] Figure 2Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark proposed in the embodiment of the present application Figure 1 ;

[0024] Figure 3 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box provided in the embodiment of the present application Figure 2 ;

[0025] Figure 4 A schematic diagram showing a three-dimensional bounding box according to an embodiment of the present application;

[0026] Figure 5 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark provided in the embodiment of the present application Figure 3 ;

[0027] Figure 6 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark provided in the embodiment of the present application Figure 4 ;

[0028] Figure 7 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark provided in the embodiment of the present application Figure 5 ;

[0029] Figure 8 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark provided in the embodiment of the present application Figure 6 ;

[0030] Fig. 9 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark provided in the embodiment of the present application Figure 7 ;

[0031] Fig.10 A schematic diagram of the structure of a device for generating a three-dimensional bounding box mark proposed in an embodiment of the present application Figure 1 ;

[0032] Fig.11 A schematic diagram of the structure of a device for generating a three-dimensional bounding box mark proposed in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0034] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0037] At present, in the field of AR, 3D object recognition is a commonly used object recognition technology. In 3D object recognition, it is necessary to first obtain relevant data (including depth map, RGB map, internal and external parameters of the scanning device, etc.) by scanning the object, and then train the data to reconstruct the feature library, which is then used for subsequent object recognition and pose estimation.

[0038] When scanning an object, the object needs to be placed on a plane with a clean background. In order to eliminate background interference and reconstruct the object more accurately, the target object needs to be framed before scanning begins to determine the three-dimensional space bounding box of the object.

[0039] However, related technologies generally use manual framing of objects to generate bounding boxes. Figure 1 This is a schematic diagram of manually framing an object, such as Figure 1 As shown in the figure, the interface sets parameters, namely, the adjustment buttons for the length, width, and height of the bounding box. The length, width, and height of the three-dimensional bounding box can be adjusted manually by adjusting the parameter adjustment buttons to frame the object as much as possible. This manual framing method has the problem of poor object framing accuracy, which results in a large deviation in the bounding box of the object.

[0040] In view of this, how to achieve accurate framing of an object to improve the accuracy of the object bounding box is the content to be discussed in the embodiments of the present application, and will be explained below in conjunction with the following specific embodiments.

[0041] The embodiment of the present application provides a method and device for generating a three-dimensional bounding frame mark, and a computer storage medium, by collecting a current scene image containing a target object; determining a three-dimensional coordinate data set corresponding to the current scene image; determining a coordinate direction corresponding to the target object and a target three-dimensional coordinate data subset based on the three-dimensional coordinate data set; determining a three-dimensional coordinate range of the target object in the three-dimensional coordinate direction according to a coordinate direction and the target three-dimensional coordinate data subset; and generating a three-dimensional bounding frame mark corresponding to the target object based on the three-dimensional coordinate direction and the three-dimensional coordinate range. In this way, by determining the three-dimensional coordinate data corresponding to the target object and the three-dimensional coordinate range of the target object in the three-dimensional direction from the current scene image, the object to be scanned is automatically and accurately framed from the current scene image, thereby generating a three-dimensional space bounding box of the object to be scanned.

[0042] The method for generating a three-dimensional bounding frame marker provided in the embodiment of the present application is applied to a device for generating a three-dimensional bounding frame marker. The following describes an exemplary application of the device for generating a three-dimensional bounding frame marker provided in the embodiment of the present application. The device for generating a three-dimensional bounding frame marker provided in the embodiment of the present application can be implemented as a mobile phone, a laptop computer, a tablet computer, a desktop computer, a smart TV, a vehicle-mounted device, a wearable device, an industrial device, etc.

[0043] Below, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application.

[0044] An embodiment of the present application provides a method for generating a three-dimensional bounding box mark. Figure 2 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark proposed in the embodiment of the present application Figure 1 ,like Figure 2 As shown, in an embodiment of the present application, the method for generating a three-dimensional bounding box mark may include the following steps:

[0045] S101: Acquire a current scene image containing a target object.

[0046] In an embodiment of the present application, a device for generating a three-dimensional bounding box mark can perform image acquisition processing on the current scene and obtain an image corresponding to the current scene; wherein, the target object is in the current scene, and correspondingly, the current scene image is an image containing the target object.

[0047] In some embodiments, an image acquisition device, i.e., a camera, may be configured in the device for generating the three-dimensional encircling frame mark. For example, when the device for generating the three-dimensional encircling frame mark is a smart phone, the smart phone may include a front camera and a rear camera, and the image acquisition device may be the front camera or the rear camera; when the device for generating the three-dimensional encircling frame mark is a laptop computer, the image acquisition device may be the computer camera.

[0048] It is understandable that when using a camera to collect and process the image of a target object, it is impossible to achieve that the collected image only contains the target object, but a large scene image containing the target object and the background in which the target object is located is collected. For example, if the target object is placed on the ground or on a table, then the corresponding scene image is an image containing the ground background, the target object and other objects on the ground that are close to it, or an image containing the table background, the target object and other adjacent objects placed on the table. Here, in order to avoid the influence of other objects on the plane on the target object, the target object is placed on a plane with a clean background as much as possible.

[0049] It should be noted that when the camera is used for image acquisition, the target object can be placed at the target position in the device image display interface; optionally, the target position can be the middle position of the display interface. Further, in the embodiment of the present application, after acquiring the current scene image containing the target object, the three-dimensional coordinate data corresponding to the current scene image can be further determined.

[0050] Step 102: Determine a three-dimensional coordinate data set corresponding to the current scene image.

[0051] In an embodiment of the present application, after acquiring a current scene image containing a target object, the device for generating a three-dimensional bounding frame mark can further determine a three-dimensional coordinate data set corresponding to the current scene image.

[0052] It should be noted that the three-dimensional coordinate data set includes the three-dimensional space coordinates of each pixel point in the current scene image.

[0053] It can be understood that the process from real scene to imaging is the process from three-dimensional to two-dimensional, so the process of determining the three-dimensional spatial coordinates from the image is the three-dimensional reconstruction process. Here, the three-dimensional spatial coordinates of each pixel point in the current scene image can be determined based on the camera calibration process.

[0054] Specifically, Figure 3 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark proposed in the embodiment of the present application Figure 2 ,like Figure 3 As shown, the specific method for determining the three-dimensional coordinate data set corresponding to the current scene image may include the following steps:

[0055] Step 102a: Determine the image coordinate data and depth value of each pixel in the current scene image.

[0056] Step 102b: read the shooting calibration parameters.

[0057] Step 102c: Perform three-dimensional space mapping processing based on the shooting calibration parameters, the image coordinate data of each pixel point, and the depth value to obtain a three-dimensional coordinate data set.

[0058] Specifically, in an embodiment of the present application, the image coordinate data and depth value of each pixel point in the current scene image, that is, the coordinate data of each pixel point in the image coordinate system, can be determined first, and then the shooting calibration parameters, that is, the calibration parameters of an image acquisition device such as a camera, are read, including camera intrinsic parameters and camera extrinsic parameters, and then the image coordinate data and depth value of each pixel point are processed by coordinate conversion and mapping from the image coordinate system to the camera coordinate system and further to the world coordinate system in combination with the camera intrinsic parameters and the camera extrinsic parameters, so as to obtain the three-dimensional spatial coordinates of each pixel point in the world coordinate system.

[0059] Furthermore, in an embodiment of the present application, after determining the three-dimensional coordinate data of each pixel point in the current scene image, a three-dimensional coordinate data subset corresponding to the target object and a direction in one dimension corresponding to the target object can be further determined from the three-dimensional coordinate data set.

[0060] Step 103: Determine a coordinate direction corresponding to the target object and a target three-dimensional coordinate data subset based on the three-dimensional coordinate data set.

[0061] In an embodiment of the present application, after determining the three-dimensional coordinate data set corresponding to the current scene image, the device for generating the three-dimensional bounding frame mark can further determine a three-dimensional coordinate data subset of the target object in the scene image and a direction of the target object based on the three-dimensional coordinate data set.

[0062] It should be noted that a direction of the target object may be any coordinate direction corresponding to the target object, and in the embodiment of the present application, the Z-axis coordinate direction is used for illustration.

[0063] It can be understood that the Z-axis coordinate direction corresponding to the target object is perpendicular to the plane where the target object is located. Therefore, in an embodiment of the present application, the Z-axis coordinate direction corresponding to the target object can be further determined by determining the plane where the target object is located.

[0064] It can be understood that the target object is a partial area of ​​the current scene image. Then in an embodiment of the present application, after the three-dimensional spatial coordinates of each pixel point in the current scene image are obtained, the three-dimensional coordinate data subset corresponding to the target object can be determined from the current scene image based on the coordinate segmentation method.

[0065] Specifically, the three-dimensional coordinate data set corresponding to the current scene image may be segmented based on the position information of the target object, thereby obtaining a three-dimensional coordinate data subset corresponding to the target object.

[0066] Furthermore, in the embodiment of the present application, after determining a coordinate direction of the target object and a three-dimensional coordinate subset corresponding to the target object, the other two directions corresponding to the target object may be determined.

[0067] Step 104 : Determine the three-dimensional coordinate range of the target object in the three-dimensional coordinate direction according to a coordinate direction and the target three-dimensional coordinate data subset.

[0068] In an embodiment of the present application, after determining a coordinate direction of the target object and a three-dimensional coordinate subset corresponding to the target object, the device for generating a three-dimensional bounding box mark can continue to determine the other two directions corresponding to the target object and the coordinate range of the target object in the three-dimensional coordinate direction based on one direction and the three-dimensional coordinate subset of the target object.

[0069] Here, on the basis that one direction of the target object is the Z-axis coordinate direction, the other two coordinate directions of the target object may be the X-axis coordinate direction and the Y-axis coordinate direction.

[0070] It can be understood that the plane where the target object is located is perpendicular to the Z-axis coordinate direction of the target object, while the X-axis coordinate direction and the Y-axis coordinate direction of the target object are parallel to the plane where the target object is located. Specifically, in an embodiment of the present application, a three-dimensional bounding frame marking generating device can refer to the relative position relationship between the Z-axis coordinate direction and the plane, and the relative position relationship between the X-axis coordinate direction and the Y-axis coordinate direction and the plane, and first determine the other two directions of the target object, including the X-axis coordinate direction and the Y-axis coordinate direction, based on the Z-axis coordinate direction and a subset of the three-dimensional coordinate data of the target object.

[0071] Furthermore, after determining the three-dimensional coordinate direction corresponding to the target object, the coordinate range of the target object in the X-axis coordinate direction, the Y-axis coordinate direction and the Z-axis coordinate direction can be determined based on the target three-dimensional coordinate data subset corresponding to the target object, that is, the maximum and minimum values ​​of the coordinate values ​​in each coordinate direction.

[0072] Furthermore, in the embodiment of the present application, after the three-dimensional coordinate range of the target object in the three-dimensional direction is determined, the generation of the three-dimensional bounding frame mark can be continued.

[0073] Step 105: Generate a three-dimensional bounding frame marker corresponding to the target object based on the three-dimensional coordinate direction and the three-dimensional coordinate range.

[0074] In an embodiment of the present application, after determining the three-dimensional coordinate range of the target object in the three-dimensional direction, the three-dimensional bounding box mark generating device can further generate a three-dimensional bounding box mark corresponding to the target object based on the three-dimensional coordinate direction and the three-dimensional coordinate range.

[0075] It should be understood that the three-dimensional bounding frame mark can be considered as a three-dimensional bounding box that frames the target object. In some embodiments, a cube with corresponding length, width and height can be directly generated based on the three-dimensional coordinate range of the target object in the three-dimensional coordinate direction determined above, and the cube is the three-dimensional bounding box corresponding to the target object.

[0076] In other embodiments, the three-dimensional bounding box marker can pre-set extension parameters. After obtaining the three-dimensional coordinate range of the target object in the three-dimensional coordinate direction, the three-dimensional coordinate range can be extended to a preset size based on the preset extension parameters, and a cube with length, width and height is generated based on the extended coordinate range. The cube is the three-dimensional bounding box corresponding to the target object.

[0077] Furthermore, in the embodiment of the present application, the generating device of the three-dimensional bounding frame mark is configured with a visualization interface, and the three-dimensional bounding box corresponding to the generated target object can be displayed on the visualization interface for the user to view. The visualization interface is an interface for image preview during image acquisition, and the current scene containing the target object can be displayed in the preview interface, and the three-dimensional bounding box surrounds the target object.

[0078] It can be understood that the image is collected in real time, so the device for generating the three-dimensional bounding box mark can frame the target object in each frame of the image, generate a three-dimensional bounding box corresponding to each frame of the image, and display the three-dimensional bounding box corresponding to the target object in each frame of the image in the image preview interface.

[0079] Figure 4 This is a schematic diagram of displaying a three-dimensional bounding box proposed in an embodiment of the present application, such as Figure 4 As shown, the interface does not need to set any parameter adjustment buttons. The camera only needs to be pointed at the object to capture the image, and the method for generating the three-dimensional bounding box mark proposed in the embodiment of the present application can be applied to automatically generate a three-dimensional bounding box corresponding to the target object and accurately frame the target object.

[0080] Furthermore, in the process of generating a three-dimensional bounding box, the bounding box in each frame image corresponding to the target object can be superimposed and displayed on the image preview interface, and when the user determines that the bounding box is correct, the scanning operation is triggered. For example, when the user clicks the scan button, the device for generating the three-dimensional bounding box mark can receive the corresponding scanning instruction, and then respond to the scanning instruction to scan the target object based on the three-dimensional bounding box.

[0081] The embodiment of the present application provides a method for generating a three-dimensional bounding frame mark, which includes collecting a current scene image containing a target object; determining a three-dimensional coordinate data set corresponding to the current scene image; determining a coordinate direction corresponding to the target object and a target three-dimensional coordinate data subset based on the three-dimensional coordinate data set; determining a three-dimensional coordinate range of the target object in the three-dimensional coordinate direction according to a coordinate direction and the target three-dimensional coordinate data subset; and generating a three-dimensional bounding frame mark corresponding to the target object based on the three-dimensional coordinate direction and the three-dimensional coordinate range. In this way, by determining the three-dimensional coordinate data corresponding to the target object and the three-dimensional coordinate range of the target object in the three-dimensional direction from the current scene image, the object to be scanned is automatically and accurately framed from the current scene image, thereby generating a three-dimensional space bounding box of the object to be scanned.

[0082] Based on the above embodiments, in one embodiment of the present application, Figure 5 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark proposed in the embodiment of the present application Figure 3 ,like Figure 5 As shown, the method of performing coordinate segmentation processing on the three-dimensional coordinate data set to determine the first coordinate direction corresponding to the target object and the target three-dimensional coordinate data subset includes the following steps:

[0083] Step 103a: perform plane fitting processing on the three-dimensional coordinate data set to determine a first coordinate direction and a first subset of the three-dimensional coordinate data set above the plane where the target object is located.

[0084] It should be noted that in the embodiment of the present application, the three-dimensional coordinate data set corresponding to the current scene image can be subjected to coordinate segmentation processing to determine the Z-axis coordinate direction corresponding to the target object and the three-dimensional coordinate data subset corresponding to the target object.

[0085] It can be understood that the target object is on a plane, and the plane is perpendicular to the Z-axis coordinate direction of the target object. Therefore, the three-dimensional coordinate data set of the current scene image can be coordinate segmented by determining the plane information on which the target object is located, and the Z-axis coordinate direction of the target object can be determined based on the plane information.

[0086] in, Figure 6 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark proposed in the embodiment of the present application Figure 4 ,like Figure 6 As shown, the method of performing plane fitting processing on the three-dimensional coordinate data set to determine the first coordinate direction and the first subset of the three-dimensional coordinate data set above the plane where the target object is located includes the following steps:

[0087] Step 103a1: perform sampling processing on the three-dimensional space coordinate data set to obtain a second subset of the three-dimensional coordinate data set.

[0088] Step 103a2: Perform plane fitting processing on the second subset to obtain a fitted plane.

[0089] Step 103a3: determine the normal direction perpendicular to the plane where the target object is located according to the fitted plane, and determine the normal direction as the first coordinate direction corresponding to the target object.

[0090] Step 103a4: perform coordinate segmentation processing on the three-dimensional coordinate data set based on the fitted plane to obtain a first subset.

[0091] Specifically, the three-dimensional space coordinate data set corresponding to the current scene image can be randomly sampled or uniformly collected to obtain a sampled three-dimensional coordinate data subset, i.e., a second subset. Then, a plane fitting process can be performed based on the second subset to determine the plane information where the target object is located, i.e., the fitted plane.

[0092] Optionally, in an embodiment of the present application, the plane information on which the target object is located may be determined based on any plane fitting algorithm, such as a random sampling consensus (RANSAC) plane fitting algorithm.

[0093] Here, after the plane information where the target object is located is determined, the direction perpendicular to the plane, that is, the normal direction of the plane, can be determined as the Z-axis coordinate direction of the target object.

[0094] It is understandable that if the target object is located on a plane, the fitted plane can be used to first perform coordinate segmentation processing on the three-dimensional coordinate data set corresponding to the current scene image to obtain a three-dimensional coordinate data subset located on the plane, namely, the first subset.

[0095] For example, the fitted plane can be represented by a plane equation, such as Ax+By+Cz+D=0. The three-dimensional spatial coordinates of each pixel point can be substituted into the plane equation. If the result is less than 0, it is below the plane. If the result is greater than 0, it is determined that the pixel point is above the plane.

[0096] Step 103b: Obtain the target position of the target object in the current scene image.

[0097] Step 103c: perform coordinate segmentation processing on the first subset based on the target position to obtain a target three-dimensional coordinate data subset.

[0098] It should be understood that there may be other objects above the plane, including the target object. In order to better determine the pixel points corresponding to the target object from the pixel points above the plane, so as to determine the three-dimensional coordinate data subset corresponding to the target object, the three-dimensional coordinate data subset above the plane, i.e., the first subset, can be further subjected to coordinate segmentation processing based on the position information of the target object in the current scene image, so as to obtain the three-dimensional spatial coordinate data of each pixel point on the target object.

[0099] Based on step 101, it can be known that when performing image acquisition, the target object can be placed at the target position of the display interface, such as the middle position of the display interface. Therefore, we can perform coordinate segmentation processing on the first subset based on the position information, thereby determining the three-dimensional coordinate data subset of the target object located at the middle position.

[0100] It can be understood that pixels located adjacent to each other on the same target object have the same pixel value, and coordinate identification and segmentation can be performed based on connected domain analysis to determine the independent connected domains existing on the plane and mark them. Then, the connected domain corresponding to the target object is determined based on the position information of the target object, and its corresponding three-dimensional coordinate data subset is the three-dimensional coordinate data subset corresponding to the target object.

[0101] It can be seen that in the embodiment of the present application, the device for generating a three-dimensional bounding box identification determines the plane in which the target object is located through plane fitting, and determines the Z-axis direction corresponding to the target object; and performs coordinate segmentation based on the plane information to determine the three-dimensional space coordinate data above the plane, and determines the three-dimensional space coordinate data corresponding to the target object based on the position information of the target object and the connected domain analysis, which can reduce unnecessary operation steps and reduce interaction costs, so as to further improve the accuracy of determining the three-dimensional bounding box of the target object.

[0102] Based on the above embodiments, in one embodiment of the present application, Figure 7 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark proposed in the embodiment of the present application Figure 5 ,like Figure 7 As shown, the method of performing coordinate segmentation processing on the three-dimensional coordinate data set to determine the first coordinate direction corresponding to the target object and the target three-dimensional coordinate data subset includes the following steps:

[0103] Step 104a, determining a second coordinate direction and a third coordinate direction corresponding to the target object based on the target three-dimensional coordinate data subset and the first coordinate direction; wherein the first coordinate direction, the second coordinate direction and the third coordinate direction are perpendicular to each other.

[0104] Specifically, in an embodiment of the present application, the second coordinate direction may be the X-axis coordinate direction of the target object, and the third coordinate direction may be the Y-axis coordinate direction of the target object; or the second coordinate direction may be the Y-axis coordinate direction of the target object, and the third coordinate direction may be the X-axis coordinate direction of the target object.

[0105] Among them, we can refer to the perpendicular relationship between the Z-axis coordinate direction and the plane, and the parallel relationship between the X-axis coordinate direction and the fitted plane, and project the three-dimensional spatial coordinates of each pixel point in the three-dimensional coordinate data set to the plane based on the Z-axis coordinate direction to further determine the three-dimensional coordinate direction of the target object.

[0106] Here, the three-dimensional coordinate directions of the target object, including the Z axis, the Y axis, and the Z axes are perpendicular to each other.

[0107] in, Figure 8 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark proposed in the embodiment of the present application Figure 6 ,like Figure 8 As shown, the method for determining the second coordinate direction and the third direction coordinate corresponding to the target object based on the target three-dimensional coordinate data subset and the first coordinate direction includes the following steps:

[0108] Step 104a1: project the target three-dimensional coordinate data subset onto the fitted plane based on the first coordinate direction to obtain a projected coordinate data set.

[0109] Step 104a2: Perform feature recognition on the projected coordinate data set to determine the second coordinate direction and the third coordinate direction.

[0110] Specifically, after the three-dimensional space coordinates of each pixel point in the target object are projected onto a plane based on the Z-axis coordinate direction, the projected three-dimensional space coordinates of each pixel point are obtained.

[0111] It should be noted that the Z-axis coordinate value of the three-dimensional space coordinate of each pixel point after projection is 0. For example, the three-dimensional space coordinate of the first pixel point is ( ), the three-dimensional space coordinates after projection are ( ).

[0112] Further, feature recognition can be performed on the projected coordinate data set to determine the X-axis coordinate direction and the Y-axis coordinate direction of the target object. Here, the principal component analysis method can be used to extract and analyze the features of the projected coordinate data set to further determine the X-axis coordinate direction and the Y-axis coordinate direction of the target object.

[0113] Step 104b: determine the three-dimensional coordinate range based on the first coordinate direction, the second coordinate direction, the third coordinate direction, and the target three-dimensional coordinate data subset.

[0114] Furthermore, after the X-axis coordinate direction, Y-axis coordinate direction and Z-axis coordinate direction of the target object are determined, the maximum and minimum coordinate values ​​in each direction can be determined based on the three-dimensional spatial coordinates of each pixel point in the target object.

[0115] in, Fig. 9 Schematic diagram of the implementation process of the method for generating a three-dimensional bounding box mark proposed in the embodiment of the present application Figure 7 ,like Fig. 9 As shown, the method for determining the three-dimensional coordinate range based on the first coordinate direction, the second coordinate direction, the third coordinate direction and the target three-dimensional coordinate data subset specifically includes the following steps:

[0116] Step 104b1: Determine the maximum coordinate value and the minimum coordinate value in each coordinate direction from the three-dimensional coordinate data of the target three-dimensional coordinate data subset according to the first coordinate direction, the second coordinate direction or the third coordinate direction.

[0117] Step 104b2: determine the coordinate range of the target object in each coordinate direction based on the maximum coordinate value and the minimum coordinate value in each coordinate direction.

[0118] Step 104b3: determine the three-dimensional coordinate range based on the coordinate range in each coordinate direction.

[0119] The generating device of the three-dimensional bounding frame mark can first determine the maximum x value and the minimum x value in the X-axis coordinate direction from the three-dimensional space coordinate data of each pixel point in the target object according to the X-axis coordinate direction, that is, ( Similarly, according to the Y-axis coordinate direction, the maximum y value and the minimum y value in the Y-axis coordinate direction are determined from the three-dimensional space coordinate data of each pixel point in the target object, that is, ( Similarly, according to the Z-axis coordinate direction, the maximum z value and the minimum z value in the Z-axis coordinate direction are determined from the three-dimensional space coordinate data of each pixel point in the target object, that is, ( .

[0120] It should be understood that the minimum z value is the plane, in other words, the minimum z value is 0.

[0121] Furthermore, the maximum x value to the minimum x value is the coordinate range of the target object in the X-axis direction, the maximum y value to the minimum y value is the coordinate range of the target object in the Y-axis direction, and the maximum z value to 0 is the coordinate range of the target object in the Z-axis direction.

[0122] It should be noted that, in the embodiment of the present application, the order of determining the coordinate range of the target object in each coordinate direction does not distinguish between first and last order, and can also be performed simultaneously.

[0123] Furthermore, after determining the coordinate ranges of the target object in the X-axis coordinate direction, the Y-axis coordinate direction, and the Z-axis coordinate direction, a three-dimensional bounding box corresponding to the target object can be further generated.

[0124] It can be seen that in the embodiment of the present application, after the Z-axis coordinate direction of the target object and the three-dimensional spatial coordinates of each pixel in the target object are determined through plane fitting processing and connected domain analysis, the three-dimensional spatial coordinates of each pixel are projected onto the plane based on the Z-axis coordinate direction, and the X-axis coordinate direction and Y-axis coordinate direction of the target object are determined based on the principal component analysis method, and then the coordinate range of the target object in each coordinate direction is determined, thereby further generating a three-dimensional bounding box corresponding to the target object. In this way, the object to be scanned is automatically and accurately framed from the current scene image, thereby generating a three-dimensional spatial bounding box of the object to be scanned.

[0125] Based on the above embodiments, in one embodiment of the present application, Fig.10 A schematic diagram of the structure of a device for generating a three-dimensional bounding box mark proposed in an embodiment of the present application Figure 1 ,like Fig.10 As shown, the three-dimensional bounding box identification generating device 10 includes a collecting unit 11, a calculating unit 12, a determining unit 13, a generating unit 14, a receiving unit 15 and a scanning unit 16.

[0126] The acquisition unit 11 is used to acquire a current scene image containing a target object;

[0127] The calculation unit 12 is used to determine the three-dimensional coordinate data set corresponding to the current scene image;

[0128] The determining unit 13 is used to determine a coordinate direction corresponding to the target object and a target three-dimensional coordinate data subset based on the three-dimensional coordinate data set;

[0129] The determining unit 13 is further configured to determine a three-dimensional coordinate range of the target object in the three-dimensional coordinate direction according to the one coordinate direction and the target three-dimensional coordinate data subset;

[0130] The generating unit 14 is configured to generate a three-dimensional bounding frame marker corresponding to the target object based on the three-dimensional coordinate direction and the three-dimensional coordinate range.

[0131] In some embodiments, the determination unit 13 is specifically configured to perform coordinate segmentation processing on the three-dimensional coordinate data set to determine the first coordinate direction corresponding to the target object and the target three-dimensional coordinate data subset.

[0132] In some embodiments, the determination unit 13 is specifically used to perform plane fitting processing on the three-dimensional coordinate data set to determine the first coordinate direction and a first subset of the three-dimensional coordinate data set above the plane where the target object is located; and to obtain the target position of the target object in the current scene image; and to perform coordinate segmentation processing on the first subset based on the target position to obtain the target three-dimensional coordinate data subset.

[0133] In some embodiments, the determination unit 13 is specifically used to perform sampling processing on the three-dimensional space coordinate data set to obtain a second subset of the three-dimensional coordinate data set; and perform plane fitting processing on the second subset to obtain a fitted plane; and determine the normal direction perpendicular to the plane where the target object is located based on the fitted plane, and determine the normal direction as the first coordinate direction corresponding to the target object; and perform the coordinate segmentation processing on the three-dimensional coordinate data set based on the fitted plane to obtain the first subset.

[0134] In some embodiments, the determination unit 13 is specifically configured to determine the second coordinate direction and the third coordinate direction corresponding to the target object based on the target three-dimensional coordinate data subset and the first coordinate direction; wherein the first coordinate direction, the second coordinate direction and the third coordinate direction are perpendicular to each other. And the three-dimensional coordinate range is determined based on the first coordinate direction, the second coordinate direction, the third coordinate direction and the target three-dimensional coordinate data subset.

[0135] In some embodiments, the determination unit 13 is specifically used to project the target three-dimensional coordinate data subset to the fitted plane based on the first coordinate direction to obtain a projected coordinate data set; and perform feature recognition on the projected coordinate data set to determine the second coordinate direction and the third coordinate direction.

[0136] In some embodiments, the determination unit 13 is further specifically used to determine the maximum coordinate value and the minimum coordinate value in each coordinate direction from the three-dimensional coordinate data of the target three-dimensional coordinate data subset according to the first coordinate direction, the second coordinate direction or the third coordinate direction; and determine the coordinate range of the target object in each coordinate direction based on the maximum coordinate value and the minimum coordinate value in each coordinate direction; and determine the three-dimensional coordinate range based on the coordinate range in each coordinate direction.

[0137] In some embodiments, the receiving unit 15 is used to receive a scanning instruction.

[0138] In some embodiments, the scanning unit 16 is used to respond to the scanning instruction and scan the target object based on the three-dimensional bounding box mark.

[0139] In the embodiments of the present application, further, Fig.11 Schematic diagram of the composition structure of the device for generating a three-dimensional enclosing frame mark proposed in this application Figure 2 ,like Fig.11 As shown, the device 10 for generating a three-dimensional bounding box mark proposed in an embodiment of the present application may also include a processor 17 and a memory 18 storing executable instructions of the processor 17. Furthermore, the device 10 for generating a three-dimensional bounding box mark may also include a communication interface 19 and a bus 110 for connecting the processor 17, the memory 18 and the communication interface 19.

[0140] In an embodiment of the present application, the processor 17 may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the function of the processor may also be other, and the embodiment of the present application is not specifically limited. The device 10 for generating a three-dimensional enclosing frame mark may also include a memory 18, which may be connected to the processor 17, wherein the memory 18 is used to store executable program code, the program code includes computer operation instructions, and the memory 18 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0141] In the embodiment of the present application, the bus 110 is used to connect the communication interface 19, the processor 17 and the memory 18, and the mutual communication between these devices.

[0142] In the embodiment of the present application, the memory 18 is used to store instructions and data.

[0143] Furthermore, in an embodiment of the present application, the above-mentioned processor 17 is used to acquire a current scene image containing a target object; determine a three-dimensional coordinate data set corresponding to the current scene image; determine a coordinate direction corresponding to the target object and a target three-dimensional coordinate data subset based on the three-dimensional coordinate data set; determine a three-dimensional coordinate range of the target object in the three-dimensional coordinate direction according to the one coordinate direction and the target three-dimensional coordinate data subset; and generate a three-dimensional bounding frame marker corresponding to the target object based on the three-dimensional coordinate direction and the three-dimensional coordinate range.

[0144] In practical applications, the memory 18 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 17.

[0145] In addition, each functional module in this embodiment can be integrated into a recommended unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0146] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0147] The embodiment of the present application provides a device for generating a three-dimensional bounding frame mark, which can collect a current scene image containing a target object; determine a three-dimensional coordinate data set corresponding to the current scene image; determine a coordinate direction corresponding to the target object and a target three-dimensional coordinate data subset based on the three-dimensional coordinate data set; determine a three-dimensional coordinate range of the target object in the three-dimensional coordinate direction according to a coordinate direction and the target three-dimensional coordinate data subset; and generate a three-dimensional bounding frame mark corresponding to the target object based on the three-dimensional coordinate direction and the three-dimensional coordinate range. In this way, the object to be scanned can be automatically and accurately framed from the current scene image, thereby generating a three-dimensional space bounding box of the object to be scanned.

[0148] An embodiment of the present application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method for generating a three-dimensional bounding box marker as described above.

[0149] Specifically, the program instructions corresponding to the method for generating a three-dimensional bounding frame mark in this embodiment can be stored in a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to the method for generating a three-dimensional bounding frame mark in the storage medium are read or executed by an electronic device, the following steps are included:

[0150] Acquire a current scene image containing a target object;

[0151] Determine a three-dimensional coordinate data set corresponding to the current scene image;

[0152] Determine a coordinate direction corresponding to the target object and a target three-dimensional coordinate data subset based on the three-dimensional coordinate data set;

[0153] Determining a three-dimensional coordinate range of the target object in the three-dimensional coordinate direction according to the one coordinate direction and the target three-dimensional coordinate data subset;

[0154] A three-dimensional bounding frame marker corresponding to the target object is generated based on the three-dimensional coordinate direction and the three-dimensional coordinate range.

[0155] Accordingly, an embodiment of the present application further provides a computer program product, which includes computer executable instructions, and the computer executable instructions are used to implement the steps in the three-dimensional bounding box identification method provided in the embodiment of the present application.

[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0157] The present application is described with reference to the implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0158] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0160] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A method for generating a three-dimensional bounding box mark, characterized in that: The method comprises: Acquire a current scene image containing a target object; Determine a three-dimensional coordinate data set corresponding to the current scene image; Performing plane fitting processing on the three-dimensional coordinate data set to determine a first coordinate direction and a first subset of the three-dimensional coordinate data set on the plane where the target object is located; Acquire a target position of the target object in the current scene image; Performing coordinate segmentation processing on the first subset based on the target position to obtain a three-dimensional coordinate data subset of the target object; Determine a second coordinate direction and a third coordinate direction corresponding to the target object based on the three-dimensional coordinate data subset of the target object and the first coordinate direction; wherein the first coordinate direction, the second coordinate direction and the third coordinate direction are perpendicular to each other; Determine a three-dimensional coordinate range based on the first coordinate direction, the second coordinate direction, the third coordinate direction, and a subset of the three-dimensional coordinate data of the target object; A three-dimensional bounding frame marker corresponding to the target object is generated based on the first coordinate direction, the second coordinate direction, the third coordinate direction and the three-dimensional coordinate range.

2. The method according to claim 1, characterized in that: The performing plane fitting processing on the three-dimensional coordinate data set to determine the first coordinate direction and a first subset of the three-dimensional coordinate data set on the plane where the target object is located, comprises: Performing sampling processing on the three-dimensional coordinate data set to obtain a second subset of the three-dimensional coordinate data set; Performing plane fitting processing on the second subset to obtain a fitted plane; Determine a normal direction perpendicular to the plane where the target object is located according to the fitted plane, and determine the normal direction as a first coordinate direction corresponding to the target object; The coordinate segmentation process is performed on the three-dimensional coordinate data set based on the fitted plane to obtain the first subset.

3. The method according to claim 2, characterized in that Determining a second coordinate direction and a third coordinate direction corresponding to the target object based on the three-dimensional coordinate data subset of the target object and the first coordinate direction includes: Projecting the three-dimensional coordinate data subset of the target object onto the fitted plane based on the first coordinate direction to obtain a projected coordinate data set; Perform feature recognition on the projected coordinate data set to determine the second coordinate direction and the third coordinate direction.

4. The method according to claim 1, characterized in that: The determining the three-dimensional coordinate range based on the first coordinate direction, the second coordinate direction, the third coordinate direction and the three-dimensional coordinate data subset of the target object includes: According to the first coordinate direction, the second coordinate direction or the third coordinate direction, determining the maximum coordinate value and the minimum coordinate value in each coordinate direction from the three-dimensional coordinate data of the three-dimensional coordinate data subset of the target object; Determine the coordinate range of the target object in each coordinate direction based on the maximum coordinate value and the minimum coordinate value in each coordinate direction; The three-dimensional coordinate range is determined based on the coordinate ranges in the respective coordinate directions.

5. The method according to claim 1, characterized in that The determining of the three-dimensional coordinate data set corresponding to the current scene image includes: Determine image coordinate data and depth value of each pixel in the current scene image; Read the shooting calibration parameters; A three-dimensional space mapping process is performed based on the shooting calibration parameters, the image coordinate data of each pixel point, and the depth value to obtain the three-dimensional coordinate data set.

6. The method according to claim 1, characterized in that The method further comprises: Receive scanning instructions; In response to the scanning instruction, the target object is scanned based on the three-dimensional bounding box mark.

7. A device for generating a three-dimensional bounding frame mark, characterized in that: The device comprises a collection unit, a calculation unit, a determination unit and a generation unit. The acquisition unit is used to acquire a current scene image containing a target object; The computing unit is used to determine a three-dimensional coordinate data set corresponding to the current scene image; The determining unit is used to perform plane fitting processing on the three-dimensional coordinate data set to determine a first coordinate direction and a first subset of the three-dimensional coordinate data set on the plane where the target object is located; Acquire a target position of the target object in the current scene image; Performing coordinate segmentation processing on the first subset based on the target position to obtain a three-dimensional coordinate data subset of the target object; The determination unit is further used to determine a second coordinate direction and a third coordinate direction corresponding to the target object based on the three-dimensional coordinate data subset of the target object and the first coordinate direction; wherein the first coordinate direction, the second coordinate direction and the third coordinate direction are perpendicular to each other; and determine a three-dimensional coordinate range based on the first coordinate direction, the second coordinate direction, the third coordinate direction and the three-dimensional coordinate data subset of the target object; The generating unit is used to generate a three-dimensional bounding frame marker corresponding to the target object based on the first coordinate direction, the second coordinate direction, the third coordinate direction and the three-dimensional coordinate range.

Images