Object detection method, object recognition model training method and related device

Abstract

The present application discloses an object detection method, a training method for an object recognition model, a device, an electronic device and a computer-readable storage medium. The object detection method comprises: obtaining a point cloud obtained by scanning a target scene, wherein the target scene comprises a target object to be recognized; performing voxel division on the point cloud according to at least two voxel parameters to obtain at least two voxel sets corresponding to the at least two voxel parameters; performing three-dimensional feature extraction on each voxel in each voxel set to obtain the voxel feature of the voxel, and converting the voxel set into a two-dimensional pseudo image based on the voxel feature of each voxel in each voxel set; recognizing the target object based on the fusion result of the two-dimensional features extracted from the at least two two-dimensional pseudo images, thereby improving the accuracy of recognizing the target object as a whole.

Description

Technical Field

[0001] The present application relates to the field of object detection, and in particular to an object detection method, an object recognition model training method, a device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In scanning scenarios, it is necessary to identify the target object contained in the scan data. For some scenarios with high scanning accuracy requirements, such as fixed scanning scenarios, the target object is often fixed on a scanning plane, and during the scanning process, the scanning plane and the fixed objects used to fix the target object and other miscellaneous data will also be included in the scanning data. It is necessary to separate the three-dimensional data of the target object from the scanning data. However, since the shape and position of the fixed object vary during each recognition process, and the fixed position of the target object also varies a lot, the recognition accuracy of the target object in the related technology is low. Summary of the invention

[0003] In view of the above technical problems, the present application provides an object detection method, an object recognition model training method, a device, an electronic device and a computer-readable storage medium. The technical solutions are as follows:

[0004] According to a first aspect of the present application, a method for object detection is provided, the method comprising:

[0005] Acquire a point cloud obtained by scanning a target scene, wherein the target scene includes a target object to be identified;

[0006] Perform voxel division on the point cloud according to at least two voxel parameters to obtain at least two voxel sets corresponding to the at least two voxel parameters;

[0007] Performing three-dimensional feature extraction on each voxel in each of the voxel sets to obtain voxel features of the voxel, and converting the voxel sets into a two-dimensional pseudo image based on the voxel features of each voxel in each of the voxel sets;

[0008] The target object is identified based on a fusion result of two-dimensional features respectively extracted from at least two of the two-dimensional pseudo images.

[0009] According to a second aspect of the present application, there is provided an object detection device, the device comprising:

[0010] An acquisition unit, configured to acquire a point cloud obtained by scanning a target scene, wherein the target scene includes a target object to be identified;

[0011] A division unit, configured to perform voxel division on the point cloud according to at least two voxel parameters, to obtain at least two voxel sets corresponding to the at least two voxel parameters;

[0012] a conversion unit, configured to extract three-dimensional features from each voxel in each of the voxel sets to obtain voxel features of the voxel, and convert the voxel sets into a two-dimensional pseudo image based on the voxel features of each voxel in each of the voxel sets;

[0013] The recognition unit is used to recognize the target object based on the fusion result of the two-dimensional features respectively extracted from at least two of the two-dimensional pseudo images.

[0014] According to a third aspect of the present application, an electronic device is provided, the electronic device comprising:

[0015] processor;

[0016] a memory for storing processor-executable instructions;

[0017] The processor is configured to implement the method as described in the first aspect.

[0018] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the method described in the first aspect are implemented.

[0019] The technical solution provided by this application uses an object detection method based on voxel division to detect target objects (such as three-dimensional bounding box detection), which can intuitively and efficiently extract target objects. It is suitable for complex and changeable scanning scenes with high precision requirements, and the detection process does not require manual intervention, achieving an automated and intelligent target object detection process. When dividing voxels, the same point cloud is divided according to a variety of voxel parameters. Voxel parameters of different sizes have different division receptive fields. The features of different receptive fields can complement each other. The features of large receptive fields have more global information and can better detect larger target objects. The features of small receptive fields have more detailed local information and can better locate smaller target objects. The combination of features between different receptive fields can simultaneously cope with the detection of large and small objects, and improve the accuracy of identifying target objects as a whole.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0022] Figure 1A is a schematic diagram of an object scanning scene in the related art;

[0023] Figure 1B is another schematic diagram of an object scanning scene in the related art;

[0024] Figure 2 is a flow chart of an object detection method according to an embodiment of the present application;

[0025] Figure 3A This is a schematic diagram of a voxel partitioning scene according to an embodiment of the present application;

[0026] Figure 3B is a schematic diagram of a voxel partitioning scene according to another embodiment of the present application;

[0027] Figure 4 is a schematic diagram of a voxel partitioning scene according to another embodiment of the present application;

[0028] Figure 5 This is a schematic diagram of a three-dimensional feature extraction process according to an embodiment of the present application;

[0029] Figure 6 This is a schematic diagram of a three-dimensional feature extraction scene according to an embodiment of the present application;

[0030] Figure 7 It is a schematic diagram of a two-dimensional pseudo image conversion process according to an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of a two-dimensional pseudo image conversion process according to another embodiment of the present application;

[0032] Fig. 9 is a schematic diagram of a two-dimensional pseudo image conversion scenario according to an embodiment of the present application;

[0033] Fig.10 This is a schematic diagram of a feature fusion process of an embodiment of the present application;

[0034] Fig.11 This is a schematic diagram of a feature fusion scenario of an embodiment of the present application;

[0035] Fig.12 This is a schematic diagram of a decoding scenario for detecting a decoding head according to an embodiment of the present application;

[0036] Fig.13A is a schematic diagram of a target object recognition result according to an embodiment of the present application;

[0037] Fig. 13B is a schematic diagram of target object recognition results of another embodiment of the present application;

[0038] Fig.14It is a flowchart of a training method for an object recognition model according to an embodiment of the present application;

[0039] Fig.15 is a schematic structural diagram of an object detection device according to an embodiment of the present application;

[0040] Fig.16 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of this application.

[0042] In scanning scenarios, it is necessary to identify the target object contained in the scan data. For some scenarios with high scanning accuracy requirements, such as fixed scanning scenarios, the target object is often fixed on a scanning plane, and during the scanning process, the scanning plane and the fixed objects used to fix the target object and other miscellaneous data will also be included in the scanning data. It is necessary to separate the three-dimensional data of the target object from the scanning data. However, since the shape and position of the fixed object vary during each recognition process, and the fixed position of the target object also varies a lot, the recognition accuracy of the target object in the related technology is low.

[0043] Below Figure 1A-1B Taking this as an example, an object scanning scenario in the related art is introduced exemplarily:

[0044] like Figure 1A As shown, for some scenes with high scanning accuracy requirements, taking a fixed scanning scene as an example, the object to be scanned, that is, Figure 1A The target object in the scanning process is often fixed on a scanning plane. For example, the scanning plane may be located at Figure 1A On the tray shown, the target object can be scanned 360° around by rotating the tray to obtain the most realistic and complete three-dimensional data of the target object. For some special target objects, they may not be able to be directly fixed on the tray, such as the base, implant or scanning rod (implant scanning body) in the oral medical scene. It is often necessary to use additional tools such as plasticine as a fixing to fix the base, implant or scanning rod (implant scanning body) on the scanning plane of the tray to achieve the above scanning process. After the scanning is completed, for example, the following may be obtained Figure 1B The three-dimensional scanning data shown, such as Figure 1BAs shown, during the scanning process, miscellaneous data such as the tray and plasticine will also be included in the scanned three-dimensional data. It is necessary to extract the three-dimensional data of the base, implant or scanning rod (implant scanning body) from the scanned three-dimensional data, and eliminate the miscellaneous data such as the tray, scanning plane and plasticine to obtain the final three-dimensional scanning data of the target object.

[0045] Exemplarily, there are at least the following methods for separating the three-dimensional data of the target object from the scanned three-dimensional data in the related art:

[0046] (1) Manual separation method: With the pallet as the horizontal plane, the pallet is manually dragged up and down along the Z-axis direction of the scanning plane. The appropriate cutting plane is found by visual observation. Based on the cutting plane, at least the three-dimensional data of the pallet is removed. However, this method has a low degree of automation, high labor costs, and low efficiency.

[0047] (2) Automatic plane fitting algorithm: It is necessary to separate the 3D data of the abutment, implant or scanning rod (implant scanning body) from the 3D data of the tray by fitting the scanning plane of the tray, and obtain the 3D scanning data of the independent abutment, implant or scanning rod (implant scanning body). However, since the fixtures that fix the abutment, implant or scanning rod (implant scanning body) to the tray, such as plasticine, are handmade, their shapes, positions and sizes are often complex and varied, which also leads to the complexity and randomness of the scanning scene. Therefore, the traditional automatic plane fitting algorithm is difficult to adapt to the complex and varied scanning scenes, that is, it is difficult to fit a suitable scanning plane, resulting in low quality of the 3D scanning data of the target object finally extracted.

[0048] (3) Point cloud segmentation-based method: directly classify all point clouds in the 3D scanning data to separate the 3D data of the target object. However, the point cloud segmentation-based method is prone to blurred segmentation boundaries, resulting in incomplete segmented data. In addition, since it is a point cloud-level classification method, it is easy to introduce small noise and requires additional post-processing steps, so a single separation process takes a long time.

[0049] It is understandable that for fixed scanning scenarios, such as the abutment, implant or scanning rod (implant scanning body) as the target object in the oral medical scenario, due to the particularity of its application scenario, that is, it is often used for oral treatment of oral patients, slight deviations in its shape and size may have a greater impact on the patients. Therefore, the scanning accuracy of the abutment, implant or scanning rod (implant scanning body) as the target object is significantly higher than that in other scenarios.

[0050] It is worth noting that the above introduction to the object scanning scene in the related art is only an exemplary display. In actual application, other object scanning scenes are not excluded and no specific limitation is made to this.

[0051] In view of the above problems, the present application provides an object detection method, which can be applied to complex and changeable scanning scenes with high precision requirements, and can improve the accuracy of identifying target objects as a whole. Figure 2 As shown, the method comprises the following steps:

[0052] S201: Obtain a point cloud obtained by scanning a target scene.

[0053] The target scene includes a target object to be identified.

[0054] S202 , performing voxel division on the point cloud according to at least two voxel parameters, to obtain at least two voxel sets corresponding to the at least two voxel parameters.

[0055] S203 , extracting three-dimensional features from each voxel in each voxel set to obtain voxel features of the voxel, and converting the voxel set into a two-dimensional pseudo image based on the voxel features of each voxel in each voxel set.

[0056] S204: Identify the target object based on a fusion result of the two-dimensional features respectively extracted from at least two two-dimensional pseudo images.

[0057] The technical solution provided in the embodiment of the present application uses an object detection method based on voxel division to detect target objects (such as three-dimensional bounding box detection), which can intuitively and efficiently extract target objects. It is suitable for complex and changeable scanning scenes with high precision requirements, and the detection process does not require manual intervention, so as to achieve an automated and intelligent target object detection process. When dividing voxels, the same point cloud is divided according to a variety of voxel parameters. Voxel parameters of different sizes have different division receptive fields. The features of different receptive fields can complement each other. The features of large receptive fields have more global information and can better detect larger target objects. The features of small receptive fields have more detailed local information and can better locate smaller target objects. The combination of features between different receptive fields can simultaneously cope with the detection of large and small objects, and improve the accuracy of identifying target objects as a whole.

[0058] The embodiments of the present application can be applied to a variety of scenarios. As an example, the target scene can be a fixed scanning scene. In the fixed scanning scene, the target scene is a scene containing a target object to be identified and a non-target object used to fix the target object, wherein the target object is an abutment, an implant and / or a scanning rod (implant scanning body) in an oral medical scenario. It is worth noting that the above introduction to the application scenarios of the embodiments of the present application and the specific implementation of the target object is only an exemplary display. In actual applications, other application scenarios are not excluded. The target object can also be any object in a scanning scenario with high scanning accuracy requirements except the abutment, implant and / or scanning rod (implant scanning body). The scanning rod can be a coded scanning rod or a scanning rod without coding.

[0059] As another example, the target scene may also be an industrial scanning scene. In the industrial scanning scene, the target objects may be bolts, tools and equipment, small workpieces and / or small figures; as another example, the target scene may also be a large scene scanning scene. In the large scene scanning scene, the target objects may be buildings or cars; as another example, the target scene may also be an ear scanning scene. In the ear scanning scene, the target objects may be artificial cochleas and / or ear prints; as another example, the embodiments of the present application may also be applied to fixed scanning scenes of desktop scanners, or mobile scanning scenes such as handheld scanners. Therefore, there are no specific limitations on the application scenarios and implementation of the target objects of the embodiments of the present application.

[0060] As another example, the non-target object used to fix the target object may be plasticine and brackets in an oral medical setting. It is understandable that plasticine is used to fix abutments, implants and / or scanning rods (implant scanning bodies) on brackets. Since plasticine is kneaded by hand, and multiple plasticines may be kneaded on the same bracket to fix multiple abutments, implants and / or scanning rods (implant scanning bodies), the shapes, sizes and positions of the multiple plasticines on the same bracket may be different.

[0061] After obtaining the point cloud obtained by scanning the target scene in S201, the target object can be identified from the point cloud in a variety of ways. As an example, the obtained point cloud can be input into a pre-trained object recognition model, and the object recognition model executes the steps described in S202-S204 above to identify the target object in the target scene.

[0062] It is understandable that voxel is the abbreviation of volume pixel, which is a concept used to represent discrete volume units in three-dimensional space in digital imaging. Voxel is a basic unit in three-dimensional space, similar to pixels in two-dimensional images, but it adds a depth dimension. Voxels are usually presented in the shape of cubes with three dimensions of width, height and depth, and are used to discretize continuous three-dimensional space. Each voxel represents a specific volume area in space. When voxel division is performed on a point cloud, the entire three-dimensional space is divided into several voxels, and several voxels form a three-dimensional grid, that is, a voxel set. Each voxel can be used to store data in three-dimensional space, such as point cloud data, medical imaging data (CT, MRI) and other data, and each voxel can include a variety of feature information. For example, for a point cloud, a voxel can include feature information such as the number of points in the volume of the voxel block, the average color or normal. The size of the voxel (i.e., the resolution of the voxel) can determine the degree of refinement of the three-dimensional space discretization and the amount of data. Smaller voxels can provide a more detailed spatial representation.

[0063] There are many ways to perform voxel division on the obtained point cloud. As an example, the planes formed by the point cloud along the first axis and the second axis can be divided according to at least two voxel parameters, thereby obtaining at least two voxel sets corresponding to the at least two voxel parameters, wherein the at least two voxel sets contain different numbers of voxels.

[0064] The above point cloud itself can have at least three axes in three-dimensional space, such as the orthogonal X-axis, Y-axis and Z-axis. As an example, the first axis and the second axis can be the X-axis and the Y-axis respectively, that is, the plane formed by the point cloud along the X-axis and the Y-axis can be divided according to at least two voxel parameters, and at least two voxel sets corresponding to the at least two voxel parameters can be obtained. It can be understood that in the at least two voxel sets obtained by dividing the same point cloud, the volume of each voxel is determined by the width of the voxel on the X-axis, the Y-axis and the Z-axis respectively.

[0065] Considering that in the related art, the division interval during voxel division is often fixed, that is, for the same point cloud, only one fixed division interval is used to perform voxel division on the point cloud. However, taking a fixed scanning scene as an example, the point cloud includes target objects and non-target objects. The target objects are multiple bases, implants and / or scanning rods (implant scanning bodies), and the non-target objects are multiple plasticines used to fix the target objects. From the above introduction, it can be seen that the shapes, sizes and positions of the multiple target objects and multiple plasticines on the same bracket may be different. The fixed division interval is difficult to adapt to such differences. In addition, when facing point clouds of different sizes, the fixed division interval is also difficult to adapt to changes in the size of the point cloud, resulting in the recognition accuracy being relatively low.

[0066] The above-mentioned voxel parameters can be implemented in many ways. As an example, the voxel parameter can be a division interval ratio, which can be used to describe the ratio of the width of the voxel in the width of the point cloud; as another example, the plane composed of the point cloud along the first axial direction and the second axial direction is divided, and specifically, the width of the voxel in the first axial direction and the width of the point cloud in the second axial direction are determined based on the division interval ratio, the width of the point cloud in the first axial direction and the width of the point cloud in the second axial direction; based on the width of the voxel in the first axial direction and the width of the second axial direction, the plane composed of the point cloud along the first axial direction and the second axial direction is divided. The point cloud is divided according to the ratio. When the same ratio is used for dividing point clouds of different sizes, the number of divided voxels will not be affected by the size of the point cloud, that is, it is insensitive to the size range of the input point cloud data and can accept point cloud data of any size range. When the same size of point cloud is divided by multiple ratios of different sizes, different ratios have different division receptive fields, and the features of different receptive fields can complement each other. The features of large receptive fields have more global information and can better detect larger target objects, while the features of small receptive fields have more detailed local information and can better locate smaller target objects, thereby improving the accuracy of identifying target objects as a whole.

[0067] As an example, when the voxel parameter is a division interval ratio, the at least two voxel parameters may be three division interval ratios, which may include: 1 / 400, 1 / 200, 1 / 100. The division interval ratio may also be other values, which are not specifically limited.

[0068] Below Figure 3A , Figure 3B , Figure 4 Taking the example of FIG. 1 as an example, the voxel division of the embodiment of the present application is exemplarily introduced:

[0069] See also Figure 3A For the same point cloud, three voxel parameters (i.e. Figure 3A The planes formed by the point cloud along the X-axis and the Y-axis are divided into three types (voxel set A, voxel set B, voxel set C) respectively, and three types of voxel sets corresponding to the three types of division interval ratios are obtained. Obviously, the number of voxels contained in the three types of voxel sets is different. Figure 3AThe widths of the same point cloud corresponding to the three voxel sets are the same, and the division interval ratios corresponding to the three voxel sets can describe the ratio of the voxel width to the width of the point cloud. For any voxel set, after knowing the division interval ratio corresponding to the voxel set, the width of the point cloud on the X axis, and the width of the point cloud on the Y axis, the width of any voxel in the voxel set in the first axis, the width of the X axis, and the width of the Y axis can be determined, so that the plane formed by the point cloud along the X axis and the Y axis can be divided based on the width of any voxel in the voxel set in the first axis and the width of the second axis. See Figure 3B It can be understood that after dividing the point cloud along the plane formed by the X-axis and the Y-axis, the point cloud data can be enclosed in a voxel block with a volume (determined by the width of the voxel corresponding to the X-axis, Y-axis and Z-axis respectively) formed along the Z-axis direction, that is, the voxel is the result of dimensional compression of the point cloud in the Z-axis direction, and the result can be used as the basis for converting three-dimensional point cloud data into a two-dimensional pseudo image.

[0070] As an example, when the voxel parameter is a division interval ratio, the at least two voxel parameters may be three division interval ratios, and the three division interval ratios may include: 1 / 400, 1 / 200, and 1 / 100.

[0071] See also Figure 4, for the acquired point cloud, it contains the three-dimensional coordinates [x, y, z] and normal coordinates [nx, ny, nz] of each point. When the voxel parameter is the division interval ratio, it is assumed that the point cloud is divided into voxels along the plane composed of the X and Y axes using three voxel parameters. The three voxel parameters can be defined as a set of division interval ratios, which are [1 / k1, 1 / k2, 1 / k3]. Assuming k1=400, k2=200, k3=100, the three voxel parameters are 1 / 400, 1 / 200, and 1 / 100, respectively. In the three different division methods corresponding to the three voxel parameters, the sum of the number of voxel grids whose point cloud number is not zero in the voxel grid is recorded as [p1, p2, p3], and the voxel coordinates of each voxel grid whose point cloud number is not zero in the original voxel feature are recorded. Assuming that the size of the acquired point cloud is: X-axis range [-48, +48], Y-axis range [-48, +48], Z-axis range [-8, +8], then according to the three division interval ratios of 1 / 400, 1 / 200, and 1 / 100, the actual voxel division interval along the X-axis is [0.24, 0.48, 0.96], the actual voxel division interval along the Y-axis is [0.24, 0.48, 0.96], and the actual voxel division interval along the Z-axis is

[16] . In order to realize the subsequent convolution process, it is necessary to limit the number of point clouds in each voxel grid corresponding to the three division interval ratios. Assuming that the maximum number of point clouds in the voxel corresponding to the three division interval ratios does not exceed [n1, n2, n3], if the number of point clouds in the voxel actually exceeds the given value, the number of point clouds can be downsampled to the given value, and the number less than the given value can be filled with 0 to ensure that the number of point clouds in each divided voxel grid corresponding to the same division interval ratio remains consistent. As an example, it can be assumed that n1=16, n2=24, and n3=32.

[0072] There are many ways to extract three-dimensional features from each voxel in each voxel set. As an example, three-dimensional features may be extracted and aggregated from the three-dimensional points contained in each voxel in each voxel set.

[0073] In some embodiments, extracting and aggregating three-dimensional features of three-dimensional points contained in each voxel in each voxel set may include: Figure 5 Steps shown:

[0074] For the three-dimensional points contained in each voxel:

[0075] S501, extracting three-dimensional features based on three-dimensional position information of three-dimensional points in the same voxel to obtain first-category three-dimensional features of the three-dimensional points in the same voxel;

[0076] S502, extracting three-dimensional features based on the first three-dimensional features of the three-dimensional points in the same voxel to obtain second three-dimensional features of the three-dimensional points in the same voxel;

[0077] S503 , aggregate the second type of three-dimensional features of each three-dimensional point in the same voxel to obtain the voxel features of the same voxel.

[0078] The first type of three-dimensional features can have multiple specific implementations. As an example, the first type of three-dimensional features may include the three-dimensional coordinates of a three-dimensional point, a normal vector, a distance from the center of the point cloud within the voxel, and a distance from the geometric center of the voxel; there is no limitation on the specific implementation of the first type of three-dimensional features.

[0079] The second type of three-dimensional features may be implemented in a variety of ways. As an example, the second type of three-dimensional features may be features obtained by performing feature dimension upgrade on the first type of three-dimensional features. The specific implementation of the first type of three-dimensional features is not limited.

[0080] Below Figure 6 Taking the example of FIG. 1 as an example, the three-dimensional feature extraction of the embodiment of the present application is exemplarily introduced:

[0081] See also Figure 6 , taking three different partitioning interval ratios as examples, the voxel partitioning is performed at the three partitioning interval ratios, and the three-dimensional points contained in each voxel in the three voxel sets divided are subjected to three-dimensional feature extraction and aggregation, wherein the feature extraction can be divided into at least a manual feature design part and an automatic feature learning part. For the three-dimensional points contained in each voxel:

[0082] The manual feature design part is used to extract the first type of three-dimensional features of the three-dimensional points. Specifically, the manual feature design part can extract the d-dimensional features of the point cloud in each voxel grid. Assuming d=12, the d-dimensional features can be the three-dimensional coordinates of the three-dimensional point [x, y, z], the normal vector of the three-dimensional point [nx, ny, nz], the distance between the three-dimensional point and the center of the point cloud in the voxel [xc, yc, zc], and the distance between the three-dimensional point and the geometric center of the voxel [xp, yp, zp]. For the three voxel sets, after the feature extraction of the manual feature design part, three groups of first-type three-dimensional features [p1*n1*d, p2*n2*d, p3*n3*d] corresponding to the three voxel sets can be obtained.

[0083] The automatic feature learning part is used to extract the second type of three-dimensional features of the three-dimensional points based on the first type of three-dimensional features of the three-dimensional points. The automatic feature learning part can upgrade the d-dimensional features of the point cloud in each voxel grid based on the pointnet grid structure. For three voxel sets, the d-dimensional features are upgraded to [c1, c2, c3] respectively. As an example, c1=64, c2=128, c3=256.

[0084] After the feature extraction of the manual feature design part and the automatic feature learning part, the second-type three-dimensional features of each three-dimensional point in the same voxel are finally obtained. The second-type three-dimensional features of each three-dimensional point in the same voxel can be aggregated by global pooling to obtain the voxel features of the same voxel. The voxel features of the same voxel are the features that characterize the significant features of the voxel as a whole. For three voxel sets, three groups of voxel features corresponding to the three voxel sets can be obtained [p1*c1, p2*c2, p3*c3].

[0085] In some embodiments, after aggregating the second type of three-dimensional features of each three-dimensional point in the same voxel, the obtained voxel features of the same voxel may include sub-features of at least two channels, and based on the voxel features of each voxel in each voxel set, converting the voxel set into a two-dimensional pseudo image may include the following: Figure 7 Steps shown:

[0086] S701 , performing pooling processing on sub-features of the same channel of each voxel in the same voxel set to obtain features of the same channel, and mapping the features of the same channel to channel weights.

[0087] The channel weight is used to represent the importance of the sub-features of the same channel in the sub-features of at least two channels.

[0088] S702 , fusing sub-features of at least two channels based on channel weights to obtain voxel features after voxel fusion.

[0089] S703 : Based on the fused voxel features of each voxel in each voxel set, convert the voxel set into a two-dimensional pseudo image.

[0090] In some embodiments, based on the voxel features of each voxel in each voxel set, converting the voxel set into a two-dimensional pseudo image may further include: Figure 8 Steps shown:

[0091] S801 , performing channel pooling processing on the voxel features of each voxel in the same voxel set to obtain the position features of each voxel, and mapping the position features of each voxel to a position weight.

[0092] The position weight is used to characterize the importance of the position feature of a voxel among the position features of each voxel in the voxel set.

[0093] S802, fusing the voxel feature of the voxel with the position weight corresponding to the voxel to obtain the voxel feature after voxel fusion;

[0094] S803 : Based on the fused voxel features of each voxel in each voxel set, convert the voxel set into a two-dimensional pseudo image.

[0095] It can be understood that the fused voxel features of each voxel in each voxel set can be the voxel features obtained by fusing only the sub-features of at least two channels, or the voxel features obtained by fusing only the voxel features of the voxels with the position weights corresponding to the voxels, or the voxel features obtained by fusing the sub-features of at least two channels and fusing the voxel features with the position weights corresponding to the voxels. There is no specific limitation on this.

[0096] When considering converting a three-dimensional point cloud into a two-dimensional pseudo image in related technologies, the three-dimensional point cloud is directly converted into a two-dimensional pseudo image by means of data filling and dimensional transformation. This approach lacks the global correlation between voxels and cannot obtain sufficient global receptive field information.

[0097] In response to the above problems, the following Fig. 9 Taking the example of FIG. 1 as an example, the process of converting a voxel set into a two-dimensional pseudo image in an embodiment of the present application is exemplarily introduced:

[0098] See also Fig. 9 According to the above description, taking three voxel sets as an example, for the three voxel sets, after obtaining the three sets of voxel features [p1*c1, p2*c2, p3*c3] corresponding to the three voxel sets, firstly convert the voxel features [p1*c1, p2*c2, p3*c3] (i.e. Fig. 9 In the “input 3D features”) is restored to three sets of 2D pseudo images [w1, w2, w3], where the size of w1 is 1*c1*k1*k1, the size of w2 is 1*c2*k2*k2, and the size of w3 is 1*c3*k3*k3. In the case where the voxel feature of the same voxel includes sub-features of at least two channels, the sub-features of the same channel of each voxel in the same voxel set can be pooled (i.e. Fig. 9 The features of the same channel are obtained by performing channel pooling (i.e., global channel pooling) and then using an S-type activation function (such as a sigmoid activation function) to map the features of the same channel to channel weights. The sub-features of at least two channels are then fused based on the channel weights. The voxel features of each voxel in the same voxel set are processed by channel pooling (i.e., Fig. 9 The position feature of each voxel is obtained by using the "global adaptive pooling" in the above method, and the position feature of each voxel is mapped to a position weight by using the sigmoid activation function. Then, the voxel feature of the voxel is fused with the position weight corresponding to the voxel. Based on the voxel feature obtained by fusing the sub-features of at least two channels and fusing the voxel feature with the position weight corresponding to the voxel, the voxel set is converted into a two-dimensional pseudo image, that is, Fig. 9The "pixel-level fusion" process finally obtains a two-dimensional pseudo image with global feature information expression. It can further enhance useful information and weaken useless information from a global perspective, and output the features of the pseudo image that are more consistent with two-dimensional calculations.

[0099] As an example, the sub-features of at least two channels included in the voxel feature of the same voxel may include at least sub-features of the three channels [x, y, z] in the voxel coordinates, and may also include sub-features of other channels, without specific limitation.

[0100] In some embodiments, the at least two voxel parameters may include a first voxel parameter and a second voxel parameter, and the at least two voxel sets may include a first voxel set corresponding to the first voxel parameter and a second voxel set corresponding to the second voxel parameter; wherein the volume of the voxels in the first voxel set is greater than the volume of the voxels in the second voxel set, and identifying the target object based on the fusion result of the two-dimensional features extracted from the at least two two-dimensional pseudo images may include: Fig.10 Steps shown:

[0101] S1001, performing two-dimensional feature extraction on the two-dimensional pseudo image corresponding to the first voxel set to obtain a first two-dimensional feature; and performing two-dimensional feature extraction on the two-dimensional pseudo image corresponding to the second voxel set to obtain a second two-dimensional feature;

[0102] S1002, downsampling the size of the first two-dimensional feature to the size of the second two-dimensional feature to obtain a first intermediate feature, and fusing the first intermediate feature with the second two-dimensional feature to obtain a fused feature;

[0103] S1003, upsampling the size of the fused feature to the size of the first two-dimensional feature to obtain a second intermediate feature, and fusing the second intermediate feature with the first two-dimensional feature to obtain a fusion result;

[0104] S1004: Identify the target object based on the fusion result.

[0105] Below Fig.11 , Fig.12 , Fig.13A , Fig. 13B Taking the example of FIG. 1 as an example, the process of fusing the two-dimensional features extracted from at least two two-dimensional pseudo images in the embodiment of the present application and identifying the target object by fusing the results is exemplarily introduced:

[0106] Taking the example that the at least two voxel parameters include the first voxel parameter, the second voxel parameter and the third voxel parameter, the at least two voxel sets may include a first voxel set corresponding to the first voxel parameter, a second voxel set corresponding to the second voxel parameter and a third voxel set corresponding to the third voxel parameter; wherein the volume of the voxels in the first voxel set may be greater than the volume of the voxels in the second voxel set, and the volume of the voxels in the second voxel set may be greater than the volume of the voxels in the third voxel set, and a two-dimensional pseudo image w1 corresponding to the first voxel set, a two-dimensional pseudo image w2 corresponding to the second voxel set, and a two-dimensional pseudo image w3 corresponding to the third voxel set are set, see Fig.11 First, two-dimensional feature extraction can be performed on w1 to obtain two-dimensional feature m1. After m1 is downsampled, it is fused with the two-dimensional features extracted from the pseudo image w2 to obtain fused feature m2. Feature extraction and downsampling of fused feature m2 are performed and fused with the pseudo image w3 to obtain fused feature m3, where the feature size of m1 is 1*c1*k1*k1, the feature size of m2 is 1*c2*k2*k2, and the feature size of m3 is 1*c3*k3*k3. Finally, m3 and m2 are upsampled to the size of m1 and fused with m1 to obtain the final fusion result, i.e., the two-dimensional coding feature. A network architecture similar to a segmentation network (such as a unet segmentation network) is used to realize the cascade fusion of two-dimensional features of two-dimensional pseudo images corresponding to voxel sets of different voxel volumes, which provides rich original feature information while simplifying the original unet upsampling step and improving the detection efficiency.

[0107] See also Fig.12 As an example, a detection decoding head may be used to decode the above two-dimensional coding features to obtain a recognition result for the target object; see Fig.13A , Fig. 13BAs an example, the recognition result of the target object may include the three-dimensional bounding box information of the target object, as well as the category label and confidence information of each target object; as another example, the three-dimensional bounding box information may be represented in the form of center point coordinates x, y, z and length, width and height l, w, h. As another example, the detection decoding head may adopt an anchor-free decoding method that does not rely on a priori anchor box (anchor) and predict the recognition result in the form of a heat map (heatmap, hm); as another example, the detection decoding head may include 4 branches, namely, hm branch, first regression (regression, reg) branch, second regression (height) branch, object size (dimesion, dim) branch, where the hm branch is responsible for predicting whether there is a target object (confidence) and the target The category of the target object has a size of 1*c*k1*k1, where c is the number of set categories; the reg branch is responsible for predicting the x-axis and y-axis coordinate information of the target object, that is, regressing the x and y parameters of the target object, with a size of 1*2*k1*k1; the height branch is responsible for predicting the z-axis coordinate information of the target object, that is, regressing the z parameters of the target object, with a size of 1*1*k1*k1; the dim branch is responsible for predicting the size information of the target object, that is, the length, width and height information, with a size of 1*3*k1*k1; as another example, the threshold of the above categories can be set to 0.1, and the intersection over union (IoU) threshold can be set to 0.7.

[0108] Corresponding to the above-mentioned object detection method embodiment, the present application embodiment also provides a method for training an object recognition model, such as Fig.14 As shown, the method comprises the following steps:

[0109] S1401. Obtain a sample data set.

[0110] The sample data set includes several samples, each of which includes a point cloud and a target object recognition result.

[0111] S1402: Input the sample data set into the object recognition model to be trained for supervised training to obtain a trained object recognition model.

[0112] The trained object recognition model can be applied to the object detection method described in any of the above embodiments, and the details will not be repeated here.

[0113] As an example, in a fixed scanning scenario, point clouds of target objects such as abutments, implants and / or scanning rods (implant scanning bodies) of different shapes and sizes and data related to recognition results can be collected. The abutments, implants and / or scanning rods (implant scanning bodies) are fixed on the scanning plane of the tray using plasticine. The area size and height of the plasticine are not limited, so as to simulate different scanning scenarios as randomly as possible to obtain the above-mentioned sample data set.

[0114] Corresponding to the above method embodiment, the present application embodiment also provides an object detection device, see Fig.15 As shown, the device may include:

[0115] An acquisition unit 1501 is used to acquire a point cloud obtained by scanning a target scene, where the target scene includes a target object to be identified;

[0116] A division unit 1502 is used to perform voxel division on the point cloud according to at least two voxel parameters to obtain at least two voxel sets corresponding to the at least two voxel parameters;

[0117] The conversion unit 1503 is used to extract three-dimensional features from each voxel in each voxel set to obtain voxel features of the voxel, and convert the voxel set into a two-dimensional pseudo image based on the voxel features of each voxel in each voxel set;

[0118] The recognition unit 1504 is configured to recognize a target object based on a fusion result of two-dimensional features respectively extracted from at least two two-dimensional pseudo images.

[0119] As an example, the division unit 1502 is specifically configured to divide the planes formed by the point cloud along the first axis and the second axis according to at least two voxel parameters, and the at least two voxel sets contain different numbers of voxels.

[0120] As an example, the conversion unit 1503 is specifically used to extract and aggregate three-dimensional features of the three-dimensional points contained in each voxel in each voxel set.

[0121] As an example, the voxel parameter is a division interval ratio, which is used to describe the ratio of the width of the voxel in the width of the point cloud; the division unit 1502 is specifically used to determine the width of the voxel in the first axis and the width of the point cloud in the second axis based on the division interval ratio, the width of the point cloud in the first axis and the width of the point cloud in the second axis; based on the width of the voxel in the first axis and the width of the second axis, the plane composed of the point cloud along the first axis and the second axis is divided.

[0122] As an example, the at least two voxel parameters are three division interval ratios, and the three division interval ratios include: 1 / 400, 1 / 200, and 1 / 100.

[0123] As an example, the conversion unit 1503 is specifically used to: for the three-dimensional points contained in each voxel, perform three-dimensional feature extraction based on the three-dimensional position information of the three-dimensional points within the same voxel to obtain the first type of three-dimensional features of the three-dimensional points within the same voxel; perform three-dimensional feature extraction based on the first type of three-dimensional features of the three-dimensional points within the same voxel to obtain the second type of three-dimensional features of the three-dimensional points within the same voxel; aggregate the second type of three-dimensional features of each three-dimensional point within the same voxel to obtain the voxel features of the same voxel.

[0124] As an example, the first type of three-dimensional features include the three-dimensional coordinates of the three-dimensional point, the normal vector, the distance to the point cloud center within the voxel, and the distance to the geometric center of the voxel; the second type of three-dimensional features are the features obtained by upgrading the first type of three-dimensional features.

[0125] As an example, the voxel features of a voxel include sub-features of at least two channels; the conversion unit 1503 is specifically used to perform pooling processing on the sub-features of the same channel of each voxel in the same voxel set to obtain the features of the same channel, and map the features of the same channel to channel weights, which are used to characterize the importance of the sub-features of the same channel in the sub-features of at least two channels; based on the channel weights, the sub-features of at least two channels are fused to obtain the voxel features after the voxel fusion; based on the fused voxel features of each voxel in each voxel set, the voxel set is converted into a two-dimensional pseudo image.

[0126] As an example, the conversion unit 1503 is specifically used to perform channel pooling processing on the voxel features of each voxel in the same voxel set to obtain the position features of each voxel, and map the position features of each voxel to a position weight, which is used to characterize the importance of the position feature of the voxel in the position features of each voxel in the voxel set; fuse the voxel features of the voxel with the position weight corresponding to the voxel to obtain the voxel features after the voxel fusion; based on the fused voxel features of each voxel in each voxel set, convert the voxel set into a two-dimensional pseudo image.

[0127] As an example, at least two voxel parameters include a first voxel parameter and a second voxel parameter, and at least two voxel sets include a first voxel set corresponding to the first voxel parameter and a second voxel set corresponding to the second voxel parameter; wherein the volume of the voxels in the first voxel set is greater than the volume of the voxels in the second voxel set; the identification unit 1504 is specifically used to perform two-dimensional feature extraction on the two-dimensional pseudo image corresponding to the first voxel set to obtain a first two-dimensional feature; and perform two-dimensional feature extraction on the two-dimensional pseudo image corresponding to the second voxel set to obtain a second two-dimensional feature; downsample the size of the first two-dimensional feature to the size of the second two-dimensional feature to obtain a first intermediate feature, and fuse the first intermediate feature with the second two-dimensional feature to obtain a fused feature; upsample the size of the fused feature to the size of the first two-dimensional feature to obtain a second intermediate feature, and fuse the second intermediate feature with the first two-dimensional feature to obtain a fused result; and identify the target object based on the fusion result.

[0128] As an example, the above-mentioned object detection device is applied to a fixed scanning scene, and the target scene is a scene including a target object to be identified and a non-target object used to fix the target object; the target object is a base, implant and / or scanning rod (implant scanning body) in an oral medical scene.

[0129] The present application also provides an electronic device, such as Fig.16 As shown, the electronic device includes: a processor 1601; a memory 1602 for storing processor executable instructions; wherein the processor 1601 is configured to implement the object detection method or the object recognition model training method described in any of the above embodiments.

[0130] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the object detection method or the object recognition model training method described in any of the above embodiments is implemented.

[0131] The above is only a specific implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An object detection method, characterized in that: The method comprises: Acquire a point cloud obtained by scanning a target scene, wherein the target scene includes a target object to be identified; The plane formed by the point cloud along the first axial direction and the second axial direction is divided according to at least two voxel parameters to obtain at least two voxel sets corresponding to the at least two voxel parameters, and the at least two voxel sets respectively contain different numbers of voxels; the at least two voxel parameters include a first voxel parameter and a second voxel parameter, and the at least two voxel sets include a first voxel set corresponding to the first voxel parameter and a second voxel set corresponding to the second voxel parameter; wherein the volume of the voxels in the first voxel set is greater than the volume of the voxels in the second voxel set; For the three-dimensional points contained in each voxel in each of the voxel sets: Extracting three-dimensional features based on the three-dimensional position information of the three-dimensional points in the same voxel to obtain a first type of three-dimensional features of the three-dimensional points in the same voxel; Extracting three-dimensional features based on the first three-dimensional features of the three-dimensional points in the same voxel to obtain second three-dimensional features of the three-dimensional points in the same voxel; Aggregating the second type of three-dimensional features of each three-dimensional point in the same voxel to obtain the voxel features of the same voxel, and converting the voxel set into a two-dimensional pseudo image based on the voxel features of each voxel in each voxel set; wherein the first type of three-dimensional features include the three-dimensional coordinates of the three-dimensional point, the normal vector, the distance from the center of the point cloud in the voxel, and the distance from the geometric center of the voxel; the second type of three-dimensional features are the features obtained by performing feature dimensionality upgrade on the first type of three-dimensional features; Performing two-dimensional feature extraction on the two-dimensional pseudo image corresponding to the first voxel set to obtain a first two-dimensional feature; and performing two-dimensional feature extraction on the two-dimensional pseudo image corresponding to the second voxel set to obtain a second two-dimensional feature; Downsampling the size of the first two-dimensional feature to the size of the second two-dimensional feature to obtain a first intermediate feature, and fusing the first intermediate feature with the second two-dimensional feature to obtain a fused feature; Upsampling the size of the fused feature to the size of the first two-dimensional feature to obtain a second intermediate feature, and fusing the second intermediate feature with the first two-dimensional feature to obtain the fusion result; The target object is identified based on the fusion result.

2. The method according to claim 1, characterized in that: After acquiring the point cloud obtained by scanning the target scene, the step in the method is to input the point cloud into a pre-trained object recognition model for execution by the object recognition model.

3. The method according to claim 1, characterized in that The voxel parameter is a division interval ratio, and the division interval ratio is used to describe the ratio of the width of the voxel in the width of the point cloud; The dividing of the planes formed by the point cloud along the first axial direction and the second axial direction according to at least two voxel parameters comprises: Determine the width of the voxel in the first axial direction and the width of the voxel in the second axial direction based on the division interval ratio, the width of the point cloud in the first axial direction and the width of the point cloud in the second axial direction; The plane formed by the point cloud along the first axial direction and the second axial direction is divided based on the width of the voxel in the first axial direction and the width of the voxel in the second axial direction.

4. The method according to claim 3, characterized in that The at least two voxel parameters are three division interval ratios, and the three division interval ratios include: 1 / 400, 1 / 200, and 1 / 100.

5. The method according to claim 1, characterized in that: The voxel feature of the voxel includes sub-features of at least two channels; and based on the voxel feature of each voxel in each of the voxel sets, converting the voxel sets into a two-dimensional pseudo image comprises: Performing pooling processing on the sub-features of the same channel of each voxel in the same voxel set to obtain the features of the same channel, and mapping the features of the same channel into channel weights, where the channel weights are used to characterize the importance of the sub-features of the same channel in the sub-features of the at least two channels; fusing the sub-features of the at least two channels based on the channel weights to obtain the voxel-fused voxel features; Based on the fused voxel features of each voxel in each of the voxel sets, converting the voxel sets into a two-dimensional pseudo image, and / or The step of converting the voxel sets into a two-dimensional pseudo image based on the voxel features of each voxel in each voxel set further comprises: Channel pooling is performed on the voxel features of each voxel in the same voxel set to obtain the position features of each voxel, and the position features of each voxel are mapped to a position weight, which is used to characterize the importance of the position feature of the voxel in the position features of each voxel in the voxel set; Fusing the voxel feature of the voxel with the position weight corresponding to the voxel to obtain the voxel feature after the voxel fusion; Based on the fused voxel features of each voxel in each of the voxel sets, the voxel sets are converted into a two-dimensional pseudo image.

6. The method according to claim 1, characterized in that The method is applied to a fixed scanning scene, wherein the target scene is a scene including the target object to be identified and a non-target object used to fix the target object; the target object is a base, an implant and / or a scanning rod in an oral medical scene.

7. A method for training an object recognition model, characterized in that: include: Acquire a sample data set, wherein the sample data set includes a plurality of samples, each sample including a point cloud and a target object recognition result; Inputting the sample data set into the object recognition model to be trained for supervised training to obtain a trained object recognition model; The object recognition model is applied to the object detection method according to any one of claims 1 to 6.

8. An object detection device, characterized in that: include: An acquisition unit, configured to acquire a point cloud obtained by scanning a target scene, wherein the target scene includes a target object to be identified; A division unit, which divides the plane formed by the point cloud along the first axial direction and the second axial direction according to at least two voxel parameters, to obtain at least two voxel sets corresponding to the at least two voxel parameters, respectively, and the at least two voxel sets respectively contain different numbers of voxels; the at least two voxel parameters include a first voxel parameter and a second voxel parameter, and the at least two voxel sets include a first voxel set corresponding to the first voxel parameter and a second voxel set corresponding to the second voxel parameter; wherein the volume of the voxels in the first voxel set is greater than the volume of the voxels in the second voxel set; A conversion unit, for a three-dimensional point contained in each voxel in each of the voxel sets: Extracting three-dimensional features based on the three-dimensional position information of the three-dimensional points in the same voxel to obtain a first type of three-dimensional features of the three-dimensional points in the same voxel; Extracting three-dimensional features based on the first three-dimensional features of the three-dimensional points in the same voxel to obtain second three-dimensional features of the three-dimensional points in the same voxel; Aggregating the second type of three-dimensional features of each three-dimensional point in the same voxel to obtain the voxel features of the same voxel, and converting the voxel set into a two-dimensional pseudo image based on the voxel features of each voxel in each voxel set; wherein the first type of three-dimensional features include the three-dimensional coordinates of the three-dimensional point, the normal vector, the distance from the center of the point cloud in the voxel, and the distance from the geometric center of the voxel; the second type of three-dimensional features are the features obtained after the first type of three-dimensional features are upgraded; The recognition unit is used to perform two-dimensional feature extraction on the two-dimensional pseudo image corresponding to the first voxel set to obtain a first two-dimensional feature; and perform two-dimensional feature extraction on the two-dimensional pseudo image corresponding to the second voxel set to obtain a second two-dimensional feature; downsample the size of the first two-dimensional feature to the size of the second two-dimensional feature to obtain a first intermediate feature, and fuse the first intermediate feature with the second two-dimensional feature to obtain a fused feature; upsample the size of the fused feature to the size of the first two-dimensional feature to obtain a second intermediate feature, and fuse the second intermediate feature with the first two-dimensional feature to obtain the fused result; and identify the target object based on the fused result.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images