Method, device, storage medium and electronic device for generating object spatial information label

Abstract

This application discloses a method for generating object spatial information labels, including: using a scene video including a target object captured from multiple angles to generate a scene point cloud and image sensor pose change information at each moment; aligning the target object's three-dimensional model to the scene point cloud based on the target object's feature points selected in the scene point cloud to obtain the target object's pose change information at the initial moment in the image sensor coordinate system; determining the target object's pose change information at each moment in the image sensor coordinate system based on the image sensor's pose change information at each moment and the target object's pose change information at the initial moment, and using the target object's pose change information at each moment as the pose label of the target object at the corresponding moment. Application of this application can improve the efficiency of generating object spatial information labels, thereby improving the efficiency of generating training data.

Description

Technical Field

[0001] The present application relates to computer vision technology, and in particular to a method, device, storage medium, and electronic device for generating object spatial information tags. Background Art

[0002] Computer vision-based object pose estimation is a key technology for enabling robotic grasping and dexterous manipulation. It is crucial for improving robots' adaptability to various environments and tasks, broadening their application areas, and enhancing their flexibility and effectiveness in scenarios such as intelligent manufacturing, warehousing and logistics, and home services. Furthermore, this technology holds broad application prospects in areas such as autonomous driving, augmented reality, and virtual reality.

[0003] In recent years, with the rapid development of deep learning technology, deep learning-based object pose estimation has achieved promising results. In unstructured scenarios such as background clutter, stacked objects, and varying lighting, deep learning methods outperform traditional pose estimation methods in terms of robustness, accuracy, and real-time performance. However, deep learning methods are data-driven algorithms, and to achieve ideal results, they require a large amount of training data with spatial labels. However, in the field of object pose estimation, manual labeling is currently commonly used to generate spatial labels for objects, which are then used to obtain training data with spatial labels. Obviously, this method of generating labels and training data is very inefficient. Summary of the Invention

[0004] The present application provides a method, device, storage medium and electronic device for generating object space information labels, which can improve the generation efficiency of object space information labels and thereby improve the generation efficiency of training data.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] A method for generating a spatial information label of an object, comprising:

[0007] Using a scene video including a target object captured from multiple angles, a scene point cloud and image sensor posture change information representing the image sensor posture change at each moment are generated; wherein the scene point cloud includes a point set of the scene captured at each moment;

[0008] Using the target object feature points selected from the scene point cloud, aligning the target object three-dimensional model with the scene point cloud to obtain the target object pose change information at the initial moment in the image sensor coordinate system;

[0009] Determine the pose change information of the target object at each moment in the image sensor coordinate system according to the pose change information of the image sensor at each moment and the pose change information of the target object at the initial moment;

[0010] Determine the pose change information of the target object at each moment as the pose label of the target object at the corresponding moment.

[0011] Preferably, it is characterized in that the determination of the pose change information of the target object at time t i in the image sensor coordinate system (R ti | T ti ) includes:

[0012] (R ti | T ti ) = (R ti-1 ×R ci-1,ci | R ti-1 ×T ci-1,ci +T ti-1 ),

[0013] where, (R ti-1，ti | T ti-1,ti ) is the rotation matrix of the pose change of the image sensor from time t i-1 to time t i , and is the translation vector of the pose change of the image sensor from time t i-1 to time t i , and 0 < i ≤ n is a positive integer.

[0014] Preferably, the registration of the three-dimensional model of the target object to the scene point cloud includes:

[0015] a. Use the target object feature points selected in the scene point cloud and the feature points selected on the three-dimensional model of the target object corresponding to the target object feature points to construct a least squares problem, and obtain the rough-precision pose change information that minimizes the sum of the squared errors between the feature points on the three-dimensional model mapped to the points in the image sensor coordinate system and the target object feature points selected in the scene point cloud;

[0016] b. Map the three-dimensional model of the target object to the scene point cloud using the rough-precision pose change information. When the coincidence degree of the mapping result and the scene point cloud meets the set requirements, obtain all the points within the set range of the mapping result in the scene point cloud, and form a target object surface point cloud point set S for fine registration, and perform the registration of the three-dimensional model of the target object to the target object surface point cloud point set S; when the coincidence degree of the mapping result and the scene point cloud does not meet the set requirements, reselect the target object feature points and return to step a.

[0017] Preferably, after determining the pose label of the target object, the method further comprises:

[0018] The three-dimensional model of the target object is adjusted in position according to the position change information of the target object at each moment, and the two-dimensional label of the target object at each moment is determined.

[0019] Preferably, determining the two-dimensional label of the target object at each moment includes:

[0020] Perform two-dimensional projection on the three-dimensional model of the target object at each moment after the posture adjustment;

[0021] The projection area at each moment obtained by projection is determined as the two-dimensional pixel-level segmentation label of the target object at the corresponding moment;

[0022] Taking a minimum bounding rectangle of the projection area at each moment, and determining the minimum bounding rectangle as a two-dimensional target detection label of the target object at the corresponding moment;

[0023] Set the category label of the target object based on user input.

[0024] A device for generating a spatial information label of an object, comprising: a memory and a processor;

[0025] The memory stores scene videos including the target object collected from multiple angles;

[0026] The processor uses the scene video including the target object stored in the memory to generate a scene point cloud and the posture change information of the image sensor representing the position change of the image sensor at each moment; uses the target object feature points selected in the scene point cloud to align the three-dimensional model of the target object to the scene point cloud, and determines the posture change information obtained by the alignment as the posture change information of the target object at the initial moment in the image sensor coordinate system; determines the posture change information of the target object at each moment in the image sensor coordinate system based on the posture change information of the image sensor at each moment and the posture change information of the target object at the initial moment, and determines the posture change information of the target object at each moment as the posture label of the target object at the corresponding moment; wherein, the scene point cloud includes a point set of the captured scene at each moment.

[0027] Preferably, the generating device further includes the image sensor, which collects scene videos including the target object from multiple angles and saves them in the memory.

[0028] Preferably, in the processor, the time t in the image sensor coordinate system is determined i The target object pose change information (R ti |Tti ) includes:

[0029] (R ti |T ti )(t) = (R ti-1 × R ci-1,ci |R ti-1 × T ci-1,ci + T ti-1 )(t),

[0030] where (R ti-1，ti |T ti-1,ti )(t) is the rotation matrix of the pose change of the image sensor from time t i-1 to time t i , and is the translation vector of the pose change of the image sensor from time t i-1 to time t i , and 0 < i ≤ n is a positive integer.

[0031] Preferably, in the processor, the registration of the three-dimensional model of the target object to the scene point cloud includes:

[0032] a. Using the feature points of the target object selected in the scene point cloud and the corresponding feature points selected on the three-dimensional model of the target object, construct a least-squares problem to obtain the rough-precision pose change information that minimizes the sum of the squared errors between the points on the three-dimensional model mapped to the points in the image sensor coordinate system and the feature points of the target object selected in the scene point cloud;

[0033] b. Map the three-dimensional model of the target object to the scene point cloud using the rough-precision pose change information. When the coincidence degree of the mapping result and the scene point cloud meets the set requirements, obtain all the points within the set range of the mapping result in the scene point cloud to form a point set S of the target object surface point cloud for fine registration, and perform the registration of the three-dimensional model of the target object to the point set S of the target object surface point cloud; when the coincidence degree of the mapping result and the scene point cloud does not meet the set requirements, re-select the feature points of the target object and return to process a.

[0034] Preferably, after determining the pose label of the target object, the processor adjusts the pose of the three-dimensional model of the target object according to the pose change information of the target object at each moment, and determines the two-dimensional label of the target object at each moment.

[0035] Preferably, in the processor, the determination of the two-dimensional label of the target object at each moment includes:

[0036] Performing two-dimensional projection on the three-dimensional model of the target object at each moment after pose adjustment;

[0037] The projection area at each moment obtained by projection is determined as the two-dimensional pixel-level segmentation label of the target object at the corresponding moment, and the minimum bounding rectangle of the projection area at each moment is taken, and the minimum bounding rectangle is determined as the two-dimensional object detection label of the target object at the corresponding moment;

[0038] Set the category label of the target object based on user input.

[0039] A computer-readable storage medium stores computer instructions, which, when executed by a processor, can implement the above-mentioned method for generating object space information tags.

[0040] An electronic device comprising at least a computer-readable storage medium and a processor;

[0041] The processor is configured to read the executable instructions from the computer-readable storage medium and execute the instructions to implement the above-mentioned method for generating object space information tags.

[0042] As can be seen from the above technical solution, in this application, a scene video including a target object collected from multiple angles is used to generate a scene point cloud and the posture change information of the image sensor that characterizes the position change of the image sensor at each moment; the target object feature points selected in the scene point cloud are used to align the target object three-dimensional model with the scene point cloud, and the posture change information obtained by alignment is determined as the target object posture change information at the initial moment in the image sensor coordinate system; based on the target object posture change information at the initial moment and the posture change information of the image sensor at each moment, the target object posture change information at each moment in the image sensor coordinate system is determined, and the target object posture change information at each moment is determined as the posture label of the target object at the corresponding moment. Through the above method, the initial posture label of the target object in the image sensor coordinate system can be obtained by aligning the target object three-dimensional model with the scene point cloud, and then the posture change information of the image sensor itself during the shooting process is used to infer the posture label of the target object at other moments. In this way, the label of the object in three-dimensional space can be automatically obtained, which greatly improves the efficiency of label generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the basic process of the method for generating the object space information label in the embodiment of the present application;

[0044] Figure 2 An example flow chart of a method for generating object spatial information labels;

[0045] Figure 3 This is a schematic diagram of the basic structure of the device for generating the object space information label in this application;

[0046] Figure 4 This is a schematic diagram of the specific functional unit composition of the generating device in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical means and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings.

[0049] The basic idea of ​​this application is to use the registration of the object's 3D model to the scene point cloud to obtain the object's pose change information in the image sensor coordinate system at the initial moment (i.e., the pose label at the initial moment), and refer to the relative pose of the image sensor when generating the scene point cloud to calculate the pose change information of the object at other moments, and then determine the 2D label based on the pose labels at all moments. Among them, the image sensor coordinate system is also the imaging coordinate system of the image sensor, and the image sensor can be a camera, and the corresponding image sensor coordinate system is also the camera coordinate system;

[0050] Figure 1 This is a basic flow chart of the method for generating object space information labels in the embodiment of this application. Figure 1 As shown, the method includes:

[0051] Step 101 : Using scene videos including a target object collected from multiple angles, a scene point cloud and position change information of an image sensor representing position changes of the image sensor at various moments are generated.

[0052] In a scene including a target object, a front-end device captures scene images from multiple angles within a set time period to obtain a scene video within the set time period. The specific method for capturing the scene video can adopt various existing methods. For example, the image sensor used to capture the scene video (such as a camera) can be implemented using an RGBD sensor. Specifically, the RGBD sensor can be mounted on a robot arm or simply handheld, and the position of the RGBD sensor can be changed within the set time period to perform data collection.

[0053] The processing of generating a scene point cloud using the collected scene video can adopt various existing methods. For example, a 3D reconstruction method that provides image sensor posture tracking (such as the ElasticFusion algorithm with default parameter settings) is used to generate a scene point cloud. The scene point cloud includes a point set of the scene captured at each moment, which also includes a point set of the target object at each moment. While generating the scene point cloud, the posture change information of the image sensor that represents the position change of the image sensor at each moment is recorded. Among them, the representation of the posture change information of the image sensor can be represented in a variety of ways. In this embodiment, the posture change information is represented by a posture change matrix, and the posture change matrix is ​​also used to represent various subsequent posture change information in this embodiment. Assume that the scene video starts from time t0 to time t n End, then record the time from t0 to t1, the time from t1 to t2, ..., t i-1 to t i time, ..., t n-1 to t n The pose change matrix of the image sensor at the moment {(R c0,c1 |T c0,c1 )、(R c1,c2 |T c1,c2 ),…,(R ci-1,ci |T ci-1,ci ),…,(R cn-1,cn |T cn-1,cn )}, where t0 corresponds to the initial moment of the scene video, t n The last moment of the corresponding scene video, R ci-1,ci Indicates time t i-1 At time t i The rotation matrix of the image sensor pose change, T ci-1,ci Indicates time t i-1 At time t i The translation vector representing the change in the image sensor's pose. Here, the interval between adjacent moments is fixed and can be determined based on the label's requirements. Generally, the interval between adjacent moments is the interval between video frames, with each moment corresponding to a specific video frame.

[0054] Step 102: Acquire a three-dimensional model of the target object.

[0055] The 3D model of the target object can be pre-generated or generated in real time. Various existing methods can be used for this purpose, such as the Intel RealSense SR300 RGBD sensor solution or the Artec Space Spider 3D scanner solution. This embodiment uses a mesh as an example 3D model. This step obtains the generated 3D model of the target object.

[0056] Step 103 : performing registration of the target object 3D model to the scene point cloud based on the target object feature points selected in the scene point cloud, and obtaining the target object pose change information at the initial moment in the image sensor coordinate system.

[0057] In this step, the target object mesh is aligned with the scene point cloud. The purpose of the alignment in this application is to obtain the pose change of the target object mesh relative to the image sensor A at the initial moment of the video. Therefore, when selecting feature points for alignment, it is necessary to select target object feature points in the scene point cloud. Generally, several points with obvious texture features on the surface of the target object can be selected. In addition, it is not necessary to obtain very accurate alignment results in this application. Therefore, a smaller number of feature points can be selected to participate in the alignment. The specific number of feature points N is at least 3, which can be determined based on the balance between the processing power and accuracy requirements of the equipment. In this embodiment, the selection of three feature points p1, p2, and p3 is used as an example for explanation. Accordingly, three points q1, q2, and q3 are selected on the surface of the target object mesh that roughly correspond to the positions of p1, p2, and p3.

[0058] Next, the selected feature points are used for registration. The specific registration process can adopt existing methods. An example of a specific implementation is given below:

[0059] a) Generate coarse-precision pose: Use the point pairs (p1, q1), (p2, q2), and (p3, q3) to construct a least squares problem and find the pose change information that minimizes the sum of square errors between the points q1′, q2′, q3′ mapped to the image sensor coordinate system and p1, p2, and p3. In this embodiment, this is the pose change matrix (R cu |T cu ), as follows:

[0060]

[0061]

[0062] T cu =[t x t y t z ] T , (3)

[0063] Where n = 3, α, β, and γ represent the rotation angles of the feature points selected from the scene point cloud along the x, y, and z axes, respectively, and t x , t y , t z They represent the translation of the feature points selected in the scene point cloud along the x, y, and z axes respectively. The three groups of corresponding points can uniquely determine α, β, γ, and t x , t y, t z These 6 unknown parameters are obtained through these 6 parameters (R cu |T cu ), as the coarse-precision pose change matrix;

[0064] b) Generate a fine registration area: The target object mesh is transformed into a coarse-precision pose change matrix (R cu |T cu ) is mapped to the scene point cloud, and it is determined whether the overlap between the mapping result and the scene point cloud meets the set requirements (for example, the overlap is greater than 70%). If so, all points within the set range of the mapping result (for example, within 1 cm) are obtained in the scene point cloud to form a target object surface point cloud set S for fine registration, and the following step c) is executed; otherwise, the target object feature points p1, p2, p3 and q1, q2, q3 are reselected, and the process returns to step a);

[0065] c) Generate a fine pose change matrix: align the target object mesh to the target object surface point cloud point set S to obtain the corrected fine pose change information, which is the corrected fine pose change matrix (R jing |T jing ). Registration can be performed using existing methods, such as using point cloud registration (ICP) algorithms.

[0066] Through the above processing, the position change information of the target object relative to the image sensor A at the initial moment, that is, the position change matrix of the image sensor A at the initial moment, can be obtained.

[0067] Step 104 : Calculate the target object's pose change information at each moment in the image sensor coordinate system based on the image sensor's pose change information at each moment and the target object's pose change information at the initial moment.

[0068] Through the processing of step 102, the posture change information of the image sensor at each moment, that is, the posture change matrix, has been obtained. Through the processing of step 103, the posture change matrix of the target object at the initial moment is obtained, which can be regarded as the initial posture change matrix of the target object. Then, after the posture change matrix of the target object at the initial moment and the relative posture of the image sensor at each moment in the video (that is, the posture change matrix) are known, the posture change matrix of the target object at other moments in the video sequence except the initial moment can be solved based on the initial posture change matrix and the matrix operation of the relative posture of the image sensor at the current moment and the previous moment. That is to say, based on the posture change matrix of the image sensor at each moment and the posture change matrix of the target object at the initial moment, the posture change matrix of the target object at each moment in the image sensor coordinate system is determined.

[0069] The following describes how to determine the target object pose change matrix at each moment. Specifically, starting from the initial moment t0, the pose change matrix at each moment t is determined in sequence. i The target object pose change matrix (R ti |T ti ):

[0070] (R t0 |T t0 )=(R jing |T jing ),

[0071] (R t1 |T t1 )=(R t0 ×R c0,c1 |R t0 ×T c0,c1 +T t0 ),

[0072] (R t2 |T t2 )=(R t1 ×R c1,c2 |R t1 ×T c1,c2 +T t1 ),

[0073] …

[0074] (R ti |T ti )=(R ti-1 ×R ci-1,ci |R ti-1 ×T ci-1,ci +T ti-1 )

[0075] …

[0076] (R tn |T tn )=(R tn-1 ×R cn-1,cn |R tn-1 ×T cn-1,cn +T tn-1 ),

[0077] Among them, (R ti |T ti )=(R ti-1 ×R ci-1,ci |R ti-1 ×T ci-1,ci +T ti-1 ) means R ti =R ti-1 ×R ci-1,ci , T ti =R ti-1×T ci-1,ci +T ti-1 .

[0078] Step 105 : determining the target object's posture change information at each moment as the target object's posture label at the corresponding moment.

[0079] The target object posture change information at each moment is obtained through step 104, which is the target object posture change matrix in this embodiment. The corresponding posture change matrix is ​​the posture label of the target object at the corresponding moment.

[0080] So far, Figure 1 The basic process of the label generation method shown can be ended. Through the above processing, by using the three-dimensional modeling of the target object and generating the scene point cloud containing the target object, according to the selected feature points of as few as 3 areas, the three-dimensional labels of the target object can be automatically generated in batches within a set time, which greatly improves the generation efficiency of the object space information label, and further improves the generation efficiency of the training data. Next, preferably, Figure 1 The method shown may further include step 106 after step 105 , in which the three-dimensional label is used to generate a two-dimensional label of the target object at each moment.

[0081] Step 106 : Adjust the position of the three-dimensional model of the target object according to the position change information of the target object at each moment, and determine the two-dimensional label of the target object at each moment.

[0082] From time t0 to time t n The pose change information of the target object in the image sensor coordinate system (in this embodiment, the pose change matrix of the target object) is used to adjust the pose of the target object mesh. The adjusted target object mesh at each moment is projected from 3D to 2D. Then, based on the projection results corresponding to each moment, the 2D label of the target object at the corresponding moment is determined according to existing methods. Specifically, the 2D pixel-level segmentation label can be the projected area obtained by projection. The minimum bounding rectangle of the projected area is taken to obtain the 2D target detection label of the target object, and the category label of the target object is set according to user input.

[0083] At this point, it is possible to further automatically generate two-dimensional labels for target objects in batches within a set time, greatly improving the efficiency of generating object spatial information labels, and thus improving the efficiency of generating training data.

[0084] Here is a typical example of a method for generating object spatial information labels: Figure 2The detailed processing flow of the generation method is shown in Figure 1. The RGBD sensor is used to shoot the scene video, and the object space information label at the corresponding moment of each video frame in the scene video is obtained. The posture change information is still represented by the posture change matrix. Figure 2 As shown, the detailed process includes:

[0085] 201, receiving the scene video captured by the RGBD sensor;

[0086] 202, generating scene point cloud;

[0087] 203, obtaining the relative position of the camera between frames;

[0088] 204, receiving the generated target object mesh;

[0089] 205, receiving three feature points of the target object surface selected in the scene point cloud;

[0090] 206, receiving the corresponding three feature points selected in the target object mesh;

[0091] 207, generating a coarse-precision pose change matrix of the target object;

[0092] 208, determining whether the overlap degree of the target object mesh when mapped to the scene point cloud using the coarse-precision pose change matrix is ​​greater than 70%. If so, execute 209, otherwise return to 205 and 206;

[0093] 209, obtaining a fine point cloud area to be registered of the target object;

[0094] 210, generating an initial fine pose change matrix of the target object;

[0095] 211, obtaining the target object pose change matrix for each frame;

[0096] 212, generate three-dimensional and two-dimensional labels of the target object in each frame.

[0097] The present application also provides a device for generating object space information labels, which can be used to implement the label generation method of the present application. Figure 3 This is a basic structural diagram of the device for generating object space information tags in this application. Figure 3 As shown, the generating device includes: a memory and a processor.

[0098] The memory stores scene videos including target objects collected from multiple angles.

[0099] A processor uses the scene video saved in a memory to generate a scene point cloud and the pose change information of an image sensor that characterizes the position change of the image sensor at each moment; uses the target object feature points selected from the scene point cloud to perform registration of the target object three-dimensional model to the scene point cloud, and determines the pose change information obtained by registration as the target object pose change information at the initial moment in the image sensor coordinate system; determines the target object pose change information at each moment in the image sensor coordinate system according to the pose change information of the image sensor at each moment and the target object pose change information at the initial moment, and determines the target object pose change information at each moment as the pose label of the target object at the corresponding moment. Wherein, the scene point cloud includes a point set of the captured scene at each moment.

[0100] Optionally, the generating device may further include an image sensor that collects the scene video including the target object from multiple angles and saves it in the memory.

[0101] Optionally, in the processor, to determine the target object pose change information (R i |T ti ) at moment t ti : (R ti |T ti ) = (R ti-1 ×R ci-1,ci |R ti-1 ×T ci-1,ci +T ti-1 ), where (R ti-1，ti |T ti-1,ti ) is the rotation matrix of the pose change of the image sensor from moment t i-1 to moment t i , and is the translation vector of the pose change of the image sensor from moment t i-1 to moment t i , and 0 < i ≤ n is a positive integer.

[0102] Optionally, in the processor, the method for registering the target object three-dimensional model to the scene point cloud may include:

[0103] a. Using the target object feature points selected from the scene point cloud and the feature points selected on the target object three-dimensional model corresponding to the target object feature points, construct a least squares problem, and obtain the rough accuracy pose change information that minimizes the sum of the squared errors between the feature points on the three-dimensional model mapped to the points in the image sensor coordinate system and the target object feature points selected from the scene point cloud;

[0104] b. Map the three-dimensional model of the target object to the scene point cloud using coarse-precision pose change information. When the degree of overlap between the mapping result and the scene point cloud meets the set requirements, obtain all points within the set range of the mapping result in the scene point cloud to form a target object surface point cloud set S for fine registration, and perform registration from the target object three-dimensional model to the target object surface point cloud set S. When the degree of overlap between the mapping result and the scene point cloud does not meet the set requirements, reselect the target object feature points and return to process a.

[0105] Optionally, after determining the pose label of the target object, the processor adjusts the pose of the three-dimensional model of the target object according to the pose change information of the target object at each moment, and determines the two-dimensional label of the target object at each moment.

[0106] Optionally, in the processor, a method of determining the two-dimensional label of the target object at each moment may include:

[0107] The three-dimensional model of the target object at each moment after the posture adjustment is projected in two dimensions; the projection area at each moment obtained by projection is determined as the two-dimensional pixel-level segmentation label of the target object at the corresponding moment, the minimum enclosing rectangle of the projection area at each moment is taken, and the minimum enclosing rectangle is determined as the two-dimensional target detection label of the target object at the corresponding moment; the category label of the target object is set according to the user input.

[0108] The specific implementation of the above generating device is described below through a specific embodiment. Figure 4 It is composed of the specific functional units of the generating device in the embodiment of the present application, specifically including: a storage unit, a scene point cloud generating unit, a registration unit and a label generating unit.

[0109] The storage unit is used to store scene videos including target objects collected from multiple angles. The scene point cloud generation unit is used to generate a scene point cloud and image sensor posture change information representing changes in the image sensor position at each moment using the scene video received by the receiving unit; wherein the scene point cloud includes a set of points of the scene captured at each moment. The registration unit is used to align the target object's three-dimensional model to the scene point cloud based on the target object's feature points selected in the scene point cloud, and obtain the target object's posture change information at the initial moment in the image sensor coordinate system. The label generation unit is used to determine the target object's posture change information at each moment in the image sensor coordinate system based on the image sensor's posture change information at each moment and the target object's posture change information at the initial moment, and determine the target object's posture change information at each moment as the posture label of the target object at the corresponding moment.

[0110] Preferably, the generating device may further include the image acquisition unit, which acquires scene videos including the target object from multiple angles and saves the videos into the storage unit.

[0111] Preferably, in the label generation unit, the pose change information of the target object at time t in the image sensor coordinate system i can include (R ti |T ti ):

[0112] (R ti |T ti ) = (R ti-1 ×R ci-1,ci |R ti-1 ×T ci-1,ci +T ti-1 ),

[0113] where (R ti-1，ti |T ti-1,ti ) is the rotation matrix of the pose change of the image sensor from time t i-1 to time t i , and is the translation vector of the pose change of the image sensor from time t i-1 to time t i , and 0 < i ≤ n is a positive integer.

[0114] Preferably, the number of feature points of the selected target object can be 3.

[0115] Preferably, in the registration unit, the process of registering the 3D model of the target object to the scene point cloud can include:

[0116] a. Using the feature points of the target object selected in the scene point cloud and the feature points selected on the 3D model of the target object corresponding to the feature points of the target object, construct a least squares problem, and obtain the pose change information that makes the sum of the squared errors between the feature points on the 3D model mapped to the points in the image sensor coordinate system and the feature points of the target object selected in the scene point cloud reach the minimum value as the coarse-precision pose change information;

[0117] b. Map the 3D model of the target object to the scene point cloud using the coarse-precision pose change information. When the coincidence degree of the mapping result and the scene point cloud meets the set requirements, obtain all the points within the set range of the mapping result in the scene point cloud, and form a point set S of the surface point cloud of the target object for fine registration, and perform the registration of the 3D model of the target object to the point set S of the surface point cloud of the target object; when the coincidence degree of the mapping result and the scene point cloud does not meet the set requirements, reselect the feature points of the target object, and return to step a.

[0118] Preferably, the label generation unit can also be used to perform pose adjustment on the 3D model of the target object according to the pose change information of the target object at each moment, and determine the 2D label of the target object at each moment.

[0119] In the label generation unit, the process of determining the two-dimensional label of the target object at each moment may include:

[0120] The three-dimensional model of the target object at each moment after posture adjustment is projected two-dimensionally, and the projection area obtained at each moment is used as the two-dimensional pixel-level segmentation label of the target object at the corresponding moment. The minimum circumscribed rectangle of the projection area at each moment is taken to obtain the two-dimensional target detection label of the target object at the corresponding moment, and the category label of the target object is set according to the user input.

[0121] This application also provides a computer-readable storage medium storing instructions that, when executed by a processor, can perform the steps of the method for generating object space information labels as described above. In practical applications, the computer-readable medium can be included in the devices / apparatuses / systems described above, or it can exist independently and not be incorporated into the devices / apparatuses / systems. The computer-readable storage medium stores instructions that, when executed by a processor, can perform the steps of the method for generating object space information labels as described above.

[0122] According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but is not intended to limit the scope of protection of the present application. In the embodiments disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0123] Figure 5 This application also provides an electronic device. Figure 5 , which shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0124] The electronic device may include a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 502 is executed, a method for generating an object spatial information tag may be implemented.

[0125] Specifically, in actual applications, the electronic device may further include components such as a power supply 503 and an input / output unit 504. Those skilled in the art will appreciate that Figure 5The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0126] The processor 501 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 502 and calling data stored in the memory 502, it executes various functions of the server and processes data, thereby monitoring the electronic device as a whole.

[0127] The memory 502 can be used to store software programs and modules, i.e., the computer-readable storage medium mentioned above. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application required for a function, etc.; the data storage area may store data created based on the use of the server, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0128] The electronic device also includes a power supply 503 for supplying power to various components. This power supply 503 can be logically connected to the processor 501 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 503 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0129] The electronic device may further include an input / output unit 504. The input / output unit 504 may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. The input / output unit 504 may also be used to display information input by the user or information provided to the user, as well as various graphical user interfaces. These graphical user interfaces may be composed of graphics, text, icons, videos, or any combination thereof.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating an object spatial information label, characterized in that: include: Using a scene video including a target object captured from multiple angles, a scene point cloud and image sensor posture change information representing the image sensor posture change at each moment are generated; wherein the scene point cloud includes a point set of the scene captured at each moment; Using the target object feature points selected from the scene point cloud, aligning the target object three-dimensional model with the scene point cloud to obtain the target object pose change information at the initial moment in the image sensor coordinate system; Determining the position change information of the target object at each moment in the image sensor coordinate system according to the position change information of the image sensor at each moment and the position change information of the target object at the initial moment; Determining the target object's posture change information at each moment as a posture label of the target object at the corresponding moment; Wherein, the moment of determining the image sensor coordinate system is t i The target object pose change information include: ， in, For the moment t i-1 At the time t i The pose change matrix of the image sensor, Indicates time t i-1 At the time t i The rotation matrix of the image sensor pose change, For the moment t i-1 At the time t i The translation vector of the image sensor’s pose change, is a positive integer, express and .

2. The generation method according to claim 1, characterized in that The registering of the target object three-dimensional model to the scene point cloud includes: a. constructing a least squares problem using the target object feature points selected in the scene point cloud and the feature points selected on the three-dimensional model of the target object corresponding to the target object feature points, to obtain coarse-precision pose change information that minimizes the sum of squared errors between the feature points on the three-dimensional model mapped to the image sensor coordinate system and the target object feature points selected in the scene point cloud; b. Map the three-dimensional model of the target object to the scene point cloud using the coarse-precision pose change information. When the degree of overlap between the mapping result and the scene point cloud meets the set requirements, obtain all points within the set range of the mapping result in the scene point cloud to form a target object surface point cloud point set S for fine registration, and perform registration of the target object three-dimensional model to the target object surface point cloud point set S. When the degree of overlap between the mapping result and the scene point cloud does not meet the set requirements, reselect the target object feature points and return to step a.

3. The generation method according to claim 1, characterized in that After determining the pose label of the target object, the method further includes: The three-dimensional model of the target object is adjusted in position according to the position change information of the target object at each moment, and the two-dimensional label of the target object at each moment is determined.

4. The generation method according to claim 3, characterized in that Determining the two-dimensional label of the target object at each time comprises: Perform two-dimensional projection on the three-dimensional model of the target object at each moment after the posture adjustment; The projection area at each moment obtained by projection is determined as the two-dimensional pixel-level segmentation label of the target object at the corresponding moment; Taking a minimum bounding rectangle of the projection area at each moment, and determining the minimum bounding rectangle as a two-dimensional target detection label of the target object at the corresponding moment; Set the category label of the target object based on user input.

5. A device for generating an object space information label, characterized in that: include: memory and processor; The memory stores scene videos including the target object collected from multiple angles; The processor generates a scene point cloud and image sensor posture change information representing position changes of the image sensor at various moments using the scene video including the target object stored in the memory; Using the target object feature points selected in the scene point cloud, aligning the three-dimensional model of the target object to the scene point cloud, and determining the pose change information obtained by the alignment as the pose change information of the target object at the initial moment in the image sensor coordinate system; determining the pose change information of the target object at each moment in the image sensor coordinate system based on the pose change information of the image sensor at each moment and the pose change information of the target object at the initial moment, and determining the pose change information of the target object at each moment as the pose label of the target object at the corresponding moment; The scene point cloud includes a point set of the scene captured at each moment; In the processor, the time in the image sensor coordinate system is determined t i The target object pose change information include: ， in, For the moment t i-1 At the time t i The pose change matrix of the image sensor, Indicates time t i-1 At the time t i The rotation matrix of the image sensor pose change, For the moment t i-1 At the time t i The translation vector of the image sensor’s pose change, is a positive integer, express and .

6. The generating device according to claim 5, characterized in that The generating device further includes the image sensor, which collects scene videos including the target object from multiple angles and saves them in the memory.

7. The generating device according to claim 5, characterized in that In the processor, registering the target object three-dimensional model to the scene point cloud includes: a. constructing a least squares problem using the target object feature points selected in the scene point cloud and the feature points selected on the three-dimensional model of the target object corresponding to the target object feature points, to obtain coarse-precision pose change information that minimizes the sum of squared errors between the feature points on the three-dimensional model mapped to the image sensor coordinate system and the target object feature points selected in the scene point cloud; b. Map the three-dimensional model of the target object to the scene point cloud using the coarse-precision pose change information. When the degree of overlap between the mapping result and the scene point cloud meets the set requirements, obtain all points within the set range of the mapping result in the scene point cloud to form a target object surface point cloud point set S for fine alignment, and perform alignment between the three-dimensional model of the target object and the target object surface point cloud point set S. When the degree of overlap between the mapping result and the scene point cloud does not meet the set requirements, reselect the target object feature points and return to process a.

8. The generating device according to claim 5, characterized in that After determining the pose label of the target object, the processor adjusts the pose of the three-dimensional model of the target object according to the pose change information of the target object at each moment, and determines the two-dimensional label of the target object at each moment.

9. The generating device according to claim 8, characterized in that In the processor, determining the two-dimensional label of the target object at each moment includes: Perform two-dimensional projection on the three-dimensional model of the target object at each moment after the posture adjustment; The projection area at each moment obtained by projection is determined as the two-dimensional pixel-level segmentation label of the target object at the corresponding moment, and the minimum bounding rectangle of the projection area at each moment is taken, and the minimum bounding rectangle is determined as the two-dimensional object detection label of the target object at the corresponding moment; Set the category label of the target object based on user input.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by a processor, the method for generating an object space information label according to any one of claims 1 to 4 can be implemented.

11. An electronic device, characterized in that: The electronic device includes at least a computer-readable storage medium and also includes a processor; The processor is configured to read executable instructions from the computer-readable storage medium and execute the instructions to implement the method for generating an object space information tag according to any one of claims 1 to 4.

Images